A digital library and/or search engine much be able to handle many different kinds of multimedia, huge numbers of simultaneous requests, and attempts by some parties to manipulate automated search engine processes.
In order to index effectively, search engines must use algorithms that can:
-Determine what content should be indexed
-Divvy up tasks between servers
-Analyze content
-Collect new URLs
-Ignore dupicate pages
-Add new URLS are added to a cue
-Ignore spam
-Avoid overloading servers which host the pages they are crawling
-Save page content for indexing indexing.
-Process simplistic or ambiguous queries like "the onion" and weed out content effectively
-multiple factors should be included in this determination
Suggested focii for improving search engine quality (Henzinger): spam, content quality, webmaster deviation from web conventions, duplicate hosts, and vaguely structured data.
Some websites use text, links, or cloaking mechanisms to improve their ranking in search result sets. This often includes adding content (key words, links, false content) in an attempt to fool search engine page ranking algorithms. Spam may be structured in the form of deceptive text (white text on a white background with text that is invisible to the user but not the webcrawler), link spam (or link farms which collect links pointing to every other page in that site), and providing entirely different content for the user and the webcrawler. Work can be done towards assessing the quality of content included in a given webpage. Some sites advertise false content (like celebrity names) in an attempt to redirect users to their site.
Most search engine users do not travel beyond the first page of search results. Since many website generate income from their traffic, there is great motivation to increases a page's ranking so that it appears on the first page of a result set. Search engine developers must constantly tweak their indexing and page ranking algorithms in order to stay one step ahead of those who wish to manipulate these processes in order to increase traffic to their own page.
David Hawking , Web Search Engines: Part 1 and Part 2 IEEE Computer, June 2006.
M. Henzinger et al. challenges in Web Search Engines. ACM SIGIR 2002.
Friday, October 16, 2009
Muddiest Point (10/16/09)
Will there be a Penapto recording for this class? I was unable to attend class due to illness. I would like to view it online.
Friday, October 9, 2009
Muddiest Point (10/28/09)
Are we supposed to submit our muddiest point and reading notes via email as well as through the blog?
Reading Notes: XML
XML is a difficult concept to understand. It is not a programming language, like Java, which uses functional elements. XML doesn't designate style elements, like HTML. In a sense, it doesn't really DO anything. It's basically a carrier and organizer of information.
XML is important because it allows different computers to interchange documents via the internet. You use XML to create your own tags which label the parts of information in a document.
< name > Jane < /name >
< role student < /student >
This way a person or a computer can identify the elements in a document easily and without ambiguity. XML is advantageous because it is a somewhat simplified version of SGML that lets you set document structure. HTML is very valuable in telling a computer how to display information on the web but it says nothing about the type of information that is the content of the document.
XML allows the user to create a list of elements tagged in a standardized way. The elements can be defined with attributes which give qualifying information about the attribute (i.e. gender, color, length, number, etc.). This can be done with quotation marks inside the original statement or in a separate statement.
The user can determine things like how to manage white space, language, and allowable characters. XML is useful for creating a detailed catalog.
DTD allows users to list and specify their own tags as well as set the order of their tags (XML does not impose a specific order on a list of elements). A DTD is a formal document which defines the roles of the elements labeled by the user. XSD (XML schema definition) can be used instead of DTD to define the structure of a document. XSD offers extensibility, consistency, power, and support for data types and namespaces (w3schools.com)
Namespaces: allows a user to differentiate between two different elements with the same name (this happens most often when XML is combined with HTML).
Questions:
*Can you please explain the use of xlms atrributes and the use of URIs ? I'm not sure I fully understand this concept.
*Can you clarify the different between terminal and non-terminal elements?
Sources:
XML is important because it allows different computers to interchange documents via the internet. You use XML to create your own tags which label the parts of information in a document.
This way a person or a computer can identify the elements in a document easily and without ambiguity. XML is advantageous because it is a somewhat simplified version of SGML that lets you set document structure. HTML is very valuable in telling a computer how to display information on the web but it says nothing about the type of information that is the content of the document.
XML allows the user to create a list of elements tagged in a standardized way. The elements can be defined with attributes which give qualifying information about the attribute (i.e. gender, color, length, number, etc.). This can be done with quotation marks inside the original statement or in a separate statement.
The user can determine things like how to manage white space, language, and allowable characters. XML is useful for creating a detailed catalog.
DTD allows users to list and specify their own tags as well as set the order of their tags (XML does not impose a specific order on a list of elements). A DTD is a formal document which defines the roles of the elements labeled by the user. XSD (XML schema definition) can be used instead of DTD to define the structure of a document. XSD offers extensibility, consistency, power, and support for data types and namespaces (w3schools.com)
Namespaces: allows a user to differentiate between two different elements with the same name (this happens most often when XML is combined with HTML).
Questions:
*Can you please explain the use of xlms atrributes and the use of URIs ? I'm not sure I fully understand this concept.
*Can you clarify the different between terminal and non-terminal elements?
Sources:
- Martin Bryan. Introducing the Extensible Markup Language (XML) http://burks.bton.ac.uk/burks/internet/web/xmlintro.htm
- Uche Ogbuji. A survey of XML standards: Part 1. January 2004. http://www-128.ibm.com/developerworks/xml/library/x-stand1.html
- Extending you Markup: a XML tutorial by Andre Bergholz http://www.pdffinder.com/pdf/extending-your-markup-an-xml-tutorial.html, or at http://xml.coverpages.org/BergholzTutorial.pdf
- XML Schema Tutorial http://www.w3schools.com/Schema/default.asp
Saturday, October 3, 2009
Flickr Photo Collection
Below is a link to my Flickr photo collection, a project for my Digital Libraries course.
The Family Zoo
The Family Zoo
Friday, October 2, 2009
Muddiest Point (9/21/09 and Syllabus)
Would you like us to email you our readings notes / muddiest point and post them in our blogs or submit them via blogs alone?
Also, on 9/21 you stated that digitization is no longer considered a means for preservation because digital objects are actually more fragile than physical ones. Does this have to do with the impermanence of URLS and changing technology, or is it something else? Can you explain?
Also, on 9/21 you stated that digitization is no longer considered a means for preservation because digital objects are actually more fragile than physical ones. Does this have to do with the impermanence of URLS and changing technology, or is it something else? Can you explain?
Reading Notes: Metadata
Subject classification has long been a standard of library science. Subject classification uses hierarchical relationships to describe objects and their correlation with similar items. In contrast, metadata is any order-of-magnitude information about information (the description of a particular object) either in digital or physical format.
Digital metadata is used to describe digital objects (i.e. text documents, images, video, etc.). Digital metadata is ideally embedded in the object it describes so that is is not displaced if the object is used. Some metadata is static and never changes while some is dynamic and is used for updating information about an object and how it changes over time. Metadata can also be used to extract information about a text: language, acronyms and their meaning, names of people, time and date stamps, email addresses, phrase hierarchy, etc.
In Objects of a Biographical System, Witten states that there is little need for subject classification in a digital world. Isn't subject classification still useful for helping the user retrieve information and find other items that may be of interest? Metadata can also meet this need but is there anything subject classification can do which metadata cannot?
Witten discussed the possible future of MPEG-7 files and their potential ability to evaluate a few notes of music and identify similar melodies, to retrieve graphics or logos from a few user-made digital brush strokes, identify the source of sounds from pitch samples, and to describe movements from actions in video files. Have there been any like developments since this work was published? Are projects aiming to discern human gestures and postures from video recordings likely done using metadata from MPEG-7 files or some other technology?
Anne Gilliland discusses metadata's long-term benefits. She says the following,
"What we do know is that the existence of many types of metadata will prove critical to the continued online and intellectual accessibility and utility of digital resources and the information objects that they contain, as well as the original objects and collections to which they relate. In this sense, metadata provides us with the Rosetta stone that will make it possible to decode information objects and their transformation into knowledge in the cultural heritage information systems of the future."
Is she referring to digital objects as the future artifacts of our culture? If so, what digital objects may be of importance to our predecessors? I wonder if they will have interoperability issues or if technology will have solved problems like interoperability by then.
Weibel's discusses difficult and unanticipated problems in the creation of a universal standard for metadata creation. He says the following,
" To borrow from the oldest joke of the Dismal Profession, put all the data modelers in the world end to end, and you won't reach a conclusion (we did, but it took ten years to manage it)."
What kind of problems is he referring to? How do data couplers help to find solutions? What problems have been resolved since this article was written (Summer 2005)?
If professionals are unable to come to a consensus on metadata standards, is it possible to create software that can identify the different standards used and accommodate them simultaneously (or even convert them to a standard format)?
Digital metadata is used to describe digital objects (i.e. text documents, images, video, etc.). Digital metadata is ideally embedded in the object it describes so that is is not displaced if the object is used. Some metadata is static and never changes while some is dynamic and is used for updating information about an object and how it changes over time. Metadata can also be used to extract information about a text: language, acronyms and their meaning, names of people, time and date stamps, email addresses, phrase hierarchy, etc.
In Objects of a Biographical System, Witten states that there is little need for subject classification in a digital world. Isn't subject classification still useful for helping the user retrieve information and find other items that may be of interest? Metadata can also meet this need but is there anything subject classification can do which metadata cannot?
Witten discussed the possible future of MPEG-7 files and their potential ability to evaluate a few notes of music and identify similar melodies, to retrieve graphics or logos from a few user-made digital brush strokes, identify the source of sounds from pitch samples, and to describe movements from actions in video files. Have there been any like developments since this work was published? Are projects aiming to discern human gestures and postures from video recordings likely done using metadata from MPEG-7 files or some other technology?
Anne Gilliland discusses metadata's long-term benefits. She says the following,
"What we do know is that the existence of many types of metadata will prove critical to the continued online and intellectual accessibility and utility of digital resources and the information objects that they contain, as well as the original objects and collections to which they relate. In this sense, metadata provides us with the Rosetta stone that will make it possible to decode information objects and their transformation into knowledge in the cultural heritage information systems of the future."
Is she referring to digital objects as the future artifacts of our culture? If so, what digital objects may be of importance to our predecessors? I wonder if they will have interoperability issues or if technology will have solved problems like interoperability by then.
Weibel's discusses difficult and unanticipated problems in the creation of a universal standard for metadata creation. He says the following,
" To borrow from the oldest joke of the Dismal Profession, put all the data modelers in the world end to end, and you won't reach a conclusion (we did, but it took ten years to manage it)."
What kind of problems is he referring to? How do data couplers help to find solutions? What problems have been resolved since this article was written (Summer 2005)?
If professionals are unable to come to a consensus on metadata standards, is it possible to create software that can identify the different standards used and accommodate them simultaneously (or even convert them to a standard format)?
- Ian H. Witten. “How to Build a Digital Library”. Morgan Kaufmann Publisher. 2002. ISBN: 1-558-60790-0.
- Anne J. Gilliland. Introduction to Metadata, pathways to Digital Information: 1: Setting the Stage http://www.getty.edu/research/conducting_research/standards/intrometadata/setting.html
- Stuart L. Weibel, “Border Crossings: Reflections on a Decade of Metadata Consensus Building”, D-Lib Magazine, Volume 11 Number 7/8, July/August 2005 http://www.dlib.org/dlib/july05/weibel/07weibel.html
Subscribe to:
Posts (Atom)