The Invisible Web n Definition n Searching

The Invisible Web n. Definition n. Searching

The Invisible Web Also called: n deep content n hidden internet n dark matter

The Invisible Web The vast number of pages that search engines cannot or will not index n Restricted: login, password (such as intranets, databases; private, proprietary) n Sites not linked from anywhere (undiscovered) n Sites that use a robots. txt file to keep files off limits from spiders n Unsearchable or un-indexable file formats n Non-static - searchable databases that only produce results dynamically in response to a specific search request (such as CGI, ASP, CFM) n Real-time data – changes rapidly – too “fresh” n Sites that are too “deep”

The Invisible Web Search engines often avoid indexing web pages that are delivered dynamically, such as via database programs: n Often, the search engine may not like the URL used in order to retrieve the document. Many dynamic delivery mechanisms make use of the ? symbol. n For example, a page may be found this way: http: //www. website. com/cgi-bin/getpage. cgi? name=sitemap n Most search engines will not read past the ? in that URL. n

The Invisible Web sources tend to be: n More current n More comprehensive n Searchable (however, not by SE’s) n More specific/targeted n Deeper breadth n Often better quality

The Invisible Web n n n n Top types of “invisible” information News RSS Blogs Public company filings, stock prices Customized maps and directions Clinical trials Telephone numbers and addresses, postal codes Definitions Job postings Grant information Statistics Weather Museum, gallery, and library holdings

Finding the “Dark Matter” Search Engines n Specialized Search Engines n Directories n Vortals n

Traditional Search Engines incorporation of “Invisible” Databases n Weather n Maps n Phone directories n Catalogs n Stock prices

Traditional Search Engines Unless specially, programmed, though, spiders can’t find all the valuable resources available

Specialized Search Engines Search deeper into sites: n Go beyond top page, or homepage n Choose sources to spider—topical sites only n “Smart” ranking and indexing based on knowledge of the specific subject

Specialized Search Engines There are hundreds of specialized search engines for almost every topicn Search Engine Guide n Specialty Search Engines

Directories Collections of pre-screened web-sites into categories based on a controlled ontology n Ontology: classification of human knowledge into topics, similar to traditional library catalogs n

Directories Closed Model: paid editors; quality control (Look. Smart, Yahoo) n Open Model: volunteer editors; (Open Directory Project, Google) n

Directories Easier access to relevant results n Faster n Access to materials not always indexed by search engines—content in databases or file types not searched by spiders n

Directories Issues with directories: n Inherently small n Unseen editorial policies ¨ May charge for listing ¨ Lopsided coverage n Timeliness--Harder to keep updated

Search

Vortals n n Vortals: vertical-portal. Instead of being a horizontal, all-inclusive entry point into the Web, they are vertical, specialized entry points. Comprehensive sites focusing on gathering and providing links to the best resources in a specific topic. Usually are combined subject-specific search engines and subject-specific directories Also called “focused crawlers”; metasites; guru; authority; industry guide; subject directory site

Vortals Advantages – best of directories and subject specific search engines n More up-to-date - crawl subject specific pages more often n Deeper crawl - gets more of the content on each server n More precision, less recall

Searching the Invisible Web How do you find these sites? n Use directories known directories to find invisible web searching and browsing tools: ¨ Librarians’ Index to the Internet ¨ Open Directory ¨ Google Directory ¨ Teoma works well, too.

Searching the Invisible Web n n Rethink your search: Think key terms specific details – macro vs. micro Example you want to find the melting point of hydrogen peroxide. On the general web, you’d put in the key words melting, point, and “hydrogen peroxide” On the invisible web, you look for chemical databases, which included melting points as one feature of the database, once in the database, then you’d search for hydrogen peroxide

Searching the Invisible Web Remember some concepts are assumed n Do not use the subject a search term n Example: If you are looking for information on gender inequity in math education, exclude terms like education from your search in Ask. ERIC, an education specific search tool

Mining the Invisible Web n n n Tips: Certain kinds of sites can prove to be clearinghouses of information: Government - statistics of all kinds Professional organizations - archives of relevant research and statistics Media sites (TV and Radio) – transcripts and speeches College and university professor sites – lectures and personal publications

Mining the Invisible Web n Look for library guides and commercial portals for more guidance in finding the hidden, valuable content available for free on the Web (more on this in the next lesson): n My Ready Reference on the Web Resource