The Infocious Web Search Engine Improving Web Searching

The Infocious Web Search Engine: Improving Web Searching Through Linguis Analysis Alexandros Ntoulas 1, 2 Gerald Chao 1 1 Infocious Junghoo Cho 2 Inc. {ntoulas, gerald}@infocious. com 2 University of California Los Angeles {ntoulas, cho}@cs. ucla. edu

Motivation n n Current Web search engines identify relevant pages based on keyword matching Example: jaguar Jaguar Cars Official worldwide web site of Jaguar Cars. www. jaguar. com/ 3/19/2018 WWW 2005 Chiba Japan

Motivation n Is keyword matching enough ? Natural languages are inherently ambiguous Example: jaguar q q q 3/19/2018 The car brand ? Apple Mac OS X 10. 2 ? The animal ? Chemical software … WWW 2005 Chiba Japan

The Infocious Web Search Engine n Uses Language Analysis techniques to: q q 3/19/2018 Resolve ambiguities inside Web pages Rank the Web pages based on the coherence (quality) of the text Help users organize the results in intuitive ways through categorization Provide suggestions for query refinement WWW 2005 Chiba Japan

What is different about Infocious ? n n Search Engines today do not apply Language Analysis to the level Infocious does It is not simply a matter of applying existing algorithms: need optimizations for Web scale Features made possible only through language analysis Makes Language Analysis features intuitive (yet powerful) for the user 3/19/2018 WWW 2005 Chiba Japan

Architecture 3/19/2018 WWW 2005 Chiba Japan

Architecture Crawler • Follows links to discover Web pages • Refreshes changed pages using sampling [VLDB’ 02] • Can download pages from the Hidden Web [JCDL’ 05] 3/19/2018 WWW 2005 Chiba Japan

Architecture Linguistic Processing • Resolves language ambiguities [COLING’ 02] • Annotates Web pages • Extracts concepts • Extracts named entities • Operates at crawl speed 3/19/2018 WWW 2005 Chiba Japan

Linguistic Processing: Disambiguation n n Part-of-speech (POS) tagging Example: house plants . . . most house plants are hybrids of Adj Noun plant species Noun Verb Noun Prep Noun . . . garden built to house our most valuable plants. . . Noun n Verb. D Inf Verb Pron. P Adv Done probabilistically: Given sentence S, set of tags T find Tbest(S) = arg max. T P(T | S) 3/19/2018 WWW 2005 Chiba Japan Adj Noun

Linguistic Processing: Disambiguation n n POS information stored inside the index User can manually specify POS at query time (or click on examples) Query N: house N: plants Green. Patio. Com – Tips for buying house plants. Why keep natural indoor plants. . Tips for buying house plants. Care for indoor plants. . www. greenpatio. com/tips. shtml Low Light Plants for the House Is a common name for plants in the species Dieffenbachia. . As with most house plants … www. plantsgalore. com/articles/houseplants-low-lightplantfacts. htm 3/19/2018 WWW 2005 Chiba Japan

Linguistic Processing: Disambiguation n n POS information stored inside the index User can manually specify POS at query time (or click on examples) Query V: house N: plants Over Wintering Bonsai … One method is to build a cold frame to house your plants in the winter. . www. evergreengardenworks. com/overwint. htm Keeping Your Sunroom Cozy … And if you want to house a hot tub or plants, think about enclosing the … doityourself. com/sunroomcozy. htm 3/19/2018 WWW 2005 Chiba Japan

Linguistic Processing: Disambiguation n Word-sense disambiguation Previous Example: jaguar Approach through Web page categorization q q n Use the categories of DMOZ (~600, 000) Given set of categories C and a page d Find maxc C P(c|d) In Infocious a page may belong to multiple categories 3/19/2018 WWW 2005 Chiba Japan

Categorization Computers n n The category of a result is highlighted on. Mouse. Over() Allow users to restrict search within a category: jaguar cat: Computers n Can also be done by clicking on a category 3/19/2018 WWW 2005 Chiba Japan Recreation/Autos Jaguar Cars Official worldwide site of jaguar cars www. jaguar. com/ Apple Mac OS X The Apple Mac OS Product page www. apple. com/macosx/

Linguistic Processing: Concept Extraction n More accurate phrase identification: In the profession of cooking oil is an important ingredient n n Identify concepts through a set of rules (prespecified or automatically learned) Example: Verb. Phrase-Prep. Phrase-Noun. Phras q q 3/19/2018 lightly tossed with salad dressing tossed with oil and vinegar dressing tossed immediately with blue-cheese dressing Reduced to Concept: tossed with dressing WWW 2005 Chiba Japan

Answering a query n n n Default is AND-semantics Query disambiguation (e. g. in query train a pet Infocious knows train has to be a verb) Ranking takes into account a variety of factors q q 3/19/2018 Presence of keywords, Proximity Title, URL, formatting, font size, coloring etc. Popularity of a page measured by in/out links Text. Quality WWW 2005 Chiba Japan

Architecture Text. Quality • Summarize probabilities from Linguistic Processing into one metric • Promote coherent text • Demote incoherent text 3/19/2018 WWW 2005 Chiba Japan

Text. Quality (disabled) n Promotes well-written pages (preferable from the user perspective) Britney Spears Pictures – britney spears pictures … picture of britney spears, hot pictures of britney spears … britney-spears-pictures. hotyoungstars. com/nude/ Hot Britney Spears Pics - hot britney spears pics, . . . britney spears, new hot pics of britney spears, . . . hot-britney-spears-pics. hotyoungstars. com/nude/ Britney Spears Photos – britney spears photos … spears, britney spears nude photos, nude photos of … britney-spears-photos. hotyoungstars. com/nude/ Text. Quality DISABLED 3/19/2018 WWW 2005 Chiba Japan

Text. Quality (enabled) n Promotes well-written pages (preferable from the user perspective) Is Britney Spears over the edge? … Britney Spears is a singer … azwestern. edu/modern_lang/esl/cjones/mag/spring 2004/britney. htm IMPERSONATORS – BRITNEY SPEARS Is Proud to Present! Contact: Gary Shortall Back… www. impersonators. com/brittany/brit. html Britney Spears’ Coke Habit Destroys Her… www. emptyv. org/britney_spears. htm Text. Quality ENABLED 3/19/2018 WWW 2005 Chiba Japan

Other Language Analysis-Enhanced Features n n n Key phrases: Present a list of the salient concepts within the results Related topics: Concepts related to the present query Hone your search: Suggestion of more specific queries Spell Checking Personalization: I like Sports but not Politics 3/19/2018 WWW 2005 Chiba Japan

Evaluation of Categorization Using Naïve Bayes classifiers for illustration: Language Analysis improves accuracy Infocious actually employs an improved classification technique (76% accuracy) We used four different flavors of NB on 100, 000 Web pages: n n n q q 3/19/2018 C 1: Words C 2: Words + POS tags C 3: Words + extracted concepts C 4: Words + POS + extracted concepts WWW 2005 Chiba Japan

Evaluation of Categorization 3% accurary increase – 8% error reduction q q 3/19/2018 C 1: Words only C 2: Words + POS tags q q C 3: Words + extracted concepts C 4: Words + POS + extracted concepts WWW 2005 Chiba Japan

User Interface 3/19/2018 WWW 2005 Chiba Japan

Conclusion n n Infocious: uses language analysis to improve Web search Resolves language ambiguities Incorporates text coherence in the ranking Provides query suggestions and refinements Organizes information intuitively through categorization 3/19/2018 WWW 2005 Chiba Japan

Related Work n Web Search Engines: q n Enterprise Search q n Google, Yahoo!, MSNSearch, Ask/Teoma, Altavista, Looksmart, Vivisimo, … Autonomy, Inquira, Inxight, i. Phrase, … Answer Engines q 3/19/2018 START@MIT, Brain. Boost, … WWW 2005 Chiba Japan

Ongoing work n n Increase index size (currently ~1 billion pages) through surface & hidden Web-crawls Apply our Language Analysis algorithms to additional languages Leverage our Language-annotated repository for additional features (e. g. summarization, machine translation, …) Investigate how to use Language Analysis to improve relevance in advertisements 3/19/2018 WWW 2005 Chiba Japan

Thank you ! You can check out our Search Engine at: www. infocious. com 3/19/2018 WWW 2005 Chiba Japan