A Linguistic Solution to Perfecting Search Technology A

A Linguistic Solution to Perfecting Search Technology A new tool to improve the filtering options in advanced searching Fernando Moreno-Torres General Manager MTC Soft www. mtcsoft. es

The Problem (I): different meanings European Commission concrete Results: 967. 000 European Commission - Research: Industrial technologies - Robot. . . Concrete repair The permanence of our built environment can easily be taken for granted. Materials such as concrete, that are used to construct our roads, . . . Concrete tasks or activities contained in White Paper Financial. . . European Commission End 2006. 46) Concrete actions in mortgage credit (follow-up White Paper) European Commission 2007 and. . . UNCTAD XI: European Commission calls for concrete action to assist. . . The European Commission will call for concrete global measures to support commodity dependent developing countries and their producers at the 11. . . European Commission – Information roadshow on the European. . . European citizenship… not just words, concrete rights! - Information roadshow organised by the European Union. European Commission. . Madrid – April, 2009 www. mtcsoft. es 2

The Problem (II): no relations between words Tom Cruise bought house Results: 169. 000 Doctors not fans of Tom Cruise's baby gift - Women's health- msnbc. com 6 Dec 2005. . . Tom Cruise bought a sonogram machine so that he and girlfriend Katie Holmes can see their unborn. . Video: From 'House' to White House. . . He shows them the money - Los Angeles Times 6 May 2007. . . Despite speculation that Tom Cruise and Katie Holmes are having marital problems, he just bought a Beverly Hills home for his family that cost. . . The one-story, 13000 -square-foot house was built on speculation this year … Tom Cruise Buys $35 Million Beverly Hills Mansion - Katie Holmes. . . 7 May 2007. . . ft. structure in 1937, the house was expanded four years ago and. . . have bought a multi-million-dollar house only two minutes away. They plan to move in July. Tom Cruise's $35 million Beverly Hills mansion Photo by: Tom Cruise Bought Katie Holmes An Engagement Ring Right After. . . Tom Cruise bought wife Katie Holmes an engagement ring after their first date. . . Charlotte Church Kicks Her Boyfriend Out Of House After Heated Argument. . . Madrid – April, 2009 www. mtcsoft. es 3

The problem Several-word searches Common words Too many results, most of then not related to the query ¿Why? Search engines don’t discriminate different meanings of words Words are not related in the texts Madrid – April, 2009 www. mtcsoft. es 4

The solution Parsing and tagging the texts - Discovering the real meaning of every word in every sentence - Discovering the relations between words Ø The search engines can use this information: Ø to filter the results Ø to get accurate results Ø With these “enriched” texts we can give users new possibilities to filter the search using additional options. Madrid – April, 2009 www. mtcsoft. es 5

The advanced search query Who? Tom Cruise The sentence subject The actor What does he do? To buy The sentence verb The action Concerning what? House The sentence direct object The object Where? In Hollywood (optional) The place When? Last Month (optional) The time Madrid – April, 2009 www. mtcsoft. es 6

The new query and the new results Tom Cruise is the subject of a sentence in which buy is the verb and house is the direct object This query would find: - Tom Cruise has bought a house… - Tom Cruise and Katie Holmes will buy a new house …. - Yesterday Tom Cruise bought two dogs and a red house… This query would not find: - Tom Cruise bought a dog and visited a house in … - Tom Cruise is very famous. Madonna has bought an expensive house… - Tom Cruise doesn’t house any friends, because he has bought a new car and … Madrid – April, 2009 www. mtcsoft. es 7

How did we make it? Ø A 10 -year project Ø Based on the linguistic analysis from a philologist and professional translator, working for the European Commission for more than 20 years Ø Ø It’s not Statistical Natural Language Processing It’s not Artificial Intelligence software It’s not the work of software engineers Ø It’s the core of our new Automatic Translator (to be released next May 20 th) Ø It’s a design of a philologist, constructed by software developers, MTC Soft, over the last three years Ø We perform Word Sense Disambiguation using linguistic knowledge Madrid – April, 2009 www. mtcsoft. es 8

Where it comes from PHASE 1 PHASE 2 3 Grammatical Analysis 4 Syntactic Analysis VICTOR Translator Translation SPANISH ENGLISH Symbol identification PHASE Parsalyser The translator has 4 main phases or modules. The first 3, in English, analyze and tag every word of the text. This is the “Parsalyser. ” Madrid – April, 2009 www. mtcsoft. es 9

PHASE 1 PHASE 2 Symbol identification 3 Grammatical Analysis Syntactic Analysis Original text Germany is supporting the development of infrastructure (especially energy and water supplies), promotion of the economy and employment, advising the PISG on the restructuring of administrative structures in the education and vocational training area and the improvement of the general economic climate. In just a few seconds Tagged text Germany is supporting the development of infrastructure (especially energy and water supplies), promotion of the economy and employment, advising the PISG on the restructuring of administrative structures in the education and vocational training area and the improvement of the general economic climate. Madrid – April, 2009 How does it work? + 140 Processes + 500 linguistic rules The text is analyzed in every process Every process does its work - Word identification - Capital letter identification - “How” gets special treatment - “Could” gets special treatment -… - Disambiguation: Verb – Noun - Adjective – Pronoun - Verb – Preposition -… - + 140 Processes www. mtcsoft. es 10

PHASE 1 2 Symbol identification Ø Ø Ø PHASE 3 Grammatical Analysis Phase 1 – Symbol identification Syntactic Analysis Identification of: words, symbols and expressions Identification of root words (using our dictionary) Assignation of possible grammar functions Original text: Tom Cruise has just bought a house that costs $35 million. Words identified: Madrid – April, 2009 www. mtcsoft. es 11

PHASE 1 2 Symbol identification Ø Grammatical Analysis Phase 2 – Grammatical Analysis PHASE 3 Syntactic Analysis Disambiguation: Ø Performing an exhaustive linguistic analysis Original text: Tom Cruise has just bought a house that costs $35 million. Text after this phase: Madrid – April, 2009 www. mtcsoft. es 12

PHASE 1 2 Symbol identification Ø Ø Grammatical Analysis Phase 3 – Syntactic Analysis PHASE 3 Syntactic Analysis Identification of the syntagms: Both simple and complex Both autonomous and subordinate Identification of syntactic and semantic relations between them Original text: Tom Cruise has just bought a house that costs $35 million. Syntactic functions: Subject Madrid – April, 2009 Verb Object www. mtcsoft. es 13

PHASE 1 Symbol identification PHASE 2 Grammatical Analysis PHASE 3 Syntactic Analysis An example Original text: On the basis of an annual report from the Commission, it shall also examine the effects of the special arrangements with regard to drugs, including the progress in the fight against drugs made by countries listed in the second annex and, if progress is insufficient, the Commission will consider taking any measures to suspend in whole or in part the application of Article 7, in accordance with the established procedure in Article 32 and after consulting the country concerned. Madrid – April, 2009 www. mtcsoft. es 14

PHASE 1 PHASE 2 Symbol identification Grammatical Analysis PHASE 3 An example Syntactic Analysis After Phase 1: Words identified: Madrid – April, 2009 www. mtcsoft. es 15

PHASE 1 PHASE 2 Symbol identification Grammatical Analysis PHASE 3 An example Syntactic Analysis After Phase 2: Words disambiguated: Madrid – April, 2009 www. mtcsoft. es 16

PHASE 1 PHASE 2 Symbol identification Grammatical Analysis PHASE 3 An example Syntactic Analysis After Phase 3: Syntactic functions Madrid – April, 2009 www. mtcsoft. es 17

The Dictionary In collaboration with: Facultad de Traducción e Interpretación Granada University Dictionary Main Statistics: 7 professors 24 - 106. 634 English words students 4 months - 31. 417 Fixed expressions - 9. 587 Open expressions - 20. 268 Idiomatic verbs - Grammar functions: § 12. 558 Adjectives § 4. 388 Adverbs § 47. 150 Substantives § 8. 558 Verbs (2. 590 phrasal verbs) § § 2. 136 acronyms 2. 217 incidental phrases Madrid – April, 2009 www. mtcsoft. es 18 Págin

The Dictionary Ø 10 Dictionaries Ø Every one has its type of record Ø Every word has its data Madrid – April, 2009 www. mtcsoft. es 19

State of the “Beta” 1 1 2 3 Parser + Tagger = Parsalyser 2 Key words Indexes 90% 15% 3 Advanced Search Interface Madrid – April, 2009 www. mtcsoft. es 15% 20

2 Indexes Key words Indexes (Simplified example. In the real table, Words are codified using the dictionary) Word Grammar Syntactic Sentence File Function Tom Cruise 1 1 123 Tom Cruise 1 1 3 234 Tom Cruise 1 1 12 345 Tom Cruise 1 3 2 12 Tom Cruise 1 3 23 134 Tom Cruise 1 3 2 567 Tom Cruise 1 3 12 1. 234 Tom Cruise 1 3 33 2. 345 buy 1 1 3 1. 233 buy 1 1 124 3. 432 buy 1 2 44 4. 333 buy 3 2 3 234 buy 3 2 23 134 buy 3 2 345 buy 3 3 34 4. 322 house 1 2 3 234 house 1 2 21 1. 456 house 1 3 234 house 3 3 12 4. 444 Madrid – April, 2009 File: 234 Sentence: 3 1 Syntactic function www. mtcsoft. es 3 Substantive Adjective Verb 1 Grammar function 2 2 3 Subject Verb Direct Object 21

3 Advanced Search Interface User Interface There is a lot of work done in this area. A first approach: Ø We has not developed the interface. Ø Just a simple window to test the system. Ø Even so, it is Tom Cruise bought house very powerful Ø X Only texts with the words related Just a simple “click” to filter most of the results Madrid – April, 2009 www. mtcsoft. es 22

3 Advanced Search Interface User Interface Advanced searches: For “advanced” users, a more powerful interface Ø Ø Analyses the sentence Proposes grammar functions Proposes syntactic functions Allows declined forms for every word It works! Madrid – April, 2009 www. mtcsoft. es 23

Other uses Plain texts Germany is supporting the development of infrastructure (especially energy and water supplies), promotion of the economy and employment, advising the PISG on the restructuring of administrative structures in the education and vocational training area and the improvement of the general economic climate. Search Engines Semantic Web Data Mining Statistical Analysis … Madrid – April, 2009 www. mtcsoft. es 24

Other uses Plain texts Tagged text Germany is supporting the development of infrastructure (especially energy and water supplies), promotion of the economy and employment, advising the PISG on the restructuring of administrative structures in the education and vocational training area and the improvement of the general economic climate. Germany is supporting the development of infrastructure (especially energy and water supplies), promotion of the economy and employment, advising the PISG on the restructuring of administrative structures in the education and vocational training area and the improvement of the general economic climate. The parser + tagger can be used for a lot of Internet related tasks Search Engines Semantic Web Data Mining Statistical Analysis … Madrid – April, 2009 www. mtcsoft. es 25

A new Standard? Tagged text Germany is supporting the development of infrastructure (especially energy and water supplies), promotion of the economy and employment, advising the PISG on the restructuring of administrative structures in the education and vocational training area and the improvement of the general economic climate. The question is: Ø Why use “plain texts” to start the work with, If you can use “enriched texts”? Ø Change the “raw material” at the beginning. Ø These “enriched texts” are: Ø Ø created automatically (no human work) and have an “universal” format (no standard to discuss) Madrid – April, 2009 www. mtcsoft. es 26

Thank you for attending l. Fernando Moreno-Torres ldireccion@mtcsoft. es l l+34. 609. 575. 000 l+34. 958. 215. 280 l. C/ Concepción, 47 l 18009 – Granada (Spain) www. mtcsoft. es