Question Answering (QA) Lecture 2 © Johan Bos April 2008 Lecture 1 • • What is QA? Query Log Analysis Challenges in QA History of QA System Architecture Methods System Evaluation State-of-the-art • Question Analysis • Answer Typing • Background Knowledge Lecture 3 • • • Query Generation Document Analysis Semantic Indexing Answer Extraction Selection and Ranking
© Johan Bos April 2008 What is Question Answering? ?
What is Question Answering? © Johan Bos April 2008 • Questions in natural language, not simple queries of keywords! • Answers, not documents!
What is Question Answering? • Questions, no queries! • Answers, not documents! © Johan Bos April 2008 Q: Where did Olof Palme die? A:
What is Question Answering? • Questions, no queries! • Answers, not documents! © Johan Bos April 2008 Q: Where did Olof Palme die? A: In Stockholm.
QA versus IR • Traditional method for information access: IR (Information Retrieval) © Johan Bos April 2008 – Think of IR as finding the “right book in a library” – Think of QA as a “librarian giving you the book and opening it on the page with the information you’re looking for”
QA versus IE • Traditional method for information access: IE (Information Extraction) – Think of IE as finding answers to a predefined question (i. e. , a template) © Johan Bos April 2008 – Think of QA as asking any question you like
Why do we need QA? © Johan Bos April 2008 • Information overload problem • Accessing information using traditional methods such as IR and IE are limited • QA increasingly important because: – Size of available information grows – There is duplicate information – There is false information – More and more “computer illiterates” accessing electronically stored information
Information Avalanche • Available information is growing*: – 1999: 250 MB pp for each person on earth – 2002: 800 MB pp for each person on earth © Johan Bos April 2008 • People want specific information • People ask questions (rather than queries) * source: M. de Rijke 2005
Information Pinpointing © Johan Bos April 2008 For example --- information required: Average number of car accidents per year in Sweden. Two ways of getting this information: - Ask Google or a similar search engine (good luck!) - Ask a QA system the question: What’s the rate of car accidents in Sweden?
Excite Query Log © Johan Bos April 2008 • Search Engine Log with 2, 477, 283 queries • Collected at 20 December 1999 • Around 15% natural language questions
© Johan Bos April 2008 Excite Query Log 090039 090039 090039 090039 090039 82 F 30745 D 2 BF 5 C 2 C 9252 B 344 DB 1 B 7047 E 6 D 3 DCFA 0 A 39 BA 3 E AFCD 0 FD 4 A 65 D 34 C 3 F 54 BC 573 FF 3 C 94 B 1 3 DF 4 E 9 B 0 AFF 6 B 808 009 CD 46686 D 4 A 1 FD F 26 E 5 D 31 D 804 ED 98 D 87 CE 5 C 149126 B 4 B 91255 EEF 958730 CE 03 D 685 B 3 F 001 A 858 F 54 BC 573 FF 3 C 94 B 1 DAF 1 CD 4 E 35 BB 6 D 98 BDF 5 F 884 F 42 DB 05 F 1246 BA 311 FD 5 AF 65 9 F 0 FF 534 C 84 D 7873 E 00 F 9 C 2509761702 70 B 9 F 6 D 94 EB 08 EB 7 5 AE 9 D 0 A 42 F 05 FDE 1 golf www. Odcgov/cia/Publication/Fatbook/index www. ufallfahrzeug. de montogery ward What is the website for the newspaper in Scranton, PA? Universities in Wellington pantyfreek carman ministries Where can I find information about the television show Felicit how can i convert liquid audio files to wav files? "CD duplication software" What does the FDA say is its mandate? ebony ayres+homepage hacking VP Instant Message readers messanger thiazolidisediones chicago newspapers yahoo New Century Professions, Inc. What presidents were born in texas
Question Types © Johan Bos April 2008 • Wh-Questions: • Yes/No-Questions: 342, 075 23, 882
© Johan Bos April 2008 Question Types How is the U. N. funded? Was Don lying about the shooting? Who invened the rangefinder? what exactly is a hydraulic system? When was Fisher Price started? Where can i find exams on the internet about database management systems course with solu Where can I find yesturday's weather? What is the probability of extraterrestrial life existing in the Universe? What vacation rentals are available in the pacific Northwest? Who was president Grant's private secretary? Where can I get a recipe for sloppy joes? What are the times of sunrises and sunsets in various cities in the US? pros and cons of professional managers in government? Where can I find information about FLSA? where can I find details of property in Scotland? Who sings the song"Ice, Ice Baby"? what percentage of schools have internet access? When were gingerbread houses invented? how do I find out the purchase price of a house that sold in California? Are cmputer chips made from sand? what is lupus? where can i find how to write a resume? WHERE CAN I FIND ON-LINE GREETING CARDS?
© Johan Bos April 2008 Wh-Question Types
© Johan Bos April 2008 How-Question Types
© Johan Bos April 2008 How-Question Types How How How How How How How do I register a domain name? do genetics affect childrens diseases? do I stop a Bios I/O COnflict? does the Greenhouse effect work many hours do you practice a week for basketball? do I learn Unix? were the Christians persecuted? maney bicycle accidents in US by year? long does a DUI stay on your record in California do you make a curcitboard? does airplane wing shape affect flight? were the Christians persecuted? do you play the saxophone? do I start a travel writing career? do I stop a COnflict I/O Ports? many bicycle accidents in US by year? much weight can you lose per week? are you? do I shop online? do you take apart an alkaline battery to manufacture? does dry ice work? tall is the average man do I make a pecan pie can I find someone's Yahoo e-mail address?
Natural search © Johan Bos April 2008 • Google gets about 200 million searches per day • That would mean about 30 million natural language searches
Question Answering (QA) © Johan Bos April 2008 Lecture 1 • • Ø • • • What is QA? Query Log Analysis Challenges in QA History of QA System Architecture Methods System Evaluation State-of-the-art
© Johan Bos April 2008 QA – The Seven Challenges 1) Accounting for flexibility in natural language questions and answers 2) Dealing with ill-formed input 3) Merging across documents 4) Acquiring relevant background knowledge 5) Semantic analysis 6) User modelling 7) Answer precision
Challenge 1 (flexibility) • Variation in questions: – – What is the population of Rome? How many people live in Rome? What’s the size of Rome? How many inhabitants does Rome have? © Johan Bos April 2008 • Variation in answers: …is estimated at 2. 5 million residents… … current population of Rome is 2817000… …Rome housed over 1 million inhabitants…
Challenge 2 (ill-formed input) © Johan Bos April 2008 • Just look at the Excite query log! Who invened the rangefinder? what exactly is a hydraulic system? When was Fisher Price started? Where can I find yesturday's weather? What is the probability of extraterrestrial life existing in the Universe? What vacation rentals are available in the pacific Northwest? Who was president Grant's private secretary? Where can I get a recipe for sloppy joes? What are the times of sunrises and sunsets in various cities in the US? pros and cons of professional managers in government? Where can I find information about FLSA? where can I find details of property in Scotland? Who sings the song"Ice, Ice Baby"? what percentage of schools have internet access? When were gingerbread houses invented? how do I find out the purchase price of a house that sold in California? Are cmputer chips made from sand? what is lupus? where can i find how to write a resume? WHERE CAN I FIND ON-LINE GREETING CARDS?
Challenge 2 (ill-formed input) © Johan Bos April 2008 • Just look at the Excite query log! Who invened the rangefinder? what exactly is a hydraulic system? When was Fisher Price started? Where can I find yesturday's weather? What is the probability of extraterrestrial life existing in the Universe? What vacation rentals are available in the pacific Northwest? Who was president Grant's private secretary? Where can I get a recipe for sloppy joes? What are the times of sunrises and sunsets in various cities in the US? pros and cons of professional managers in government? Where can I find information about FLSA? where can I find details of property in Scotland? Who sings the song"Ice, Ice Baby"? what percentage of schools have internet access? When were gingerbread houses invented? how do I find out the purchase price of a house that sold in California? Are cmputer chips made from sand? what is lupus? where can i find how to write a resume? WHERE CAN I FIND ON-LINE GREETING CARDS?
Challenge 3 (merging) • Answers could be spread across different documents • Example 1: – Which European countries produce wine? [Document A contains information about Italy, and document B about France] © Johan Bos April 2008 • Example 2: – What does Bill Clinton’s wife do for a living? [Document A explains that Bill Clinton’s wife is Hillary Clinton, and Document B tells us that she’s a politician]
Challenge 4 (knowledge) Q: When was NATO established? A: NATO launched its first attack against Yugoslavia on March 24. © Johan Bos April 2008 Word. Net: launch=establish
Challenge 4 (knowledge) Q: Bing Crosby. What was his profession? A: Crosby sang her first song, Starlight. She © Johan Bos April 2008 scribbled notes for the song on the back of a menu at a New York jazz bar in 1931. . Word. Net: bar hyponym of profession
Challenge 5 (semantics) Q: Where did Ricky Williams, American football player, grow up? A: Texas running back Ricky Williams is from © Johan Bos April 2008 California. Those who follow the Longhorns don't like Williams any less because he didn't grow up in Texas.
Challenge 6 (user modelling) Q: Where is the Taj Mahal? © Johan Bos April 2008 A: The Taj Mahal is a mausoleum located in Agra, India, that was built under Mughal Emperor Shah Jahan in memory of his favorite wife, Mumtaz Mahal.
Challenge 6 (user modelling) Q: Where is the Taj Mahal? © Johan Bos April 2008 A: A: The Taj Mahal is a mausoleum located in Agra, India, that was built under Mughal Emperor Shah Jahan in memory of his favorite wife, Mumtaz Mahal. The Taj Mahal first opened its door to a receptive but cautious public back in 1964. Being the first restaurant of its kind in Stevenage, our main problems were initially to get people to try foods that they had never tried before.
Challenge 7 (precision) • Where is 3 M based? © Johan Bos April 2008 – In Maplewood, Minn. – In Minnesota – In the U. S. – In Maplewood, Minn. , USA
Challenge 7 (precision) • Where did Franz Kafka die? © Johan Bos April 2008 – In his bed – In a sanatorium – In Kierling – Near Vienna – In Austria
Question Answering (QA) © Johan Bos April 2008 Lecture 1 • • • Ø • • What is QA? Query Log Analysis Challenges in QA History of QA System Architecture Methods System Evaluation State-of-the-art
© Johan Bos April 2008 History of QA (de Rijke & Webber 2003) • QA is by no means a new area! • Simmons (1965) reviews 15 implemented and working systems • Many ingredients of today’s QA systems are rooted in these early approaches • Database oriented systems, domain independent, as opposed to today’s systems that work on large sets of unstructured texts
© Johan Bos April 2008 Examples of early QA systems • BASEBALL (Green et al. 1963) Answers English questions about scores, locations and dates of baseball games • LUNAR (Woods 1977) Accesses chemical data on lunar material compiled during the Apollo missions • PHLIQA 1 (Scha et al. 1980) Answers short questions against a database of computer installations in Europe
Recent work in QA • Since the 1990 s research in QA has by and large focused on open-domain applications © Johan Bos April 2008 • Recently interest in restricted-domain QA has increased, in particular in commercial applications – Banking – Medical applications – Entertainment
Open vs restricted domain • QA in open domain – General questions – Large collection of documents (web) – Answer redundancy © Johan Bos April 2008 • QA in restricted domain – Very specific questions – Relatively small set of documents – Answers might appear only once in document collection
Question Answering (QA) © Johan Bos April 2008 Lecture 1 • • Ø • • • What is QA? Query Log Analysis Challenges in QA History of QA System Architecture Methods System Evaluation State-of-the-art
Architecture of a QA system corpus question Question Answering System © Johan Bos April 2008 answers
Architecture of a QA system corpus question Question Analysis query documents/passages expected answer-type question representation © Johan Bos April 2008 answers IR Answer Extraction Document Analysis passage representation
Question Analysis • Input: Natural Language Question © Johan Bos April 2008 • Output: Expected Answer Type (Formal) Representation of Question • Techniques used: Machine learning syntactic & semantic parsing
Document Analysis • Input: Documents or Passages © Johan Bos April 2008 • Output: (Formal) Representation of Passages that might contain the answer • Techniques used: Tokenisation, Named Entity Recognition, Parsing
Answer Extraction • Input: Expected Answer Type Question (formal representation) Passages (formal representation) © Johan Bos April 2008 • Output: Ranked list of answers • Techniques used: Matching, Re-ranking, Validation
Example Run corpus question Question Analysis query documents/passages answer-type question representation © Johan Bos April 2008 answers IR Answer Extraction Document Analysis passage representation
Example Run How long is the river Thames? corpus question Question Analysis query documents/passages answer-type question representation © Johan Bos April 2008 answers IR Answer Extraction Document Analysis passage representation
Example Run length river thames corpus question Question Analysis query documents/passages answer-type question representation © Johan Bos April 2008 answers IR Answer Extraction Document Analysis passage representation
Example Run corpus question Question Analysis MEASURE query documents/passages answer-type question representation © Johan Bos April 2008 answers IR Answer Extraction Document Analysis passage representation
Example Run corpus question Question Analysis query answer(x) &documents/passages length(y, x) & river(y) & named(y, thames) answer-type question representation © Johan Bos April 2008 answers IR Answer Extraction Document Analysis passage representation
Example Run A: NYT 199802 -31 B: APW 199805 -12 C: NYT 200011 -07 corpus question Question Analysis query documents/passages answer-type question representation © Johan Bos April 2008 answers IR Answer Extraction Document Analysis passage representation
Example Run A: 30(u) & mile(u) & length(v, u) & river(y) query B: 60(z) & centimeter(z) & Question question height(v, z) & dog(z) Analysis C: 230(u) & kilometer(u) & length(x, u) answer-type & river(x) question representation © Johan Bos April 2008 answers Answer Extraction corpus IR documents/passages Document Analysis passage representation
Example Run corpus question Question Analysis query documents/passages C: 230 kilometer answer-type A: 30 miles B: 60 centimeter question representation © Johan Bos April 2008 answers IR Answer Extraction Document Analysis passage representation
Question Answering (QA) © Johan Bos April 2008 Lecture 1 • • • Ø • • What is QA? Query Log Analysis Challenges in QA History of QA System Architecture Methods System Evaluation State-of-the-art
Method 1: Guessing © Johan Bos April 2008 • Limit the amount of (deep) natural language processing tools • Usually based on simple pattern matching • Use shallow features, such as frequency, to guess an answer
Methods in QA • Viewed from a distance, roughly three different methods can be distinguished: © Johan Bos April 2008 – Guessing – Googling – GOFAI (good old-fashioned AI)
Guessing example Q: What country is Berlin in? © Johan Bos April 2008 Ties between Iran and Germany strained after a Berlin court verdict ruled on April 10 A: that Iran's top leaders were behind the assassination of four exiled Kurdish opposition leaders in a Berlin restaurant.
Guessing example Q: What country is Berlin in? © Johan Bos April 2008 Ties between Iran and Germany strained after a Berlin court verdict ruled on April 10 A: that Iran's top leaders were behind the assassination of four exiled Kurdish opposition leaders in a Berlin restaurant.
Another guessing example Q: Port Arthur Massacre. © Johan Bos April 2008 What was the killer's nationality? A nation asks why, the portrait of a lone gunman, Martin Bryant, should have no A: reason to be a killer. The man arrested after the Port Arthur massacre should be a wealthy man, only four years ago he inherited more than 500, 000 Australian dollars 375, 000 US dollars from one of the heirs of George Adams's great Tattersalls fortune.
Method 2: Googling © Johan Bos April 2008 • Use other, larger, corpora to find the answer (such as the internet). • Use Google tricks.
Googling example Q: What is the state bird of Alaska? A: Google search: © Johan Bos April 2008 "The state bird of Alaska is the * "
Googling example Q: What is the state bird of Alaska? © Johan Bos April 2008 A: Google results: The state bird of Alaska is the ptarmigan. The state bird of Alaska is the willow ptarmigan.
Googling example Q: What is the state bird of Alaska? © Johan Bos April 2008 A: Google results: The state bird of Alaska is the ptarmigan. The state bird of Alaska is the willow ptarmigan. The state bird of Alaska is the mosquito. The state bird of Alaska is the Mosquito.
Method 3: GOFAI © Johan Bos April 2008 • Using background knowledge to give support for an answer • Use (logical) inference when possible
GOFAI example © Johan Bos April 2008 Q: Where did Olof Palme die? A: Stockholm.
GOFAI example Q: Where did Olof Palme die? © Johan Bos April 2008 A: ………. given clause #13: (wt=2) 69 [hyper, 47, 16] vehicle($c 3). given clause #14: (wt=2) 71 [hyper, 50, 16] vehicle($c 4). given clause #15: (wt=2) 74 [hyper, 52, 18] building($c 5). given clause #16: (wt=3) 51 [] have(vincent, $c 4). given clause #17: (wt=2) 76 [hyper, 55, 18] building($c 6). given clause #18: (wt=2) 78 [hyper, 61, 16] vehicle($c 6). given clause #19: (wt=2) 80 [hyper, 65, 10] organism($c 1). given clause #20: (wt=2) 84 [hyper, 67, 10] organism($c 2). given clause #21: (wt=3) 53 [] die(palme, $c 5). given clause #22: (wt=2) 86 [hyper, 69, 15] instrument($c 3). given clause #23: (wt=2) 88 [hyper, 71, 15] instrument($c 4). given clause #24: (wt=2) 90 [hyper, 74, 7] artifact($c 5). given clause #25: (wt=2) 94 [hyper, 76, 7] artifact($c 6). given clause #26: (wt=3) 56 [] $c 7=$c 6. given clause #27: (wt=2) 96 [hyper, 78, 15] stockholm($c 6). -----> EMPTY CLAUSE at 0. 01 sec ----> 113 [hyper, 96, 24, 76] $F.
GOFAI example (future? ) Q: Where did Olof Palme die? © Johan Bos April 2008 A: 1. You are looking for location. 2. I know that Stockholm is a city. 3. Every city is a location. 4. If x is shot to death then x died. 5. I found the following evidence in document APW 20000227. 0124: "In 1986, Swedish Prime Minister Olof Palme was shot to death in central Stockholm. “ 6. Hence Stockholm is the answer.
Question Answering (QA) © Johan Bos April 2008 Lecture 1 • • • Ø • What is QA? Query Log Analysis Challenges in QA History of QA System Architecture Methods System Evaluation State-of-the-art
Evaluating QA systems © Johan Bos April 2008 • International evaluation campaigns for QA systems (open domain QA): – TREC (Text Retrieval Conference) http: //trec. nist. gov/ – TAC (Text Analysis Conference) – CLEF (Cross Language Evaluation Forum) http: //clef-qa. itc. it/ – NTCIR (NII Test Collection for IR Systems) http: //www. slt. atr. jp/CLQA/
TREC-QA (organised by NIST) • Annual event, started in 1999 • Difficulty of the QA task increased over the years: – 1999: Answers in snippets, ranked list of answers; – 2005: Exact answers, only one answer. © Johan Bos April 2008 • Three types of questions: – Factoid questions – List questions – Definition questions
QA@CLEF • CLEF is the “European edition” of TREC • Monolingual (non-English) QA – Bulgarian (BG), German (DE), Spanish (ES), Finnish (FI), French (FR), Italian (IT), Dutch (NL), Portuguese (PT) • Cross-Lingual QA © Johan Bos April 2008 – Questions posed in source language, answer searched in documents of target language – All combinations possible
Open-Domain Question Answering • TREC (1999 - 2007) – Acquint corpus – Factoid and Definition questions • TAC (2008 - …. ) © Johan Bos April 2008 – Blog 06 corpus – List and opinion questions
TREC-type questions • Factoid questions – Where is the Taj Mahal? • List questions – What actors have played Tevye in “Fiddler on the Roof? ” © Johan Bos April 2008 • Definition/biographical questions – What is a golden parachute? – Who is Vlad the Impaler?
What is a correct answer? • Example Factoid Question – When did Franz Kafka die? © Johan Bos April 2008 • Possible Answers: – Kafka died in 1923. – Kafka died in 1924. – Kafka died on June 3, 1924 from complications related to Tuberculosis. – Ernest Watz was born June 3, 1924. – Kafka died on June 3, 1924.
What is a correct answer? • Example Factoid Question – When did Franz Kafka die? © Johan Bos April 2008 • Possible Answers: Incorrect – Kafka died in 1923. – Kafka died in 1924. – Kafka died on June 3, 1924 from complications related to Tuberculosis. – Ernest Watz was born June 3, 1924. – Kafka died on June 3, 1924.
What is a correct answer? • Example Factoid Question – When did Franz Kafka die? © Johan Bos April 2008 • Possible Answers: Inexact (under-informative) – Kafka died in 1923. – Kafka died in 1924. – Kafka died on June 3, 1924 from complications related to Tuberculosis. – Ernest Watz was born June 3, 1924. – Kafka died on June 3, 1924.
What is a correct answer? • Example Question – When did Franz Kafka die? © Johan Bos April 2008 • Possible Answers: Inexact (over-informative) – Kafka died in 1923. – Kafka died in 1924. – Kafka died on June 3, 1924 from complications related to Tuberculosis. – Ernest Watz was born June 3, 1924. – Kafka died on June 3, 1924.
What is a correct answer? • Example Question – When did Franz Kafka die? © Johan Bos April 2008 • Possible Answers: – Kafka died in 1923. Unsupported – Kafka died in 1924. – Kafka died on June 3, 1924 from complications related to Tuberculosis. – Ernest Watz was born June 3, 1924. – Kafka died on June 3, 1924.
What is a correct answer? • Example Question – When did Franz Kafka die? © Johan Bos April 2008 • Possible Answers: Correct – Kafka died in 1923. – Kafka died in 1924. – Kafka died on June 3, 1924 from complications related to Tuberculosis. – Ernest Watz was born June 3, 1924. – Kafka died on June 3, 1924.
Answer Accuracy © Johan Bos April 2008 # correct answers Answer Accuracy = -------------# questions
Correct answers to list questions Example List Question Which European countries produce wine? System A: © Johan Bos April 2008 France Italy System B: Scotland France Germany Italy Spain Iceland Greece the Netherlands Japan Turkey Estonia
Evaluation metrics for list questions • Precision (P): # answers judged correct & distinct P = -----------------------# answers returned • Recall (R): © Johan Bos April 2008 • F-Score (F): # answers judged correct & distinct R = ------------------------# total answers 2*P*R F = ------P+R
Correct answers to list questions Example List Question Which European countries produce wine? System A: © Johan Bos April 2008 France Italy P = 1. 00 R = 0. 25 F = 0. 40 System B: Scotland France Germany Italy Spain Iceland Greece the Netherlands P = 0. 64 Japan R = 0. 88 Turkey F = 0. 74 Estonia
Other evaluation metrics System A: Ranked answers (Accuracy = 0. 2) Q 1 Q 2 Q 3 Q 4 Q 6 Q 7 Q 8 Q 9 …. Qn A 1 W W C W W W …. W A 2 W W W W …. W A 3 W W W W …. W A 4 W W W W …. W A 5 W C W W …. W System B: Ranked answers (Accuracy = 0. 1) Q 2 Q 3 Q 4 Q 6 Q 7 Q 8 Q 9 …. Qn A 1 W W C W W W …. W A 2 © Johan Bos April 2008 Q 1 C W W C C W …. C A 3 W C W W W …. W A 4 W W W C W W …. W A 5 W W W W …. W
Mean Reciprocal Rank (MRR) • Score for an individual question: – The reciprocal of the rank at which the first correct answer is returned – 0 if no correct response is returned • The score for a run: © Johan Bos April 2008 – Mean over the set of questions in the test
MRR in action System A: MRR = (. 2+1+1+. 2)/10 = 0. 24 Q 1 Q 2 Q 3 Q 4 Q 6 Q 7 Q 8 Q 9 …. Qn A 1 W W C W W W …. W A 2 W W W W …. W A 3 W W W W …. W A 4 W W W W …. W A 5 W C W W …. W System B: MRR = (. 5+. 33+. 5+. 25+1+. 5+. 5)/10=0. 42 Q 3 Q 4 Q 6 Q 7 Q 8 Q 9 …. Qn A 1 W W C W W W …. W A 2 © Johan Bos April 2008 Q 1 C W W C C W …. C A 3 W C W W W …. W A 4 W W W C W W …. W A 5 W W W W …. W
Question Answering (QA) © Johan Bos April 2008 Lecture 1 • • Ø What is QA? Query Log Analysis Challenges in QA History of QA System Architecture Methods System Evaluation State-of-the-art
State of the art © Johan Bos April 2008 • QA is hard • Best systems use extensive answer typing and semantics • Systems complex
Unsolved problems • Yes/No-questions Can you give me the names of Mr Jones? Are debit cards expensive? © Johan Bos April 2008 Are growing stock piles an indication of trouble in a company?
Unsolved problems • Conditional questions Can I get a mortgage if the house needs repairs? • Why-questions Why did Obama change his name? © Johan Bos April 2008 • How-questions How did Hillary Clinton win 58% in Rhode Island?
Unsolved problems • Comparison-questions Is X better than Y? Of the two governments involved over Kinmen, which has air superiority? © Johan Bos April 2008 What is the difference between X and Y?
Unsolved problems • Questions with negation Which European countries do not charge VAT tax on gold? Are there hearing aids that do not have to go inside your ear? © Johan Bos April 2008 My computer is set up to reject cookies. What will I not be able to do on your website?
Unsolved problems • Multiple questions in disguise © Johan Bos April 2008 Where and when was Franz Kafka born? Who ordered what? Where is each department located?
Unsolved problems • Questions asking for opinions What do people like about IKEA? What do young people like about IKEA? © Johan Bos April 2008 What do women like about IKEA?
Question Answering (QA) Lecture 2 © Johan Bos April 2008 Lecture 1 • • What is QA? Query Log Analysis Challenges in QA History of QA System Architecture Methods System Evaluation State-of-the-art • Question Analysis • Answer Typing • Background Knowledge Lecture 3 • • • Query Generation Document Analysis Semantic Indexing Answer Extraction Selection and Ranking
Architecture of PRONTO parsing question answer ccg answer reranking boxing drs Word. Net Nom. Lex answer selection knowledge © Johan Bos April 2008 query answer typing Indri answer extraction Indexed Documents
Lecture 2 parsing question answer ccg answer reranking boxing drs Word. Net Nom. Lex answer selection knowledge © Johan Bos April 2008 query answer typing Indri answer extraction Indexed Documents
Lecture 3 parsing question answer ccg answer reranking boxing drs Word. Net Nom. Lex answer selection knowledge © Johan Bos April 2008 query answer typing Indri answer extraction Indexed Documents
Скачать презентацию Question Answering QA Lecture 2 Johan Bos
8037f62f1b2da1b59adce550c88a76c5.ppt