Скачать презентацию Information Retrieval Search and Mining Introduction 1 Скачать презентацию Information Retrieval Search and Mining Introduction 1

3c4436fc4cb9800e2f6b76b755907bd3.ppt

  • Количество слайдов: 35

Information Retrieval, Search, and Mining Introduction 1 Information Retrieval, Search, and Mining Introduction 1

Course Outline • • Introduction Information Retrieval – – – • • Basic Information Course Outline • • Introduction Information Retrieval – – – • • Basic Information Retrieval Models Indexing, Compression, and Online Search Evaluation Methods Web Search – Challenges – Link Analysis – Other advanced methods Text Mining – Text Categorization – Text Clustering – Recommendation systems – Information extraction 2

References • Christopher D. Manning, Prabhakar Raghavan and Hinrich Schütze, Introduction to Information Retrieval, References • Christopher D. Manning, Prabhakar Raghavan and Hinrich Schütze, Introduction to Information Retrieval, Cambridge University Press. 2008. • Intelligent Information Retrieval and Web Search. A course by Raymond Mooney, U Texas. 2002. – http: //www. cs. utexas. edu/users/mooney/ir-course/ • Standford web search/mining class [Manning, Raghavan] – http: //www. stanford. edu/class/cs 276 b/courseinfo. html • Others: – S. Chakrabarti. 2003. Mining the Web: Discovering Knowledge from Hypertext Data. Morgan Kaufmann. – MG = Managing Gigabytes, by Witten, Moffat, and Bell. MIR = Modern Information Retrieval, by Baeza-Yates and Ribeiro -Neto. 3

Information Retrieval (IR) • The indexing and retrieval of textual documents. • Searching for Information Retrieval (IR) • The indexing and retrieval of textual documents. • Searching for pages on the World Wide Web is the most recent “killer app. ” • Concerned firstly with retrieving relevant documents to a query. • Concerned secondly with retrieving from large sets of documents efficiently. 4

Typical IR Task • Given: – A corpus of textual natural-language documents. – A Typical IR Task • Given: – A corpus of textual natural-language documents. – A user query in the form of a textual string. • Find: – A ranked set of documents that are relevant to the query. 5

IR System Document corpus Query String IR System Ranked Documents 1. Doc 1 2. IR System Document corpus Query String IR System Ranked Documents 1. Doc 1 2. Doc 2 3. Doc 3. . 6

Relevance • Relevance is a subjective judgment and may include: – Being on the Relevance • Relevance is a subjective judgment and may include: – Being on the proper subject. – Being timely (recent information). – Being authoritative (from a trusted source). – Satisfying the goals of the user and his/her intended use of the information (information need). 7

Keyword Search • Simplest notion of relevance is that the query string appears verbatim Keyword Search • Simplest notion of relevance is that the query string appears verbatim in the document. • Slightly less strict notion is that the words in the query appear frequently in the document, in any order (bag of words). 8

Problems with Keywords • May not retrieve relevant documents that include synonymous terms. – Problems with Keywords • May not retrieve relevant documents that include synonymous terms. – “restaurant” vs. “café” – “PRC” vs. “China” • May retrieve irrelevant documents that include ambiguous terms. – “bat” (baseball vs. mammal) – “Apple” (company vs. fruit) – “bit” (unit of data vs. act of eating) 9

Beyond Keywords • We will cover the basics of keyword-based IR, but… • We Beyond Keywords • We will cover the basics of keyword-based IR, but… • We will focus on extensions and recent developments that go beyond keywords. • We will cover the basics of building an efficient IR system, but… • We will focus on basic capabilities and algorithms rather than system’s issues that allow scaling to industrial size databases. 10

Intelligent IR • Taking into account the meaning of the words used. • Taking Intelligent IR • Taking into account the meaning of the words used. • Taking into account the order of words in the query. • Adapting to the user based on direct or indirect feedback. • Taking into account the authority of the source. 11

IR System Architecture User Interface User Need User Feedback Query Ranked Docs Text Operations IR System Architecture User Interface User Need User Feedback Query Ranked Docs Text Operations Logical View Query Operations Searching Ranking Indexing Database Manager Inverted file Index Retrieved Docs Text Database 12

IR System Components • Text Operations forms index words (tokens). – Stopword removal – IR System Components • Text Operations forms index words (tokens). – Stopword removal – Stemming • Indexing constructs an inverted index of word to document pointers. • Searching retrieves documents that contain a given query token from the inverted index. • Ranking scores all retrieved documents according to a relevance metric. 13

IR System Components (continued) • User Interface manages interaction with the user: – Query IR System Components (continued) • User Interface manages interaction with the user: – Query input and document output. – Relevance feedback. – Visualization of results. • Query Operations transform the query to improve retrieval: – Query expansion using a thesaurus. – Query transformation using relevance feedback. 14

Web Search • Application of IR to HTML documents on the World Wide Web. Web Search • Application of IR to HTML documents on the World Wide Web. • Differences: – Must assemble document corpus by spidering the web. – Can exploit the structural layout information in HTML (XML). – Documents change uncontrollably. – Can exploit the link structure of the web. 15

Web Search System Web Spider Document corpus Query String IR System 1. Page 1 Web Search System Web Spider Document corpus Query String IR System 1. Page 1 2. Page 2 3. Page 3. . Ranked Documents 16

Other IR-Related Tasks • • Automated document categorization Information filtering (spam filtering) Information routing Other IR-Related Tasks • • Automated document categorization Information filtering (spam filtering) Information routing Automated document clustering Recommending information or products Information extraction Information integration Question answering 17

Topics: Text mining • “Text mining” is a cover-all marketing term • A lot Topics: Text mining • “Text mining” is a cover-all marketing term • A lot of what we’ve already talked about is actually the bread and butter of text mining: – Text classification, clustering, and retrieval • But we will focus in on some of the higherlevel text applications: – Extracting document metadata – Topic tracking and new story detection – Cross document entity and event coreference – Text summarization

Topics: Information extraction • Getting semantic information out of textual data – Filling the Topics: Information extraction • Getting semantic information out of textual data – Filling the fields of a database record • E. g. , looking at an events web page: – What is the name of the event? – What date/time is it? – How much does it cost to attend • Other applications: resumes, health data, … • A limited but practical form of natural language understanding

Topics: Recommendation systems • Using statistics about the past actions of a group to Topics: Recommendation systems • Using statistics about the past actions of a group to give advice to an individual • E. g. , Amazon book suggestions or Net. Flix movie suggestions • A matrix problem: but now instead of words and documents, it’s users and “documents” • What kinds of methods are used? • Why have recommendation systems become a source of jokes on late night TV? – How might one build better ones?

Topics: XML search • • The nature of semi-structured data Tree models and XML Topics: XML search • • The nature of semi-structured data Tree models and XML Content-oriented XML retrieval Query languages and engines

History of IR • 1960 -70’s: – Initial exploration of text retrieval systems for History of IR • 1960 -70’s: – Initial exploration of text retrieval systems for “small” corpora of scientific abstracts, and law and business documents. – Development of the basic Boolean and vectorspace models of retrieval. – Prof. Salton and his students at Cornell University are the leading researchers in the area. 22

IR History Continued • 1980’s: – Large document database systems, many run by companies: IR History Continued • 1980’s: – Large document database systems, many run by companies: • Lexis-Nexis • Dialog • MEDLINE 23

IR History Continued • 1990’s: – Searching FTPable documents on the Internet • Archie IR History Continued • 1990’s: – Searching FTPable documents on the Internet • Archie • WAIS – Searching the World Wide Web • Lycos • Yahoo • Altavista 24

IR History Continued • 1990’s continued: – Organized Competitions • NIST TREC – Recommender IR History Continued • 1990’s continued: – Organized Competitions • NIST TREC – Recommender Systems • Ringo • Amazon • Net. Perceptions – Automated Text Categorization & Clustering 25

Recent IR History • 2000’s – Link analysis for Web Search • Google • Recent IR History • 2000’s – Link analysis for Web Search • Google • Inktomi • Teoma – Feedback based engine: • Direct. Hit – Automated Information Extraction • Whizbang • Fetch • Burning Glass – Question Answering • TREC Q/A track • Ask Jeeves 26

Recent IR History • 2000’s continued: – Multimedia IR • Image • Video • Recent IR History • 2000’s continued: – Multimedia IR • Image • Video • Audio and music – Cross-Language IR – Document Summarization 27

Related Areas • • • Database Management Library and Information Science Artificial Intelligence Natural Related Areas • • • Database Management Library and Information Science Artificial Intelligence Natural Language Processing Machine Learning 28

Database Management • Focused on structured data stored in relational tables rather than free-form Database Management • Focused on structured data stored in relational tables rather than free-form text. • Focused on efficient processing of welldefined queries in a formal language (SQL). • Clearer semantics for both data and queries. • Recent move towards semi-structured data (XML) brings it closer to IR. 29

Library and Information Science • Focused on the human user aspects of information retrieval Library and Information Science • Focused on the human user aspects of information retrieval (human-computer interaction, user interface, visualization). • Concerned with effective categorization of human knowledge. • Concerned with citation analysis and bibliometrics (structure of information). • Recent work on digital libraries brings it closer to CS & IR. 30

Artificial Intelligence • Focused on the representation of knowledge, reasoning, and intelligent action. • Artificial Intelligence • Focused on the representation of knowledge, reasoning, and intelligent action. • Formalisms for representing knowledge and queries: – First-order Predicate Logic – Bayesian Networks • Recent work on web ontologies and intelligent information agents brings it closer to IR. 31

Natural Language Processing • Focused on the syntactic, semantic, and pragmatic analysis of natural Natural Language Processing • Focused on the syntactic, semantic, and pragmatic analysis of natural language text and discourse. • Ability to analyze syntax (phrase structure) and semantics could allow retrieval based on meaning rather than keywords. 32

Natural Language Processing: IR Directions • Methods for determining the sense of an ambiguous Natural Language Processing: IR Directions • Methods for determining the sense of an ambiguous word based on context (word sense disambiguation). • Methods for identifying specific pieces of information in a document (information extraction). • Methods for answering specific NL questions from document corpora. 33

Machine Learning • Focused on the development of computational systems that improve their performance Machine Learning • Focused on the development of computational systems that improve their performance with experience. • Automated classification of examples based on learning concepts from labeled training examples (supervised learning). • Automated methods for clustering unlabeled examples into meaningful groups (unsupervised learning). 34

Machine Learning: IR Directions • Text Categorization – Automatic hierarchical classification (Yahoo). – Adaptive Machine Learning: IR Directions • Text Categorization – Automatic hierarchical classification (Yahoo). – Adaptive filtering/routing/recommending. – Automated spam filtering. • Text Clustering – Clustering of IR query results. – Automatic formation of hierarchies • Learning for Information Extraction • Text Mining 35