Скачать презентацию The Role of Document Structure in Querying Scoring
380eed88d5dac90b934aac713328b6a7.ppt
- Количество слайдов: 59
The Role of Document Structure in Querying, Scoring and Evaluating XML Full-Text Search Sihem Amer-Yahia AT&T Labs Research - USA Database Department Talk at the Universities of Toronto and Waterloo Nov. 9 th and 10 th, 2005 1
Outline n n n Introduction Querying Scoring Evaluation Open Issues 2
Outline n Introduction ¡ ¡ n n IR vs. Structured Document Retrieval (SDR) XML vs. IR Search Querying Scoring Evaluation Open Issues 3
IR vs SDR n Traditional IR is about finding relevant documents to a user’s information need, e. g. , entire book. n SDR allows users to retrieve document components that are more focussed on their information needs, e. g. , a chapter, a page. • Improve precision • Exploit visual memory 4
Conceptual Model for IR Documents Query Indexing Formulation Document representation Query representation Retrieval function Retrieval results Relevance feedback (Van Rijsbergen 1979) 5
Conceptual Model for SDR Structured documents Content + structure Documents Indexing Query tf, idf, … Formulation Document representation Inverted file + structure index Query representation Retrieval function Retrieval results Matching content + structure Relevance feedback Presentation of related components 6
Conceptual Model for SDR (XML) Structured documents Content + structure XML adopted to represent a mix of structure and text (e. g. , Library of Congress bills, IEEE INEX data collection) tf, idf, … Scoring may capture document structure query languages referring to both content and structure are being developed for accessing XML documents, e. g. XIRQL, NEXI, XQUERY FT additional constraints are imposed on structure index captures in which document component the term occurs (e. g. title, section), Matching content + structure as well as the type of document components (e. g. XML tags) e. g. a chapter and its sections may be retrieved Inverted file + structure index Presentation of related components 7
8
THOMAS: Library of Congress 10
Outline n n n Introduction Querying ¡ search context: XML nodes vs entire document. ¡ search result: XML nodes or newly constructed answers vs entire document. ¡ search expression: keyword search, Boolean operators, proximity distance, scoping, thesaurus, stop words, stemming. ¡ document structure: explicitly specified in query or used in query semantics. Scoring Evaluation Open Issues 11
Languages for XML Search n Keyword search (CO Queries) ¡ n Tag + Keyword search ¡ n book: xml Path Expression + Keyword search (CAS Queries) ¡ n “xml” /book[. /title about “xml db”] XQuery + Complex full-text search ¡ for $b in /book let score $s : = $b ftcontains “xml” && “db” distance 5 12
XRank
XRank
XIRQL
Similar Notion of Results n Nearest Concept Queries ¡ n (Schmidt et al, ICDE 2002) XKSearch ¡ (Xu & Papakonstantinou, SIGMOD 2005) 16
Languages for XML Search n Keyword search (CO Queries) ¡ n Tag + Keyword search ¡ n book: xml Path Expression + Keyword search (CAS Queries) ¡ n “xml” /book[. /title about “xml db”] XQuery + Complex full-text search ¡ for $b in /book let score $s : = $b ftcontains “xml” && “db” distance 5 17
XSearch
Languages for XML Search n Keyword search (CO Queries) ¡ n Tag + Keyword search ¡ n book: xml Path Expression + Keyword search (CAS Queries) ¡ n “xml” /book[. /title about “xml db”] XQuery + Complex full-text search ¡ for $b in /book let score $s : = $b ftcontains “xml” && “db” distance 5 19
XPath 2. 0 n fn: contains($e, string) returns true iff $e contains string //section[fn: contains(. /title, “XML Indexing”)] (W 3 C 2005) 20
XIRQL n Weighted extension to XQL (precursor to XPath) //section[0. 6 ·. //* $cw$ “XQL” + 0. 4 ·. //section $cw$ “syntax”] (Fuhr & Großjohann, SIGIR 2001) 21
XXL n Introduces a similarity operator ~ Select Z From http: //www. myzoos. edu/zoos. html Where zoos. #. zoo As Z and Z. animals. (animal)? . specimen as A and A. species ~ “lion” and A. birthplace. #. country as B and A. region ~ B. content (Theobald & Weikum, EDBT 2002) 22
NEXI n n n Narrowed Extended XPath I INEX Content-and-Structure (CAS) Queries Specifically targeted for content-oriented XML search (i. e. “aboutness”) //article[about(. //title, apple) and about(. //sec, computer)] (Trotman & Sigurbjornsson, INEX 2004) 23
Languages for XML Search n Keyword search (CO Queries) ¡ n Tag + Keyword search ¡ n book: xml Path Expression + Keyword search (CAS Queries) ¡ n “xml” /book[. /title about “xml db”] XQuery + Complex full-text search ¡ for $b in /book let score $s : = $b ftcontains “xml” && “db” distance 5 24
Schema-Free XQuery n Meaningful least common ancestor (mlcas) for $a in doc(“bib. xml”)//author $b in doc(“bib. xml”)//title $c in doc(“bib. xml”)//year where $a/text() = “Mary” and exists mlcas($a, $b, $c) return
Te. XQuery and XQuery FT n n 2003 Fully composable FT primitives. Composable with XPath/XQuery. Based on a formal model. Scoring and ranking on all predicates. Te. XQuery (Cornell U. , AT&T Labs) 2004 + 2005 IBM, Microsoft, Oracle proposals XQuery Full-Text Drafts (Amer-Yahia, Botev, Shanmugasundaram, WWW 2004) (http: //www. w 3. org/TR/xquery-full-text/, W 3 C 2005) 26
FTSelections and FTMatchoptions n FTWord | FTAnd | FTOr | FTNot | FTMild. Not | FTOrder | FTWindow | FTDistance | FTScope | FTTimes | FTSelection (FTMatch. Options)* ¡ ¡ books//title [. ftcontains “usability” case sensitive with thesaurus “synonyms” ] books//abstract [. ftcontains (“usability” || “web-testing”) ] ¡ books//content ftcontains (“usability” && “software”) window at most 3 ordered with stopwords ¡ books//abstract [. ftcontains ((“Utilisation” language “French” with stemming && “. ? site” with wildcards) same sentence] ¡ books//title ftcontains “usability” occurs 4 times && “web-testing” with special characters ¡ books//book/section [. ftcontains books/book/title ]/title 27
![FTScore Clause In any FOR $v [SCORE $s]? IN [FUZZY] Expr order LET …](https://present5.com/presentation/380eed88d5dac90b934aac713328b6a7/image-28.jpg "FTScore Clause In any FOR $v [SCORE $s]? IN [FUZZY] Expr order LET …")
FTScore Clause In any FOR $v [SCORE $s]? IN [FUZZY] Expr order LET … WHERE … ORDER BY … RETURN Example FOR $b SCORE $s in FUZZY /pub/book[. ftcontains “Usability” && “testing” and. /price < 10. 00] ORDER BY $s RETURN $b 28
Gala. Tex Architecture
Outline n Introduction n Querying Scoring Evaluation Open Issues n n n 30
Scoring n Keyword queries and Tag + Keyword queries ¡ ¡ n Path Expression + Keyword queries ¡ n initial term weights per elements with same tag may have same score propagation along document structure. overlapping elements. initial term weights based on paths. XQuery + Complex full-text queries ¡ compute scores for (newly constructed) XML fragments satisfying XQuery (structural, full-text and scalar conditions). 31
Term Weights Article 0. 5 Title 0. 9 XML 0. 4 search 0. 8 Section 1 0. 5 XML ? XML, ? search, ? retrieval 0. 2 Section 2 0. 2 XML 0. 7 retrieval q how to obtain document and collection statistics (e. g. , tf, idf) q how to estimate element scores (frequency, user studies, size)? q which components contribute best to content of Article? q do we need edge weights (e. g. , size, number of children)? q is element size an issue? 32
Score Propagation (XXL) Article Title 0. 9 XML 0. 4 search Section 1 0. 5 XML ? XML, ? search, ? retrieval Section 2 0. 2 XML 0. 7 retrieval q Compute similar terms with relevance score r 1 using an ontology (weighted distance in the ontology graph). q Compute TFIDF of each term for a given element content with relevance score r 2. q Relevance of an element content for a term is r 1*r 2. q Probabilities of conjunctions multiplied (independence assumption) along elements of same path to compute path score. (Theobald & Weikum, EDBT 2002) 33
Overlapping elements Article Title 0. 9 XML 0. 4 search Section 1 0. 5 XML ? XML, ? search, ? retrieval Section 2 0. 2 XML 0. 7 retrieval q Section 1 and article are both relevant to “XML retrieval” q which one to return so that to reduce overlap? q Should the decision be based on user studies, size, types, etc? 34
Controlling Overlap • • Start with a component ranking, elements are re-ranked to control overlap. Retrieval status values (RSV) of those components containing or contained within higher ranking components are iteratively adjusted. 1. Select the highest ranking component. 2. Adjust the RSV of the other components. 3. Repeat steps 1 and 2 until the top m components have been selected. (Clarke, SIGIR 2005) 35
Elem. Rank d 1/3 : Hyperlink edge d 3 : Containment edge w d 1/3 d 2/2 d 1: Probability of following hyperlink d 2: Probability of visiting a subelement d 3: Probability of visiting parent 1 -d 2 -d 3: Probability of random jump (Guo et al, SIGMOD 2003) 36
Scoring n Keyword queries ¡ n Tag + Keyword queries ¡ n compute scores based on tags and keywords. Path Expression + Keyword queries ¡ n compute possibly different scores. compute scores based on paths and keywords. XQuery + Complex full-text queries ¡ compute scores for (newly constructed) XML fragments satisfying XQuery (structural, full-text and scalar conditions). 37
Vector–based Scoring (Juru. XML) n Transform query into (term, path) conditions: article/bm/bibl/bb[about(. , hypercube mesh torus nonnumerical database)] n (term, path)-pairs: hypercube, article/bm/bibl/bb mesh, article/bm/bibl/bb torus, article/bm/bibl/bb nonnumerical, article/bm/bibl/bb database, article/bm/bibl/bb n Modified cosine similarity as retrieval function for vague matching of path conditions. (Mass et al, INEX 2002) 38
Juru. XML Vague Path Matching Modified vector-based cosine similarity Example of length normalization: cr (article/bibl, article/bm/bibl/bb) = 3/6 = 0. 5 39
XML Query Relaxation n Query Tree pattern relaxations: ¡ ¡ ¡ Data Leaf node deletion Edge generalization Subtree promotion author Charles Dickens info edition paperback author Dickens book info edition (paperback) author C. Dickens book edition? info author Dickens edition paperback (Schlieder, EDBT 2002)(Delobel & Rousset, 2002) (Amer-Yahia, Lakshmanan, Pandit, SIGMOD 2004) 40
A Family of Scoring Methods n Twig scoring High quality Expensive computation ¡ ¡ n n Path scoring Binary scoring ¡ book author (Dickens) info edition (paperback) author (Dickens) Low quality Fast computation ¡ info book Query book + book edition (paperback) info edition (paperback) book + book author info edition (Dickens) (paperback) author (Dickens) (Amer-Yahia, Koudas, Marian, Srivastava, Toman, VLDB 2005) 41
Scoring n Keyword queries ¡ n Tag + Keyword queries ¡ n compute scores based on tags and keywords. Path Expression + Keyword queries ¡ ¡ n compute possibly different scores. compute scores based on paths and keywords. Evaluate effectiveness of scoring methods. XQuery + Complex full-text queries ¡ ¡ compute scores for (newly constructed) XML fragments satisfying XQuery (structural, full-text and scalar conditions). compose approximation on structure and on text. 42
Outline n n n Introduction Querying Scoring Evaluation ¡ Formalization of existing XML search languages ¡ Structure-aware evaluation algorithms ¡ Implementation in Gala. Tex Open Issues 43
LOC document fragment
Sample Query on LOC Find action descriptions of bills introduced by “Jefferson” with a committee name containing the words “education” and “workforce” at a distance of no more than 5 words in the text 45
Data model 1 1. 1. 1 n R Node 1. 2. 1 . . . Workforce Education 1 2 1. 1 n tok. Pos 1. 1. 2 tok. Pos 1 1. 2 word position list workforce {1, 3} education Workforce 3 {2} 46
Data model instantiation n One relation per keyword in the document Node 1. 1 1. 2. 1 k 1, k 2, k 1 k 2 2 3 4 5 -redundant storage k 1 ; {6} k 1 ; {2, 4} -each tuple is selfcontained 1. 1. 1 k 1 ; {1} 1. 1 k 1 ; {1, 2, 4} 1 1. 2. 2 k 1 ; {6} 1. 1. 2 Instance 1: Rk 1 1. 2 tok. Pos 1. 2. 2 1 k 1 ; {1, 2, 4, 6} k 1 6 Node tok. Pos Instance 2: scu. Rk 1 1. 2. 2 k 1 ; {6} -no redundant positions 1. 1. 2 k 1 ; {2, 4} 1. 1. 1 k 1 ; {1} -smallest nbr of nodes 47
FT-Algebra and Query Plan 5 × EC ∏node × σdistance({“education”}, {”workforce”}; ≤ 5) R“Jefferson” σordered({“education”, ”workforce”}) × R“education” R“workforce” 48
Join Evaluation Node 1. 2 1. 1. 1. 2. 1 k 1, k 2, k 1 k 2 2 3 4 5 1. 2. 2 k 1 6 k 2 ; {3} k 1 ; {1, 2, 4} 1 1. 1 k 2 ; {3} k 1 ; {2, 4} 1. 1 1. 2 k 2 ; {5} k 1 ; {6} 1. 1. 2 1 tok. Pos k 2 ; {3, 5} k 1 ; {1, 2, 4, 6} × Node tok. Pos 1. 2. 2 k 1 ; {6} 1. 2. 1 k 2 ; {5} 1. 2 k 1 ; {6} 1. 2 k 2 ; {5} 1. 1. 2 k 1 ; {2, 4} 1. 1. 2 k 2 ; {3} 1. 1. 1 k 1 ; {1} 1. 1 k 2 ; {3} 1. 1 k 1 ; {1, 2, 4} 1 k 2 ; {3, 5} 1 k 1 ; {1, 2, 4, 6} 49
Join Evaluation on SCU 1 1. 2 1. 1 1. 2. 1 k 1 1 Node k 1, k 2, k 1 k 2 2 3 4 5 1. 2. 2 tok. Pos 1. 1. 2 k 2 ; {3} k 1 ; {2, 4} k 1 6 × scu. Rk 1 scu. Rk 2 Node tok. Pos 1. 2. 2 k 1 ; {6} 1. 2. 1 k 2 ; {5} 1. 1. 2 k 1 ; {2, 4} 1. 1. 2 k 2 ; {3} 1. 1. 1 k 1 ; {1} 50
Need for LCAs Node 1. 1. 1 1. 2. 1 k 1, k 2, k 1 k 2 2 3 4 5 k 2 ; {3} k 1 ; {2, 4} 1. 1 k 2 ; {3} k 1 ; {1} 1 1. 1. 2 k 2 ; {5} k 1 ; {6} 1. 1. 2 1. 1 tok. Pos 1. 2 1 k 2 ; {3, 5} k 1 ; {1, 2, 4, 6} 1. 2. 2 k 1 6 × scu. Rk 1 scu. Rk 2 Node tok. Pos 1. 2. 2 k 1 ; {6} 1. 2. 1 k 2 ; {5} 1. 1. 2 k 1 ; {2, 4} 1. 1. 2 k 2 ; {3} 1. 1. 1 k 1 ; {1} (Schmidt et al, ICDE 2002)(Li, Yu, Jagadish, VLDB 2003) (Guo et al, SIGMOD 2003)(Xu & Papakonstantinou, SIGMOD 2005) 51
SCU: is LCA enough? σdistance({“k 1”}, {”k 2”}; =2) σordered({“k 2”, ”k 1”}) 1 1. 2 1. 1. 1 k 1, k 2, k 1 k 2 2 3 4 5 pass Node k 1 6 k 2 ; {5} k 1 ; {6} 1. 1. 2. 2 tok. Pos 1. 2. 1 fail k 2 ; {3} k 1 ; {2, 4} 1. 1 k 2 ; {3} k 1 ; {1} 1 k 2 ; {3, 5} k 1 ; {1, 2, 4, 6} × scu. Rk 1 scu. Rk 2 52
SCU: is LCA enough? σdistance({“k 1”}, {”k 2”}; =2) σordered({“k 2”, ”k 1”}) 1 1. 2 1. 1. 1 k 1, k 2, k 1 k 2 2 3 4 5 pass Node k 1 6 k 2 ; {5} k 1 ; {6} 1. 1. 2. 2 tok. Pos 1. 2. 1 fail k 2 ; {3} k 1 ; {2, 4} 1. 1 k 2 ; {3} k 1 ; {1} 1 k 2 ; {3, 5} k 1 ; {1, 2, 4, 6} × scu. Rk 1 scu. Rk 2 53
SCU: is LCA enough? σdistance({“k 1”}, {”k 2”}; =2) 1 σordered({“k 2”, ”k 1”}) 1. 2 1. 1 1. 2. 1 k 1 1 1. 2 k 1, k 2, k 1 k 2 2 3 4 5 1. 2. 2 k 1 6 k 2 ; {5} k 1 ; {6} 1. 1. 2 tok. Pos k 2 ; {3} k 1 ; {2, 4} 1. 1 k 2 ; {3} k 1 ; {1} 1 1. 1. 1 Node fail k 2 ; {3, 5} k 1 ; {1, 2, 4, 6} Does not satisfy ‘ordered’ alone, but it should be an answer! × scu. Rk 1 scu. Rk 2 54
SCU: is LCA enough? σdistance({“k 1”}, {”k 2”}; =2) 1 σordered({“k 2”, ”k 1”}) 1. 2 1. 1. 1 k 1, k 2, k 1 k 2 2 3 4 5 k 1 6 k 2 ; {5} k 1 ; {6} 1. 1. 2. 2 tok. Pos 1. 2. 1 k 1 1 Node k 2 ; {3} k 1 ; {2, 4} 1. 1 k 2 ; {3} k 1 ; {1} 1 fail k 2 ; {3, 5} k 1 ; {1, 2, 4, 6} × scu. Rk 1 scu. Rk 2 55
SCU: position propagation Node 1. 1 k 2 ; {3} k 1 ; {1, 2, 4} 1 1 tok. Pos k 2 ; {3, 5} k 1 ; {1, 2, 4, 6} σdistance({“k 1”}, {”k 2”}; =2) 1. 1. 1. 2. 1 k 1, k 2, k 1 k 2 2 3 4 5 1. 2. 2 k 1 6 pass σordered({“k 2”, ”k 1”}) 1. 2 1. 1 pass Node tok. Pos 1. 2 k 2 ; {5} k 1 ; {6} 1. 1. 2 k 2 ; {3} k 1 ; {2, 4} 1. 1 k 2 ; {3} k 1 ; {1} 1 k 2 ; {3, 5} k 1 ; {1, 2, 4, 6} 56
SCU Summary n Key ideas ¡ R 1×SCUR 2 → find LCA ¡ σSCU(R) → propagation along doc. structure n n if node satisfies σ predicate, output node o/w propagate its tok. Pos to its first ancestor in R Benefit: reduces size of intermediate results Challenge: minimize computation overhead ¡ selections n n ¡ additional column in R for direct access to ancestors TRIE structures joins n record highest ancestor in EC of each node in scu. R and use sort-merge 57
Gala. Tex Architecture: in progress
Open Issues (in no particular order) n n n Difficult research issues in XML retrieval are not ‘just’ about the effective retrieval of XML documents, but also about what and how to evaluate! System architecture: DB on top of IR, IR on top of DB, true merging? Experimental evaluation of scoring methods (INEX). Score-aware algebra for XML for the joint optimization of queries on both structure and text. More details: http: //www. research. att. com/~sihem 59