Indexing and Searching File Structures Modern Information Retrieval

Indexing and Searching (File Structures) Modern Information Retrieval (Chapter 8) With G. Navarro 1

File Struces n n n Inverted Files Signatures PAT Trees Sequential Searching Compression 2

Inverted Files Information Retrieval: Data Structures and Algorithms (Chapters 3) W. B. Frakes and R. Baeza-Yates (Eds. ) 1992. 3

Inverted Files n Characteristics Ø n A word-oriented mechanism based on sorted list of keywords, with each keyword having links to the documents containing that keyword. Preprocessing Ø Ø Each document is assigned a list of keywords or attributes. Each keyword (attribute) is associated with relevance weights. 4

Inversion of Word List 1. The input text is parsed into a list of words along with their location in the text. (time and storage consuming operation) 2. This list is inverted from a list of terms in location order to a list of terms in alphabetical order. 3. Add term weights, or reorganize or compress the files. 5

Inversion of Word List 6

Structure and Construction n Structure (split the index into two files) Ø Ø n Vocabulary: O(nb) according to Heaps’ Law Occurrences : depends on the addressing granularity Construction Ø Ø The vocabulary is stored in lexicographical order and points to posting list. Posting file：the lists of occurrences are stored contiguously 7

Dictionary and Postings File (document #, frequency) 8

Vocabulary and Posting File 9

Structures used in Inverted Files n Vocabulary Ø Ø Ø n Sorted Arrays Hashing Structures Keyword Trees: Tries (digital search trees) The Search Procedure Ø Ø Ø Vocabulary search Retrieval of occurrences Manipulation of occurrences 10

Size of an Inverted File n Block addressing Ø The text is divided in blocks, and the occurrences point to the blocks instead of full inverted indices where exact occurrences are recorded 11

Cost n Advantage Ø n easy to implement Disadvantage Ø updating the index is expensive 12

Signature Files Information Retrieval: Data Structures and Algorithms (Chapters 4) W. B. Frakes and R. Baeza-Yates (Eds. ) Englewood Cliffs, NJ: Prentice Hall, 1992. 13

Signature Files n Characteristics Ø Ø Word-oriented index structures based on hashing Low overhead (10%~20% over the text size) at the cost of forcing a sequential search over the index Suitable for not very large texts Inverted files outperform signature files for most applications 14

Construction and Search n Word-oriented index structures base on hashing Ø Ø Ø n Maps words to bit masks of B bits Divides the text in blocks of b words each The mask is obtained by bitwise ORing the signatures of all the words in the text block. Search Ø Ø Hash the query to a bit mask W If W & Bi = W, the text block may contain the word 15

Example n Four blocks: Ø This is a text. A text has many words. Words are made from letters. 000101 Ø Ø Ø 110101 Hash(text) = 000101 Hash(many)= 110000 Hash(words)= 100100 Hash(made)= 001100 Hash(letters)= 100001 100100 101101 Block 4: 001100 OR 100001 101101 16

False Drop n n n Assumes that m bits are randomly set in the mask Let a=m/B For b words, the probability that a given bit of the mask is set is 1 -(1 -1/B)bm 1 -e-ba Hence, the probability that the l random bits are also set is Fd =(1 -e-ba)a. B False alarm Fd is minimized for a=ln(2)/b Fd = 2 -m m = B ln 2/b 17

Sequential Signature File (SSF) Assume documents span exactly one logical block the size of document signature F = the size of block signature B 18

Classification of Signature-Based Methods n Horizontal partitioning Grouping similar signatures together and/or providing an index on the signature matrix may result in better-thanlinear search. n Vertical partitioning Storing the signature matrix column-wise improves the response time on the expense of insertion time. 19

Classification of Signature-Based Methods n Vertical partitioning Ø without compression bit-sliced signature files (BSSF, B’SSF) frame sliced (FSSF) generalized frame-sliced (GFSSF) Ø with compression compressed bit slices (CBS) doubly compressed bit slices (DCBS) no-false-drop method (NFD) 20

Classification of Signature-Based Methods n Sequential storage of the signature matrix Ø without compression sequential signature files (SSF) Ø with compression bit-block compression (BC) variable bit-block compression (VBC) n Horizontal partitioning Ø Ø data independent partitioning Gustafson’s method partitioned signature files data dependent partitioning 2 -level signature files 5 -trees 21

Criteria n n n The storage overhead The response time on single word queries The performance on insertion, as well as whether the insertion maintains the “append-only” property 22

Vertical Partitioning n n Idea avoid bringing useless portions of the document signature in main memory Methods Ø Ø store the signature file in a bit-sliced form or in a frame -sliced form store the signature matrix column-wise to improve the response time on the expense of insertion time 23

Bit-Sliced Signature Files (BSSF) Transposed bit matrix (document signature) documents transpose documents represent 24

documents F bit-files search: (1) retrieve m bit-files. e. g. , the word signature of free is 001 000 110 010 the document contains “free”: 3 rd, 7 th, 8 th, 11 th bit are set i. e. , only 3 rd, 7 th, 8 th, 11 th files are examined. (2) “and” these vectors. The 1 s in the result N-bit vector denote the qualifying logical blocks (documents). (3) retrieve text file through pointer file. insertion: require F disk accesses for a new logical block (document), one for each bit-file, but no rewriting 25

Frame-Sliced Signature File (FSSF) n Ideas Ø Ø Ø n Random disk accesses are more expensive than sequential ones Force each word to hash into bit positions that are closer to each other in the document signature these bit files are stored together and can be retrieved with a few random accesses Procedures Ø Ø Ø The document signature (F bits long) is divided into k frames of s consecutive bits each. For each word in the document, one of the k frames will be chosen by a hash function. Using another hash function, the word sets m bits in that frame. 26

Frame-Sliced Signature File (Cont. ) documents frames Each frame will be kept in consecutive disk blocks. 27

FSSF n (Continued) Example (n=2, B=12, s=6, f=2, m=3) Word free text doc. signature n Search Ø Ø n Signature 000000 110010 010110 000000 010110 110010 Only one frame has to be retrieved for a single word query. I. E. , only one random disk access is required. e. g. , search documents that contain the word “free” ->because the word signature of “free” is placed in 2 nd frame, only the 2 nd frame has to be examined. At most k frames have to be scanned for an k word query. Insertion Ø Only f frames have to be accessed instead of F bit-slices. 28

Horizontal Partitioning 1. Goal: group the signatures into sets, partitioning the signature matrix horizontally. 2. Grouping criterion documents 29

Partitioned Signature Files n n n Using a portion of a document signature as a signature key to partition the signature file. All signatures with the same key will be grouped into a so-called “module”. When a query signature arrives, Ø Ø examine its signature key and look for the corresponding modules scan all the signatures within those modules that have been selected 30

Suffix Trees 31

Suffix Trees and Suffix Arrays n n Each position in the text is considered as a text suffix Index points are selected form the text, which point to the beginning of the text positions which will be retrievable 32

33

Suffix arrays n n n The main drawbacks of Suffix Array are its costly construction process. Allow binary searches done by comparing the contents of each pointer. Supra-indices (for large suffix array) 34

35

36

Construction of Suffix Arrays for Large Texts 37

Sequential Searching 38

Algorithms n n n Brute Force Knuth-Morris-Pratt Boyer-Moore Family Shift-Or Suffix Automaton 39

Knuth-Morris-Pratt 40

Boyer-Moore Family 41

Shift-Or 42

Suffix Automaton 43

44

Pattern Matching 45

Algorithms n Searching allowing errors Ø Ø n n Dynamic Programming Automaton Regular Expressions and Extended patterns Pattern Matching Using Indices Ø Ø Inverted files Suffix Trees and Suffix Arrays 46

Dynamic Programming 47

Automaton 48

Regular Expressions 49

Pattern Matching Using Indices n Inverted Files Ø Ø The types of queries such as suffix or substring queries, searching allowing errors and regular expressions, are solved by a sequential search The restriction is to find approximate matches or regular expressions that span many word. 50

Pattern Matching Using Indices n Suffix Trees Ø Suffix trees are able to perform complex searches • Word, prefix, suffix, substring, and Range queries • Regular expressions • Unrestricted approximate string matching Ø Useful in specific areas • Find the longest substring • Find the most common substring of a fixed size 51

Pattern Matching Using Indices n Suffix Arrays Ø Ø Some patterns can be searched directly in the suffix array without simulation the suffix tree Word, prefix, suffix, subword search and range search 52

Compression n Compressed text--Huffman coding Ø Ø n Taking words as symbols Use an alphabet of bytes instead of bits Compressed indices Ø Ø Ø Inverted Files Suffix Trees and Suffix Arrays Signature Files 53