Discovering Unexpected Information from Your Competitor s Web Sites

Discovering Unexpected Information from Your Competitor’s Web Sites Bing Liu, Yiming Ma, Philip S. Yu Héctor A. Villa Martínez 1

Objective of this article The authors presents a system to help find unexpected information in a web site. 2

Searching information in the web Many methods l Keyword based (e. g. Goggle, Yahoo). l Wrapper based (e. g. extract prices). l Web query languages (e. g. extend SQL). l User preference based (specify categories). 3

Searching information in the web Main drawbacks: l Hard to find unexpected information. l Only finds anticipated information. 4

What is unexpected anyways? A piece of information is unexpected if: l it is relevant but unknown, or l it contradicts existing beliefs or expectations relevant interesting (subjective) 5

Summary of the approach U: user web site E: knowledge about the competitor C: competitor web site Compare C vs. U and E to find unexpected information in C. 6

How to compare two web pages Use the vector space representation: l Define a set of p keywords (index terms) K = {k 1, k 2, …, kp). l Represent a document D using a vector D = {w 1, w 2, …, wp} where wi is the weight of the keyword i wi > 0 if keyword i appears in D = 0 otherwise 7

Vector space representation Example: K = {night, day, empire, barbarians, people, house} D = [“Because night is here but the barbarians have not come. And some people arrived from the borders, and said that there are no longer any barbarians. And now what shall become of us without any barbarians? Those people were some kind of solution. ”] D = {1, 0, 0, 3, 2, 0} or normalized to: D = {1/6, 0, 0, 3/6, 2/6, 0} 8

Comparing two web pages Given two web pages in vector space representation, D = {d 1, d 2, …, dn}, and Q = {q 1, q 2, …, qn} the cosine gives a measure of similarity: sim (D, Q) = (D ● Q) / (|D| * |Q|) 9

Comparing two web pages Example: P = {0. 3, 0. 0, 0. 7} Q = {0. 5, 0. 0, 0. 1, 0. 4} R = {0. 0, 0. 5, 0. 0} l Sim (P, P) = (P ● P) / (|P| * |P|) = 1. 0 l Sim (P, Q) = (P ● Q) / (|P| * |Q|) = 0. 87 l Sim (P, R) = (P ● R) / (|P| * |R|) = 0 10

Methods to find unexpected information in a site Let U = (u 1, …, um) the user web site, and C = (c 1, …, cn) the competitor web site: 1. Find the corresponding C page(s) of a U page. 2. Find unexpected terms in a C page. 3. Find unexpected pages in C. 4. Find unexpected concepts in a C page. 5. Find unexpected outgoing links. 11

1. Find the corresponding C page(s) of a U page Given a page ui in U l Compare ui with each page in C. l Order the results in descending order. 12

1. Find the corresponding C page(s) of a U page Example: l Select u 1 l Find sim(u 1, c 1), sim(u 1, c 2), …, sim(u 1, cn) l Order the results in decreasing order: say c 4, c 2, c 8, … etc. Complexity: O(G|C| + |ui||C|) where G = max number of terms in cj 13

2. Find unexpected terms in a C page Given uj and ci measure the unexpectedness of each term tr 1 – (frj / fri) if (frj / fri) ≤ 1 unexp. Trij = 0 otherwise 14

2. Find unexpected terms in a C page Example: keywords = {data, predict, classify, state} uj = {0. 4, 0. 5, 0. 0, 0. 1} ci = {0. 3, 0. 2, 0. 2} unexp. T = {0, 0, 1, 0. 5} Complexity: O(Z) where Z = number of terms in cj 15

3. Find unexpected pages in C 1. 2. 3. 4. 16 5. Combine all pages of U in a single page D u. Combine all pages of C in a single page Dc. Compute the unexpectedness of each term kt in Dc with respect to Du. (Task 2) The unexpectedness of a page Ci is the sum of the unexpectedness of its terms unexp. Pi = (Σunexp. Trcu) / m

3. Find unexpected pages in C Complexity O(Mu|U| + Mc|C|) where Mu is the maximal number of terms in a U page Mc is the maximal number of terms in a C page 17

4. Find unexpected concepts in a C page A concept is a set of keywords that occur together and express the same idea. Example: “information extraction”, “extraction of information”, and “information is extracted” express the same idea “information extraction” 18

4. Find unexpected concepts in a C page Algorithm l Divide the page in sentences. l Use the Apriori algorithm (Agrawal & Srikant) to find association rules of the form X Y with confidence c, where X and Y K, the set of keywords and c is user defined. These association rules are the concepts present in the page. 19

4. Find unexpected concepts in a C page 3. 20 Treating each concept as a term, proceed as Task 2, finding unexpected terms in C.

5. Find unexpected outgoing links Let Lu be the set of outgoing links from U Let Lc be the set of outgoing links from C unexp. L = Lc – Lu 21

Incorporating user knowledge Let E be the user knowledge about his competitor. E is specified as: l Keyword terms l Concepts l Links 22

Incorporating user knowledge The elements in E are incorporated in task 2 thru 5 to find unexpected terms, pages, concepts, and links. Elements in E are ranked low in unexpectedness. 23

System architecture C++/Win 32 l A spider or crawler. Collects information. l Keyword extractor & concepts finder. l Comparison component. Do tasks 1 -5. l User interface. 24

A running example The authors compare its own site with SGI’s Mine. Set data mining site, and not extra knowledge: http: //www. comp. nus. edu. sg/~dm 2 http: //www. sgi. com/software/mineset 25

Results l l 26 Found documentation pages in SGI site. Now the authors are planning to add their own. Found previously unknown pages describing Mine. Set technology. Found some previously unknown Mine. Set features. Found many interesting terms, concepts, and links.

Evaluation The system was further tested with three different organizations: l Travel company l Private school l Diving company 27

Evaluation The users reported the system helped them in: l Focus in interesting pages, terms, and concepts. l Make a more complete analysis of the competitor’s site. l Not missing important information. l Find unexpected things. 28

Efficiency If number of keywords is constant, the algorithms are linear in the number of pages. Tested on a Pentium II 350 PC with 64 MB of RAM #pag sim unexp. Trij unexp. Pj Assoc. mining [1] (143) 0. 0128 0. 0156 0. 0232 0. 0379 [2] (21) 0. 0134 0. 0189 0. 0213 0. 0182 [3] (66) 0. 0113 0. 0177 0. 0198 0. 0206 [4] (127) 0. 0097 0. 0201 0. 0224 0. 0115 [5] (46) 0. 0143 0. 0153 0. 0188 0. 0105 [1] http: //www. bluemartini. com [2] http: //www. datamining. com [3] http: //www. mineit. com 29 [4] http: //www. sgi. comlsoftwarelmineset [5] http: //www. spss. comlclementine

Future work l l l 30 Use of metadata. Study how links can be used to infer more unexpected information. Monitor the site, reporting any unexpected change.

Intrinsic limitations l l 31 Text oriented. Do not work with images. Can have problems with tables. Do not work with dynamic web sites.

32