Data Extraction from the Web and Security Issues

Data Extraction from the Web and Security Issues By Siddu P. Algur Head, Dept. of Information Science & Engineering S D M College of Engg. & Tech. , Dharwad. siddu_p_algur@hotmail. com

CONENT • • Motivation Solution Existing Approaches New Approach (VSAP Algorithm) Empirical Evaluation Experimental Results Conclusion

Motivation • Huge amount of information on the Internet. • Data is distributed over Internet • Presence of undesired data along with relevant information • Requirement of data from various sources in local repository for further analysis

The Solution – Web Mining • Definition - Data mining on web pages • Data mining – Extraction of useful information by observing patterns in the data • Web mining can be used to extract the useful information from web pages. • We use Web-Page Structure Mining

Web Mining : Data Mining On the Web A Term coined by “ Etzioni“ in 1996

Web Structure Mining: The structure of a typical web graph consists of Web pages nodes and hyperlinks as edges connecting between two related pages. It can be regarded as the process of discovering structure information from the web Web Usage Mining: It focuses on techniques that could predict user behavior while the user interacts with the web. Web Content Mining: It emphasizes on the content of the web page. It is an automatic process that extracts pattern from web pages and goes beyond only the keyword extraction.

Web-Page Structure Mining • Defn : Identifying relevant information by observing the visual structure of a web page. • From the visual structure of web pages, we can determine the position of relevant data on the web pages.

But… Retrieving relevant information from the web seems to be like – Finding the Needle in the Haystack. . .

• The Web is highly volatile, distributed and heterogeneous. • The Web is a huge chaotic information space without central authority. • The Web is noisy.

Existing Approaches & their Limitations MDR Algorithm ( Mining Data Records from Web Pages ) DEPTA Algorithm ( Data Extraction using Partial Tree Alignment ) VIPS Algorithm ( VIsion based Page Segmentation )

MDR Algorithm Observations Data regions. A group of data records are presented in a particular region of a page and formatted using similar HTML tags. Data records. A group of similar data records being placed in a specific region are under the same parent in a tag tree.

MDR Algorithm Steps: l Building a HTML Tag Tree of a Page. l Mine Data Regions in page based upon Tag Tree & string comparison l Identify Data records from data regions.

Data Records 4 Generalized -Nodes Data Region TAG _TREE Data Records

Limitations Of MDR Algorithm • Tag Dependent ( <TABLE>, <TR>, <TD>etc ) • Extracts irrelevant data regions also along with the Relevant data region. • Needs to do content mining to identify relevant data region. • Highly prone to HTML tag-structure irregularities. Hence, fails in case of misuse of tags. • Incorporates considerable time in building tag tree, traversing whole tag tree and string comparison.

DEPTA Algorithm Steps : Given a page, it first segments the page using visual information, to identify each data record. A novel partial tree alignment method is used to align and to extract corresponding data items from the discovered data records and put the data items in a database table.

Limitations Of DEPTA Algorithm • Constructing a tag tree using visual information has the limitation that , the tag tree can be built correctly only as long as the browser is able to render the page correctly. • Tag-dependent and hence prone to tag-structure irregularities. • The computation time for constructing the tag tree and tree matching is an overhead. • Fails to identify the data records, in cases where there may be only a single record on page.

VIPS Algorithm Ø VIPS algorithm parses the HTML page and visual separators are detected in the parse tree. Ø The separators receive weights which are adjusted depending on constraints based on separator. Ø Finally, the content structure of the page is created, by merging ”visual” blocks that are not divided by separators.

Limitations Of VIPS Algorithm • VIPS also does not correctly identify the data regions. • VIPS is dependent on number of heuristic rules which do not hold good for most of the pages.

Tool Bar Search and Filtering Panel Data Region Data Object 1 ( Data Record 1 ) Content Links Data Object 2 ( Data Record 2 ) Data Object 3 ( Data Record 3 ) Advertise - ment Links Data Object 4 ( Data Record 4 ) Copyright Statement Layout of a typical Web Page

A Data Region containing 4 Data Records

Visual Structure based Analysis of web Pages ( The Proposed Approach ) Internet HTML Page Identifying the Data Region Largest Rectangle Identifier Container Identifier Co-ordinates of Bounding Rectangles Of All Tags Parsing & Rendering Engine MSHTML. dll Data Region Identifier (Filter) Relevant Data Region VSAP

VSAP Algorithm Steps : Ø Determine the co-ordinates of all the bounding rectangles. Ø Identify the Data Region. Ø Identify the Largest Rectangle. Ø Identify the Container within the Largest Rectangle. Ø Identify the Data Region containing the Data records within that Container.

Sample Web Page Of A Product related Web-site

HTML Parsing & Rendering Engine HTML Page Parsing & Rendering Engine Co-ordinates of Bounding Rectangles Of All Tags • Component of every Browser • Function – Parse & Render HTML Pages • Used to obtain bounding rectangles for each Tag. • E. g. MSHTML for Internet Explorer.

Web Page Bounding Rectangles

Data Region Extractor Co-ordinates of Bounding Rectangles Of All Tags Data Region Extractor Data Region • Made up of two components : – Container Identifier : Identifies the innermost tag which contains the data region – Filter : Filters the identified container to get the data region

Container Identifier • Obtains largest bounding rectangle – Child of the BODY tag • Get smallest rectangle with area greater than half the area of largest bounding rectangle.

Web Page Container Identified

Filter • Find Average Height of the children of the container • Eliminate children whose height is less than average height

Container Data Region

EMPIRICAL EVALUATION • • • Data Region Identification MDR – Dependent on specific tags for identifying data regions. VSAP – Identifies data regions independent of specific tags. Data Record Extraction MDR – Identifying data records based on keyword search ( e. g. “ $ ” ) VSAP – Identifying data records based on visual structure of the web page. Overall Time Complexity MDR – O ( NK ) , N is total no. of nodes in tag tree and K is max. no. of tag nodes of a generalized node. DEPTA – O ( k 2 ) , k is the number of trees. VSAP – O ( n ) , n is the no. of tag - comparisons made.

Performance Measures Recall = Ec Nt Et Precision = Ec Et is the total number of correctly extracted records. is the total number of records on the page. is the total number of records extracted. The percentage of relevant data records identified from the web page. Precision : The correctness of the data records identified. Recall :

EXPERIMENTAL RESULTS VSAP MDR URL Cor. Wr. 1. http: //www. tigerdirect. com/…. 8 36/ 0 8 0/0 2. http: //www. amazon. com/…. 0 17/25 25 1/0 3. http: //www. cooking. com/…. 17 13/3 20 0/0 4. http: //www. ebay. com/…. . 25 30/0 25 0/0 5. http: //www. powells. com/…. . 9 47/1 10 5/0 6. http: //www. barnesandnoble. com/…. 10 30/0 10 0/0 7. http: //www. pricegrabber. com/…. . 0 0/25 25 0/0 8. http: //www. shoebuy. com/…. 12 12/84 96 0/0 9. http: //www. smartbuy. com/…. . 0 15/10 10 0/0 10. http: //www. reviews. cnet. com/…. . 24 38/1 25 3/0 11. http: //www. nothingbutsoftware. com/ 10 12/0 10 0/0 12. http: //www. refurbdepot. com/…. 0 6/15 15 0/0 13. http: //www. drugstore. com/…. 15 14/0 15 0/0 14. http: //www. bookpool. com/…. 10 7/0 10 0/0 15. http: //www. target. com/…… 0 0/12 12 1/0 140 277 / 176 316 10 / 0 Total Recall 33. 5% 96. 93% Precision 44. 3% 100%

Performance Comparison of MDR and VSAP

Data Record Extractor DATA REGION DATA RECORD EXTRACTOR DATA RECORD • Extraction of data records is based on visual clues. • Height of each record is obtained. • Average height is calculated • Data records whose height is greater than the average height is extracted.

DATA REGION DATA RECORDS

Data Record Identifier DATA RECORD IDENTIFIER FLAT DATA RECORD NESTED DATA RECORD The flat record gives description of a single entity whereas the nested data record gives multiple description of a single entity

• Identification of data records is essential in order to simplify the task of extracting the data items, which is very much needed for various applications. • The Data Identifier determines the number of data fields in each data record within the data region. • The data fields in flat records are less as compared to that of nested records. • The number of fields in the nested data records is approximately 40% more than that of the flat records.

In fig 1 the number of fields is 12 and in fig 2 the number of fields is 7. The number of fields in fig 1 is 58. 3% more than the number of fields in fig 2. Fig 1 is a nested record and fig 2 is flat record.

Extraction of Data Fields • Extraction of Data Fields is based on bounding rectangles. • Each field is associated with a bounding rectangle. • Data fields are extracted row by row • The data fields are extracted and stored in a file.

Transferring the Data Items/Fields into the Database

Application • VSAP can be used by any application that requires the most relevant information of a web page • VSAP can provide a platform for an application that requires to analyze related data from different sources on the web. • VSAP can serve as an efficient replacement of MDR, which has already found it’s place in the industry.

Conclusion • Results show that Performance of VSAP is better than other existing algorithms • VSAP is a novel & efficient method of web mining

References [1] Baeza Yates, R. Algorithms for string matching: A survey. ACM SIGIR Forum, 23(3 -4): 34— 58, 1989. [2] J. Hammer, H. Garcia Molina, J. Cho, and A. Crespo. Extracting semistructured information from the web. In Proc. of the Workshop on the Management of Semi-structured Data, 1997. [3] D. Embley, Y. Jiang, and Y. K. Ng. Record-boundary discovery in Web documents. ACM SIGMOD Conference, 1999 [4] Kushmerick, N. Wrapper Induction: Efficiency and Expressiveness. Artificial Intelligence, 118: 15 -68, 2000. Clustering-based Approach to Integrating Source Query ] [5] Chang, C-H. , Lui, S-L. IEPAD: Information Extraction Based on Pattern Discovery. WWW-01, 2001. ] [6] Crescenzi, V. , Mecca, G. and Merialdo, P. ROADRUNNER: Towards Automatic Data Extraction from Large Web Sites. VLDB-01, 2001. ] [7] Y. Yang, H. Zhang. HTML Page Analysis based on Visual Cues. 6 th International Conference on Document Analysis and Recognition, 2001. [8] D. Buttler, L. Liu, C. Pu. A Fully Automated Object Extraction System for the World Wide Web. International Conference on Distributed Computing Systems (ICDCS 2001), 2001

[9] Bing Liu , Kevin chen-chuan chang, Editorial: Special issue on web content mining, WWW 02, 2002. [10] Liu, B. , Grossman, R. and Zhai, Y. Mining Data Records in Web Pages. KDD 03, 2003. [11] Cai, D. , Yu, S. , Wen, J. -R. and Ma, W. -Y. (2003). Extracting Content Structure for Web Pages based on Visual Representation, Asia Pacific Web Conference (APWeb 2003), pp. 406417. [12] A. Arasu, H. Garcia-Molina, Extracting structured data from web pages, ACM SIGMOD 2003, 2003 [13] J. Wang, F. H Lochovsky. Data Extraction and Label Assignment for Web Databases. WWW conference, 2003. [14] H. Zhao, W. Meng, Z. Wu, Raghavan, Clement Yu. Fully Automatic Wrapper Generation For Search Engines, International WWW conference 2005, May 10 - 14, 2005, Japan. ACM 1 -59593 -046 -9/05/005 [15] Zhai, Y. , Liu, B. Web Data Extraction Based on Partial Tree Alignment , WWW-05, 2005, May 10 -14, 2005, Chiba, Japan. ACM 1 -59593 -046 -9/05/0005 [16] Kosala R. , Hendrick Blockeel. Web Mining Research : A Survey , SIGKDD Explorations, ACM SIGKDD , July 2000.

Snapshot Amazon. com

Result By VSAP By MDR

Cooking. com

Result By VSAP By MDR

Tigerdirect. com

Result By VSAP By MDR

Overall VSAP Algorithm VSAP ( HTML document ) Begin Set max. Rect = NULL Set data. Region = NULL Find. Max. Rect (BODY); Find. Data. Region ( max. Rect ); Filter. Data. Region ( data. Region ); End

Algorithm to identify Largest Rectangle Procedure Find. Max. Rect ( BODY ) Begin for each child of BODY tag begin find the co-ordinates of the bounding rectangle for the child if the area of the bounding rectangle > area of max. Rect then max. Rect = child endif end End

Algorithm to identify Container of Data Region Procedure Find. Data. Region ( max. Rect ) Begin List. Children = depth first listing of the children of the tag associated with max. Rect for each tag in List. Children begin If area of bounding rectangle of tag > half the area of max. Rect then If area of bounding rectangle data. Region > area of bounding rectangle of tag then data. Region = tag endif end End

Algorithm to Filter Data region from the container Algorithm Filter. Data. Region (data. Region ) Begin for each child of data. Region begin total. Height += height of the bounding rectangle of child end avg. Height = total. Height / no of children of data. Region for each child of data. Region begin If height of child’s bounding rectangle < avg. Height then Remove child from data. Region endif end End

Thank You