FDsys Data Analysis and Parsing 0 GPO Fdsys

FDsys Data Analysis and Parsing 0 GPO Fdsys – Search Engine Configuration 18 March 2018

Data Analysis and Parsing FDsys Agenda: • Data Management Definition • Parsing • fdsys. xml 1 GPO Fdsys – Search Engine Configuration 18 March 2018

FDsys Data Management Definition (DMD) 2 GPO Fdsys – Search Engine Configuration 18 March 2018

Data Management Definition (DMD) FDsys • Purpose of the Data Management Definition (DMD) – – Define collection-specific metadata elements Specify roles for the granules, if applicable Collection-specific schema definition for FDsys. xsd Define mappings of metadata elements for Documentum and FAST – Define mappings to metadata standards • One DMD for each collection • PMO & dev team collaborative effort for CDM documentation development • Is both a document and a process 3 GPO Fdsys – Search Engine Configuration 18 March 2018

The DMD Defines how Data Flows Through FDsys how will parser data and input files be validated FDsys what renditions are available? how will the MODS be created? how will metadata be extracted and merged? how will the HTML rendition be created what manual edits may be required? how are PDF files processed? what’s on the search form? 4 what do content how are search URLs look GPO Fdsys – results formatted? Search Engine Configuration like? how will the content and metadata be indexed what are the navigators? 2018 18 March

DMD – Table of Contents 1. 2. 3. 4. 5. 6. 7. FDsys General Description fdsys. xml Schema Elements Renditions, Plant Processing and Interractions Parser Definition – Extraction patterns and algorithms Content Management Content Publishing and Index Search and Browse • Search results, navigators, and collection browsing 8. Content Delivery • URLs, content-detail, Front page, actions 9. mods. xml mappings 5 GPO Fdsys – Search Engine Configuration 18 March 2018

Metadata Flow Diagram 6 Metadata Flow FDsys Diagram GPO Fdsys – Search Engine Configuration 18 March 2018

Metadata Flow Diagram fdsys. xml Metadata structure parsing rules Flow FDsys Diagram search index field mapping CMS metadata mapping mods mapping content-detail mapping search results mapping browse algorithm search-form mapping 7 GPO Fdsys – Search Engine Configuration 18 March 2018

Federal Register Granules FDsys • Each article is a granule • Each Part is a single granule • There are no higherlevel granules • Sections are not preserved as independent granules 8 GPO Fdsys – Search Engine Configuration 18 March 2018

Federal Register Example Metadata FDsys agencies title action summary dates contact FR Doc Number 9 Billing Code GPO Fdsys – Search Engine Configuration 18 March 2018

Content Files FDsys Input Files Renditions locator SGML CDTP text extract granules pdfsubmitted PDF OCR embedded images extract granules pdf (public) Create “Front. Matter”, “Reader. Aids”, and “Issue” PDF files 10 GPO Fdsys – Search Engine Configuration 18 March 2018

Content Files – Creating the HTML Rendition text Add HTML headers and header metadata FDsys Add URL and E-mail links embed image tags pdfsubmitted extract images as JPEG OCR images 11 html image s html (public) longdesc text GPO Fdsys – Search Engine Configuration 18 March 2018

Extracting Metadata SGML TOC SGML content CDTP parse FDsys (TOC headings) parse overwrite add Merged Metadata • Metadata is merged based on the FR Doc Number 12 GPO Fdsys – Search Engine Configuration 18 March 2018

Search Results collection volume firstpage FDsys action (first 20 chars) section rin title 73 FR 22020 - Title I-Improving the Academic Achievement of the Disadvantaged [PDF 123 KB] Federal Register. Proposed Rules. Notice of proposed rulemaking. RIN 0324 -AJ 10. Wednesday, April 23, 2008. . The Secretary proposes to amend the regulations governing programs administered under Part A of Title I of the Elementary and Secondary Education Act of 1965, as amended (ESEA). . . More Information. . . publishdate 13 teaser GPO Fdsys – Search Engine Configuration link to content-detail 18 March 2018

FR Navigators FDsys • Section • Agency • CFRs – Hierarchial + 15 CFR - Part 12 - Part 13 - Part 14 + 16 CFR - Part 412 - Part 413 14 GPO Fdsys – Search Engine Configuration 18 March 2018

Collection Browsing FDsys yearnav monthnav daynav agencynav 15 GPO Fdsys – Search Engine Configuration 18 March 2018

Advanced Search Form 16 GPO Fdsys – Search Engine Configuration FDsys 18 March 2018

Package-Level URLs FDsys • Package Content Detail – http: //www. gpo. gov/fdsys/pkg/FR-2006 -01 -01/content-detail. html • Package Metadata Standards – http: //www. gpo. gov/fdsys/pkg/FR-2006 -01 -01/mods. xml – http: //www. gpo. gov/fdsys/pkg/FR-2006 -01 -01/premis. xml • Package Table of Contents – http: //www. gpo. gov/fdsys/pkg/FR-2006 -01 -01/toc. html • Today’s Table of Contents – http: //www. gpo. gov/fdsys/html/FR/todays_toc. html 17 GPO Fdsys – Search Engine Configuration 18 March 2018

Granule-Level URLs FDsys • HTML and PDF Files – http: //www. gpo. gov/fdsys/pkg/FR-2006 -01 -01/html/E 6 -1423. html – http: //www. gpo. gov/fdsys/pkg/FR-2006 -01 -01/pdf/E 6 -1423. pdf • Granule Content Detail – http: //www. gpo. gov/fdsys/granule/FR-2006 -01 -01/ E 6 -1423/content-detail. html • Granule Metadata Standards – http: //www. gpo. gov/fdsys/granule/FR-2006 -01 -01/ E 6 -1423/mods. xml 18 GPO Fdsys – Search Engine Configuration 18 March 2018

Content Detail FDsys m a S 19 le p I U GPO Fdsys – Search Engine Configuration 18 March 2018

FDsys Parsing 20 GPO Fdsys – Search Engine Configuration 18 March 2018

Parsing Overview FDsys • Runs regular expressions to extract metadata Regular Expression: (Public Law|Pub. L. |PL|P. L. ) (1[0 -9])-([0 -9]+) Example: Pub. Produces: • Written in Java • Called from Documentum when a package needs to be parsed • Produces an instance of fdsys. xml – Parsing has an internal XML format (called the “raw” XML) which is transformed to produce the fdsys. xml 21 GPO Fdsys – Search Engine Configuration 18 March 2018

Parser Foundation Classes FDsys PContainer PParser PPackage PRendition PFile PGranule USCODE Parser USCODE Package USCODE FR Rendition USCODE FR FR File USCODE Granule • Foundation classes handle 95% of parsing needs • Derived classes handle all special cases 22 GPO Fdsys – Search Engine Configuration 18 March 2018

PContainer FDsys • Takes patterns and produces elements • Holds XML at each level of the parsing process XML Fragment PPattern used_by PContainer XML DOM used_by produces stored_in 109 123 "(Public Law|Pub. L. |P. L. ) (1[0 -9])-([0 -9]+)" 23 GPO Fdsys – Search Engine Configuration 18 March 2018

Parser Foundation Classes FDsys PContainer PParser PPackage PRendition XML DOM XML PFile XML DOM XML PGranule XML DOM XML XSLT priority merge append xml 24 GPO Fdsys – Search Engine Configuration 18 March 2018

Parsing XML Documents FDsys PContainer PParser PPackage PRendition XML DOM XML PFile XML DOM XML XSLT priority merge append XSLT fdsys. xml bills. xml 25 GPO Fdsys – Search Engine Configuration 18 March 2018

Other Parsing Considerations FDsys • Heuristics testing is integrated into the parsing – PEHelper: Checks for heuristics and adds “quality=“ attributes • Output can be automatically Schema-Validated – Schema-Validation is run on all fdsys. xml formats produced by the parser • Parser Validation Tool – Used by GPO to validate that parsers meet the 90% Service Level Agreement for accuracy – Randomly selects 100 documents or granules – Displays metadata & original text for manual review – Produces Validation Report 26 GPO Fdsys – Search Engine Configuration 18 March 2018

FDsys fdsys. xml 27 GPO Fdsys – Search Engine Configuration 18 March 2018

FDsys. xml Purpose FDsys • Internal container of metadata related to package • Is a detailed representation/model of the data structure across all of FDsys • Reduces duplication of data across metadata formats • Reduces number of required transformations • Can be transformed into standard schemas including: – METS – MODS – PREMIS 28 GPO FDsys – Data Model June 13, 2008

FDsys. xml General Structure FDsys Header Content Metadata 29 GPO FDsys – Data Model June 13, 2008

FDsys Publish and Search 30 GPO FDsys – Data Model June 13, 2008

Publish and Search FDsys Agenda: • FDsys Publish • Search Engine Configuration • Search Engine Application Services 31 GPO FDsys – Data Model June 13, 2008

FDsys Publish 32 GPO Fdsys – Search Engine Configuration 18 March 2018

High-Level SW Components 33 GPO Fdsys – Search Engine Configuration FDsys 18 March 2018

Content Publishing - Overview FDsys • Communicates from Documentum to Access • From: Documentum – Extract fdsys. xml & premis. xml – Extract renditions and content files – Uses Documentum native DFC calls • To: ACP Cache – Stores metadata and content files • To: FAST ESP Search Engine – Converts fdsys. xml to FAST. xml -> to indexer • Includes the mods. xml (indexed into ESP) • ESP pulls in content files automatically – Uses FAST ESP content_api & search_api calls 34 GPO Fdsys – Search Engine Configuration 18 March 2018

Component Interfaces FDsys UPDATE THIS 35 GPO Fdsys – Search Engine Configuration 18 March 2018

Component Interfaces 36 GPO Fdsys – Search Engine Configuration FDsys 18 March 2018

Major Architectural Decisions FDsys • Pull from Documentum, not Push – Maintenance of Access Subsystem databases becomes the responsibility of the Access Subsystem – Data is pulled from Documentum only as needed • Avoids overflow/queuing problems – Allows multiple access systems to be fielded • Search for Deletes in FAST – Packages can contain many granules – When updating the FAST indexes, use search to find the list of all nested granules in the indexes – Guaranteed to avoid any “orphan” granule problems 37 GPO Fdsys – Search Engine Configuration 18 March 2018

ACP Cache Directory Structure 38 GPO Fdsys – Search Engine Configuration FDsys 18 March 2018

Metadata Flow FDsys Diagram 39 GPO Fdsys – Search Engine Configuration 18 March 2018

Implementation Detail 40 GPO Fdsys – Search Engine Configuration FDsys 18 March 2018

FDsys Search Engine Configuration Design 41 GPO Fdsys – Search Engine Configuration 18 March 2018

Component Interfaces FDsys Update This 42 GPO Fdsys – Search Engine Configuration 18 March 2018

FAST System – Hardware & Network FDsys publish & admin document processors index & search index & search search Web Application 43 GPO Fdsys – Search Engine Configuration 18 March 2018

FAST System – Indexing Flow FDsys publish & admin index & search index & search 44 document processors search GPO Fdsys – Search Engine Configuration 18 March 2018

FAST System – Search index & search FDsys index & search search search QR server QR server Web Application 45 GPO Fdsys – Search Engine Configuration 18 March 2018

Search Engine Sizing: Columns FDsys • Total Number of Documents – Estimated 10 million records • Each granule = 1 Search Engine document – Allow 2 x expansion for estimation errors and growth – Estimated 20 million records • Sizing Recommendation: – FAST recommends: 5 million records per column • For public facing web sites – 5 columns: to account for the large number of navigators 46 GPO Fdsys – Search Engine Configuration 18 March 2018

Search Engine Sizing: Disk FDsys • Year 2006 FR – Index Sizing Test Documents Text Fixml Index Total FAST 31, 500 mb 230 mb 604 mb 834 mb • Scale to 20 million documents – Fixml: ~150 gb – Index: ~420 gb • Total index space required: – 150 gb + (420 gb)*2 = 1 tb – Add 50% for estimation error, total = 1. 5 tb 47 GPO Fdsys – Search Engine Configuration 18 March 2018

Search Engine Sizing: QPS FDsys • Queries per second – Estimated from GPO Access – 0. 8 QPS (across the whole day) – Estimated peak: 2. 4 qps (1/2 of queries in 4 hours) • Estimated Peak QPS for FDsys: – – Factor for improved search interface: 3 x Factor for growth: 2 x Estimated: 2. 4 x 2 x 3 = ~15 QPS Correllates with other websites known to ST • Each row: 20 -30 qps – Therefore: 1 row for query performance • Recommend: 2 rows – 2 nd row for redundancy, failover, and maintenance 48 GPO Fdsys – Search Engine Configuration 18 March 2018

Metadata Flow FDsys Diagram search index field mapping 49 GPO Fdsys – Search Engine Configuration 18 March 2018

Mapping to the Index Profile FDsys index profile fields: index. xslt metadata for search resultsbundle mods. xslt fdsys. xml metadata for simple search xml scope field grank 1 -6 publishdate title metadata for navigation FAST Extractors merge ACP Cache content files 50 collection specific standard navigators body GPO Fdsys – Search Engine Configuration 18 March 2018

Collections and Codes FDsys • Three types of collections: – Processing Collection = “collection. Code” or “processing. Code” • Parsing, submission, processing, workflow • Chooses which index. xslt and search. xslt to apply – FAST Index Collection • One for each processing collection • Allows easily deleting all documents in a collection – “Access Collection” = “accode” • Re-group documents into collections for public users • 98% the same as the “Processing Collection” – Reports in the Congressional Record, FR Unified Agenda • Mapping is done in index. xslt 51 GPO Fdsys – Search Engine Configuration 18 March 2018

Simple Maintenance #1 FDsys • New Collections – Just add the collection (admin GUI) – Start feeding data • Add new fields – Add the field to the index profile – Reload profile with a hot-update • Backups – Turn off feeding – Wait for documents in process to finish up – Make index backups 52 GPO Fdsys – Search Engine Configuration 18 March 2018

Simple Maintenance #2 FDsys • Archiving Log Files – Simple file copy, can happen any time • Correct Field Mapping Errors – Remove all documents in the FAST ESP collection – Re-index collection from ACP Cache • Get list of packages in the collection from Documentum • Does not require re-export (or re-publish) the packages from CMS • Reorganize Access Collections – Remove all documents from affected collections – Re-index affected collections from ACP Cache 53 GPO Fdsys – Search Engine Configuration 18 March 2018

Extensive Maintenance FDsys • Examples: – New FAST Version – New FDsys Version – Complex index-profile changes (remove fields, major restructuring) – Re-organizing collections or field mapping while maintaining searches on the old snap-shot • Process: – Servers to “stand-alone” mode – Make changes – Restore normal server operations 54 GPO Fdsys – Search Engine Configuration 18 March 2018

Monitoring FDsys • FAST standard monitoring tool (“Clarity”) • Monitors query and indexing performance • Built-in alerting mechanism 55 GPO Fdsys – Search Engine Configuration 18 March 2018

Backups FDsys • data_fixml – Holds processed copy of the indexes – Can be used to reconstruct the indexes in about 4 hours (will need to be benchmarked) • data_index – The complete indexes actually used for searching • Configuration backup • When restoring a backup: – Will need to re-push all content updates which occurred since the last backup 56 GPO Fdsys – Search Engine Configuration 18 March 2018

Disaster Recovery Scenarios FDsys • Servers crash – FAST restarts them automatically • Hanging server processes – Shut it down manually and restart it • Incremental indexing overloads the system – Should not happen in FDsys – Can “slow down” incremental indexing until situation is corrected • Severe incremental indexing problems – Revert to periodic batch index updates 57 GPO Fdsys – Search Engine Configuration 18 March 2018

FDsys Search Services API 58 GPO Fdsys – Search Engine Configuration 18 March 2018

Component Interfaces 59 GPO Fdsys – Search Engine Configuration FDsys 18 March 2018

Responsiblities: Search Services vs Web Application FDsys Search Services API Web Application • All communication to FAST • Choosing which fields when – All FAST API calls – All FAST parameters • All FAST FQL – User query strings and parameters to FAST FQL • Raw data values – Allowed values, navigator values, search results field values, etc. • Choosing Navigators 60 – Advanced Search Form – Search Results • User-interface oriented field data – Display names, help text, display widgets • Display value translation – translating from raw data values to/from user-friendly values GPO Fdsys – Search Engine Configuration 18 March 2018

Responsiblities: Search Services vs Web Application – Browse Trees FDsys Search Services API Web Application • Browse tree computation • Browse tree presentation – Selecting nodes to return – Returning an ordered list of nodes – Caching search results • Embargo Dates 61 – The definition of the levels – How many levels to display when – Presentation of tree – Translating raw data values to user friendly values • Content Detail Pages GPO Fdsys – Search Engine Configuration 18 March 2018

Component Interfaces FDsys Java method calls HTTP FAST Search Engine FAST Search API Parsing, Processing & Caching Search Services API Configuration files (XML) Search Web Application Master FR CR CFR Collection Specific Cong. Bills 62 GPO Fdsys – Search Engine Configuration 18 March 2018

Configuration File Contents FDsys • Fields – for the Advanced Search Form – for field: searches – Allowed values • Fixed enumerated list • Enumerated list built from navigator • Numeric or Date Range • Navigators per collection • FAST ESP Search Engine Connection Info • Templates to reformat data for display 63 GPO Fdsys – Search Engine Configuration 18 March 2018

Query Parsing: Syntax FDsys • atom – Atoms are space separated lists of characters, double -quoted strings, or parenthetical expressions • atom – defaults to AND • • • 64 atom and atom or atom before/# atom near/# atom adj atom • • • not atom +atom -atom field: atom range(#, #) range(, ) GPO Fdsys – Search Engine Configuration 18 March 2018

Query Parsing: Examples • • hearing “congressional hearing” congressional adj hearing congressional hearing ways and means “ways and means” ways “and” means congressional or congress • congress and (report or meeting or notice) • congnum: range(103, 110) 65 FDsys • congressional not report • congressional –report • +cardin congressional committee • congresional not (committee report) • congressional not (committee or meeting) • representative near/10 cardin • representative before/10 cardin GPO Fdsys – Search Engine Configuration 18 March 2018

Derived Hierarchy: Example FDsys • 110 th Congress – House Bills • H. R. 1 -200 • H. R. 201 -400 – H. R. 201 – H. R. 202 – H. R. 203 • Engrossed in House • Introduced in House. Condemning the persecution of labor rights advocates in Iran [PDF] [Text] • Referred in Senate – H. R. 204 • H. R. 401 -600 66 GPO Fdsys – Search Engine Configuration 18 March 2018

Specified Hierarchy: Example 67 GPO Fdsys – Search Engine Configuration FDsys 18 March 2018

Parser Foundation Classes FDsys PContainer PParser PPackage XML DOM PFile XML DOM XML PGranule XML DOM XML XSLT append xml 68 GPO Fdsys – Search Engine Configuration 18 March 2018