Integrated metadata systems History Status Vision Roadmap Rune

Integrated metadata systems History Status Vision Roadmap Rune. Gloersen@ssb. no

Integrated Metadata Systems u u Stove-piped statistical production (systems) with no, or at the best, encapsulated metainformation, represents our remains from the IT stone-age. First steps towards the consciousness of metadata(structures) were taken some 20 years ago: ü metadatadriven on-line systems ü file description- and other archives of structured documentation u u u The technological evolution has been the driving force towards a vision of a coherent statistical (IT) system However, the state-of-the-art-technology has also at all times represented one of the most important obstacles to success in addition to the human- and organisational barriers that we also discuss

Technological barriers u u u Lack of processor speed and data storage capacity lack of access possibilities across different IT systems Lack of database functionality and flexibility Lack of awareness of metainformation as a whole in the IT industry (handling of technical meta-information at the most), i. e the kind of metainformation that was handled in the first datawarehouse solutions Lack of (IT) standards, but anyhow; why didn’t we achieve more when we had all our information systems within one mainframe ? due to the human and organisational barriers ?

Our current advantages u u WWW Open standards on ü connectivity · LAN/WAN communication · database connectivity ü standardised exchange of data on · protocol level · syntactic level u u Object orientation Web services ! But what about the semantic level ?

A vision for a coherent statistical system u u The basic architecture of a coherent statistical system is formed by the structure, content and handling of metainformation The IT system will never reflect anything else but the level of standardisation and coordination of the statistical production within the organisation NSI’s must take into account all statistical IT systems currently running, having been developed over the past 20 years, which would need to fit into a new or upgraded system A coherent statistical system based on integration of what you already have, or convert everything to a new (gigantic) system ?

A vision for a coherent statistical system Design and planning Objective Content Population Sample Collection methods Knowledge Evaluation Process methods Dissemination Input data Knowledge base Expert knowledge: -Guidelines -Articles -Methods -People Local metadata Global metadata Classifications Standards -Datadoc -Stat. Activities -Stat. doc -Quality decl. -Structured metadata Stat. data Operation Establish population & sample Data collection & Edit Estimation Aggregation Local prod. data Populations Presentation Dissemination Local prod. data Observation register Dissemination database Datawarehouse Source: Bo Sundgren

Metadata Local metadata Questionnaire repository File descript. Content (Quality) declaration Variable definitions Macro database Classifications Statistical activities

Metadata Local metadata Questionnaire repository File descript. Content (Quality) declaration Census/ Survey Variable definitions Macro database Classifications Statistical activities

Metadata Local metadata Questionnaire repository File descript. Content (Quality) declaration Variable definitions What information is needed to establish consistent links between the components of your (structured) metainformation system ? Macro database Classifications Statistical activities

Metadata Local metadata Questionnaire repository File descript. Content (Quality) declaration Variable definitions Macro database Classifications Statistical activities

Metadata Local metadata Questionnaire repository File descript. Variable definitions XML Content (Quality) declaration Macro database Classifications Statistical activities

Metadata components Local metadata Content (Quality) Question- declaration naire repository File descript. Variable definitions Statistical activities Classifications Macro database

Metadata components Local metadata Metamodel Linking/Mapping Content (Quality) Question- declaration naire repository File descript. Variable definitions Statistical activities Three layered model Metadata Data Classifications Macro database

Metadata components Process Collection Content (Quality) Question- declaration naire repository File descript. Variable definitions Statistical activities Linking/Mapping Local metadata Data Editing Estimation Aggregation Classifications Macro database Dissemination

Metadata components Macro database Domain 2 Domain 1 Classifications Linking/Mapping Content (Quality) Question- declaration naire repository File descript. Variable definitions Statistical activities Domain n Local metadata Different domains

Metadata components End user needs Content (Quality) Question- declaration naire repository File descript. Variable definitions Statistical activities Classifications Macro database Access Local metadata

Non-structured metainformation u Text-mining Knowledge systems u Challenge, and upcoming reality: u u How shall we be able to store, retrieve and maintain the knowledge of the organisation much more independent of their (shifting) staff ?

Metadata in the statistical production u u u Data input Data throughput Data dissemination

Data collection Internal Business Systems Electronic Questionnaires BS ELQ Paper Questionnaires P I Mapping between statistical and in-house data definitions Optical char. recognition, intrepretation verifiying www. ssb. no NSI Data Definitions Questions Rules/Checks Questionnaires XML Questionnaire generation Metadata OCR Central Raw Data Storage Links to a (national) repository of Data definitions/Questionnaires CRDS Linked to Business Register Subject matter systems