Quality reporting in a short-term business survey based on administrative data M. Carla Congia congia@istat. it Fabio Rapiti rapitifa@istat. it ISTAT - Italy European Conference on Quality in Official Statistics Session on Quality reporting Rome, 8 -11 July 2008
Quality reporting Outline • The Italian Oros Survey • Quality issues in using administrative data • Peculiarities of data quality assessment • Oros quality indicators and reporting • Final remarks Q 2008 - Rome, 8 -11 July 2008
Quality reporting The Oros Survey • Since 2003 the Oros survey has released quarterly indicators on gross wages and total labour cost per FTE covering all size enterprises in the private non-agricultural sector (C to K sections Nace Rev. 1. 1) • Based on extensive use of administrative data (National Social Security Institute - INPS) combined with survey data on Large firms with more than 500 employees (Monthly Large Enterprise Survey) • Provisional estimates based on the “provisional population” are released with a 70 -days delay • Final estimates are produced after 5 quarters on the basis of the “whole population” and complete updated information • Meets also the requirements of the European regulations: ü STS - Short-Term Statistics ü LCI - Labour Cost Index (hourly labour cost index) Q 2008 - Rome, 8 -11 July 2008
Quality reporting The administrative source National Social Security Institute - INPS All Italian firms in the private sector with at least one employee have to pay monthly social security contributions to INPS (roughly 1. 3 million employers and 12 millions employees) DM 10 form The Monthly Declaration is a highly detailed grid where information on total employment, wage-bills, paid days, overtime hours and social contributions is identified by specific administrative codes (about 5, 000 valid codes) Each DM 10 lays in several records (8 on average) Data capturing • Every firm monthly transmits to INPS the DM 10 in electronic format, not later than 30 days after the reference period • Then the whole raw declarations are redirected to Istat at 35 days from the end of the reference period (about 10 millions records each month) Q 2008 - Rome, 8 -11 July 2008
Quality reporting The administrative data exploiting strategy A constrain became an opportunity At first INPS could not aggregate in the very strict time scheduled the DM 10 data in the format required for Oros purposes. So the Istat strategy became “Catch what you can” “as quick as you can” from a typical “one collection-for one single output/product” to focus on the “whole data source”- the wage and contribution system Advantages • microdata are exactly those sent by firms and this allows a more direct control of the aggregation/translation process • a lot of information available for many other different statistical purposes Disadvantages • a complex preliminary phase of checks and computation inside the single DM 10 to get to the target variables at micro level • a lot of data not necessarily useful for short-term objectives Q 2008 - Rome, 8 -11 July 2008
Quality reporting Quality issues in using INPS administrative data The Oros challenge is to produce short-term indicators processing • a huge quantity of very detailed microdata • in a very short time scheduled • coping with the frequent changes in the basic INPS metadata § enterprises have to use DM 10 form to take advantage of labour cost’s reduction policies and these contribution laws continuously change After preliminary studies INPS data have been considered to be suitable for Oros purposes but still statisticians have no quarterly ex-ante control over the quality of the raw administrative data Only a complex quality-oriented production process can assure ex-post quality coping with unusual problems Q 2008 - Rome, 8 -11 July 2008
Quality reporting Quality issues in using INPS administrative data Fragmented and insufficient Inps metadata In-house Metadatabase Highly disaggregated raw data Preliminary checks and accurate translation into statistical variables Integration with LE Survey data Checks to avoid double counting Continuos legislation changes Q 2008 - Rome, 8 -11 July 2008 Final key checks macroediting
Quality reporting Peculiarities of data quality assessment Relating to quality assessment of administrative data Eurostat recommends to produce: a source-specific report and a productspecific one In the Oros case the non-conventional use of administrative data implies that the two reports overlap……. while new approaches on administrative data quality assessment are empirically explored Oros practice has been developed trying: • to find better tools to assess quality • to manage the measurement of rather new indicators on: • efficient and stable data capturing • completeness and consistency of metadata • stable traslation/retrieval of target statistical variables • correct integration with LE survey data • to quarterly produce quality indicators along the whole production process • to meet both Istat and Eurostat requests on quality reporting Q 2008 - Rome, 8 -11 July 2008
Oros quality reporting: an overview + PROCESS Target group Oros Producers + QUANTITATIVE + QUALITATIVE Frequency + PRODUCT Internal users (of micro and macro data) Top managers and central quality managers External Expert users (Eurostat, IMF, BCE, . . ) General Public Survey Documentation and Methodological Handbook Once Metadata in SDDS Oros PR explanatory notes SIDI information system for survey documentation Annually LCI Quality Report Quarterly Oros Process Monitoring Report Quarterly LCI meta information Istat Quality Report
Quality reporting Survey Documentation and Methodological Handbook Initial basic quality assessment of the INPS administrative source to evaluate the suitability for the production of quarterly labour market indicators • Concepts and definitions of variables and population • Translation scheme of administrative information into statistical variables • Coverage • Reference time • Accuracy • Stability over time And obviously contening more about……. . the survey methods and the description of the whole production process Q 2008 - Rome, 8 -11 July 2008
Quality reporting Metadata in SDDS format Metadata in Special Dissemination Data Standard format used to deliver information to the IMF • Base page data, access by the public, integrity and quality • Summary methodology statements key features enabling users to assess the suitability of the data for their purposes • totally qualitative and compiled once: it is updated following the relevant changes in the methodology • compiled for the 3 outputs and different users • Oros • LCI • STS efforts to systematize Con. Istat - short-term indicators’TS database on Istat web-site Eurostat Q 2008 - Rome, 8 -11 July 2008
Quality reporting Process Monitoring Report 1 Quantitative indicators to keep continuosly under control and improve the quality along the whole Oros production process Some of them are also warning indicators : signal decisive problems or detect sources of error Main quality indicators for some key steps of the process: Data capturing MNumber of monthly records MNumber of DM 10 forms MTime lag between scheduled and actual delivery dates Metadata Database updating MDate of last updating of DM 10 metadata on INPS web-site MNumber of new and expired DM 10 codes by type MRate of new DM 10 codes to include/exclude • Number of official INPS acts to analyse Q 2008 - Rome, 8 -11 July 2008
Quality reporting Process Monitoring Report 2 Preliminary checks on administrative data MDM 10 codes error rate=Number of impossible codes/Total number of codes • DM 10 codes edit rate=Number of codes changed by editing/Number of impossible codes MRate of duplicate units=Number of duplicate units/Total number of units Micro editing MEdit rate=Number of unit edited/ Total number of units in scope for the item MTotal contribution to key estimates from edited values=Total weighted quantity for edited values on total weighted quantity for all final values Q 2008 - Rome, 8 -11 July 2008
Quality reporting Process Monitoring Report 3 • Number of units manually checked due to record linkage Integration with LE survey problems (i. e. mergers or split-ups recorded in different times) data Macroediting • Number of suspicious aggregates identified automatically by TERROR or through graphical checks • Number of outliers treated at micro or macro level • Total contribution to the estimates from treated values • Length of the homogeneous time series Q 2008 - Rome, 8 -11 July 2008
Quality reporting Istat Quality Report Still experimental Oros has been involved in the pilot test • quality indicators within a framework of a qualitative report coherent with Eurostat quality components • disseminated within the System on the Quality (SIQual) available on Istat website • external-user oriented • subset of standard quality indicators appropriately chosen within those available from the Information System for Survey Documentation (SIDI) • Response Rate • Indicators on the Revision policy (MR, MAR) • Timeliness for provisional data release • Timeliness for definitive data release • Length of the homogeneous time series • description of non-sampling error, relevance, accessibility Q 2008 - Rome, 8 -11 July 2008
Quality reporting LCI Quality Report Required by Eurostat to evaluate the quality of national LCI used to produce the European aggregate index LCI was established with an “harmonization of output” and not “harmonization of input” approach • since 2004 the LCI QR has been annually produced • standard structure based on Eurostat dimensions of quality with a further aspect “completeness” • main standard quality indicators used: • Revision policy (MR, MAR) • Timeliness for provisional data release • description of method for compiling hours worked (LCI denominator) Quarterly LCI meta information Standard Template mainly qualitative release-specific • Changes in the labour market (collective agreements, laws) which has an impact on wages and labour cost • Reasons of revisions in NSA, WDA and SA data Q 2008 - Rome, 8 -11 July 2008
Quality reporting Final remarks The Oros innovative quarterly use of administrative data forces to monitor peculiar aspects of quality not usually taken into consideration in the standard quality assessment approach suggested by Eurostat Several specific indicators to assess the quality of the process, in particular the metadata updating and the translation/aggregation of raw INPS data, have been implemented but they need to be more systematized These specific indicators are essential from the producer point of view, but they could also be used to report to the users the quality of some key issues On the other hand, the Oros survey satisfies the internal (SIDI, Si. Qual) and external (Eurostat) requests of standard quality reports A better integration of all the reviewed quality reporting tools is desirable but only partially achievable Q 2008 - Rome, 8 -11 July 2008
Скачать презентацию Quality reporting in a short-term business survey based
9d3df68df70cdd31e0961171e16c3f41.ppt