Statistical Quality Control The experience of the Banco

Statistical Quality Control The experience of the Banco de Portugal António Agostinho Maria José Valério Banco de Portugal Statistics Department Statistics Audit Unit Q 2008 - European Conference on Quality, Rome July 2008

Statistics Quality Control: the experience of the Banco de Portugal Outline 1. The Statistics Department 2. Quality Control Procedures 3. Statistics Audit 4. Institutional Cooperation 5. Conclusions and Prospects Ahead Q 2008 - European Conference on Quality, Rome, July 2008 2

Statistics Quality Control: the experience of the Banco de Portugal 1. Statistics Department v Created in 1997 to cope with rising statistical requirements (in particular in the framework of the Monetary Union) v Centralized statistical function allowing: Ø The concentration of the available statistical skills; Ø The increase of capacity to face the challenges resulting from building up the European System of Central Banks; Ø The improvement of efficiency gains and the enhancing of statistical quality control at all levels of the statistical process (collection, processing, analysis and dissemination); Ø The adoption of consistent methodological approaches; and, Ø An easier data sharing and coordination among all the staff in charge of statistical production. Q 2008 - European Conference on Quality, Rome, July 2008 3

Statistics Quality Control: the experience of the Banco de Portugal 1. Statistics Department v Organization Chart Executive Management Total: 2 (Univ. Deg. 2) Statistics Dissemination Unit Total: 10 (Univ. Deg. 8) Central Credit Register Information Unit Total: 14 (Univ. Deg. 3) Statistics Audit Unit Total: 3 (Univ. Deg. 3) Staff Support Unit Total: 13 (Univ. Deg. 3) Methodological Developments Unit Total: 3 (Univ. Deg. 3) National Financial Accounts and Securities Statistics Division Total: 23 (Univ. Deg. 17) Balance of Payments and Int. Invest. Position Statistics Division Total: 37 (Univ. Deg. 15) Employees with university degree 69 ( 44% ) Central Balance-Sheet Division Total: 32 (Univ. Deg. 23) 88 ( 56% ) Other employees Monetary and Financial Statistics and Central Credit Register Division Total: 20 (Univ. Deg. 11) Total 157 Q 2008 - European Conference on Quality, Rome, July 2008 December 2007 4

Statistics Quality Control: the experience of the Banco de Portugal 1. Statistics Department v Four Divisions Four different areas of statistical production v Vertical organization in each Division v Regular hierarchical statistical production meetings (on a monthly basis) v Internal technical task forces to deal with transversal issues Q 2008 - European Conference on Quality, Rome, July 2008 5

Statistics Quality Control: the experience of the Banco de Portugal 2. Quality Control Procedures v Quality control procedures along all the phases of statistical production process: Ø Data collecting Ø Data processing and analysis Ø Statistics dissemination Q 2008 - European Conference on Quality, Rome, July 2008 6

Statistics Quality Control: the experience of the Banco de Portugal 2. Quality Control Procedures § Data Collecting - 1 v Data collecting is based on legal provisions v Involvement of the reporting institutions is pursued v For more complex changes in collection systems working groups are created v Detailed reporting instructions and handbooks are delivered to the reporting institutions v Appointment of contact persons – respondents for each specific statistical data submission (training sessions and regular meetings) Q 2008 - European Conference on Quality, Rome, July 2008 7

Statistics Quality Control: the experience of the Banco de Portugal 2. Quality Control Procedures § Data Collecting - 2 v Specific designed software is made available to the reporting institutions - IT tests and consistency tests on elementary data v BP net – a secure electronic communication system between the Banco de Portugal and the other financial institutions v Feedback to reporters – the CCR and CBS experiences v Statistical reporting indicators Q 2008 - European Conference on Quality, Rome, July 2008 8

Statistics Quality Control: the experience of the Banco de Portugal 2. Quality Control Procedures § Data Collecting - 3 Quality Assessment Reports – Major Financial Groups v Cross-checking data reported by MFIs: Ø MFIs’ report to money and banking statistics (MBS) versus accounting data Ø MFIs’ report to MBS versus report for Central Credit Register Ø MFIs’ report to MBS versus data collected/reported for securities statistics Ø Securities portfolios versus securities issues Ø MFIs’ report to MBS versus report for Bo. P statistics (external assets and liabilities) v High level periodical statistical meetings Q 2008 - European Conference on Quality, Rome, July 2008 9

Statistics Quality Control: the experience of the Banco de Portugal 2. Quality Control Procedures § Data processing and analysis - 1 v Full compliance with the international methodological framework: Ø Ø scope of statistics Ø classification and sectorization systems Ø valuation rules Ø v concepts and definitions recording principles Adjustments and/or appropriate substitute measures are developed when the source data are not in accordance with the standards Q 2008 - European Conference on Quality, Rome, July 2008 10

Statistics Quality Control: the experience of the Banco de Portugal 2. Quality Control Procedures § Data processing and analysis - 2 v Validation tests performed on individual and/or aggregated data and also at different phases of data processing – source data, intermediate data and final statistical data: Ø Ø temporal consistency Ø v internal consistency cross-check control Additional control in the framework of Financial Accounts compilation Q 2008 - European Conference on Quality, Rome, July 2008 11

Statistics Quality Control: the experience of the Banco de Portugal 2. Quality Control Procedures § v Statistical Dissemination - 1 Meeting the users’ needs and the statistical reporting requirements of International Organizations v Main features of the statistics dissemination: Ø Release in time according to a pre-announced schedule Ø Clear and understandable presentation Ø Workshops with media and media references monitoring Ø Monthly, free of charge, to all users at the same time, in the Bd. P website: • Statistical Bulletin (also available in paper and CD-ROM) • BP stat – allows online access to the interactive database of Bd. P v Assistance to users – contact points in Bd. P website and in the Statistical Bulletin Q 2008 - European Conference on Quality, Rome, July 2008 12

Statistics Quality Control: the experience of the Banco de Portugal 2. Quality Control Procedures § Statistical Dissemination - 2 v BP stat | Statistics online | facilities: Ø Portuguese and English versions; Ø Data under two formats: time series and multidimensional analysis; Ø Data in hierarchical structure – tree of statistical domains; Ø Data and metadata may be accessed, downloaded or printed; Ø Internet links to the relevant metadata references; Ø Predefined and/or personalised tables (“Favourites” option); Ø “Alert” messages by email to the users; and, Ø Glossary, search, calendar and help options are available. Q 2008 - European Conference on Quality, Rome, July 2008 13

Statistics Quality Control: the experience of the Banco de Portugal 2. Quality Control Procedures § Quality Control Matrix Outside the production cycle Ø Appointment of contact persons Ø Detailed reporting instructions and handbooks Ø Regular meetings with the reporting institutions Ø Training sessions Ø Feedback information to the reporters Ø "Quality Assessment Reports“ Ø High level periodical statistical meetings Ø Internal technical task forces BP net – electronic communication system Specific designed software - IT tests Consistency tests on elementary data Ø Additional control in the framework of Financial Accounts Totally automated During the production cycle Ø Validation tests: § internal consistency § temporal consistency § cross-check control Ø Regular statistical production meetings Judgment needed Q 2008 - European Conference on Quality, Rome, July 2008 14

Statistics Quality Control: the experience of the Banco de Portugal 3. Statistics Audit § Main purposes (1) v Analyse and evaluate the different phases of statistical production procedures in place, from source data collection up to final statistics compilation and dissemination v Analyse the organizational and functioning aspects, in particular evaluate the burden and the share of functions and responsibilities in the statistical production process v Evaluate the efficiency of the procedures in place, identifying the good practices as well as improvement needs Q 2008 - European Conference on Quality, Rome, July 2008 15

Statistics Quality Control: the experience of the Banco de Portugal 3. Statistics Audit § Main purposes (2) v Contribute to enhance methods and applied techniques and to improve the statistical production quality by issuing, when it applies, suggestions/ recommendations on current practices or by suggesting new procedures and/or organizational arrangements v Encourage and promote the share and comparison of good practices and procedures among the different areas of the Statistics Department Q 2008 - European Conference on Quality, Rome, July 2008 16

Statistics Quality Control: the experience of the Banco de Portugal 3. Statistics Audit § Methodological approach – AK list (1) A. Initial appointment with the responsible of the unit in charge of the statistics under audit B. Collect, organize and analyse the documentation made available by the unit C. Submission of the “Self Assessment Questionnaire” D. Fieldwork E. Definition of a flow chart on the information circuit F. Preparation of the Check List of procedures Q 2008 - European Conference on Quality, Rome, July 2008 17

Statistics Quality Control: the experience of the Banco de Portugal 3. Statistics Audit § Methodological approach – AK list (2) G. H. I. J. K. Joint analysis with the unit’s manager of the Self Assessment Questionnaire Graphical assessment of the statistical production system (radar chart) Report on the statistical audit process (1 st draft) – submission to the unit under audit Joint evaluation of the comments from the unit under audit Final report on the statistical audit process – submission to the Head of the Statistics Department Q 2008 - European Conference on Quality, Rome, July 2008 18

Statistics Quality Control: the experience of the Banco de Portugal 3. Statistics Audit § Self Assessment Diagram (1) v The self assessment questionnaire is the basis for the presentation of the radar chart v Two sets of answers to the questionnaire – one, in the beginning of the audit process, from the unit under audit and another, at the end, from the Statistics Audit Unit – are contrasted graphically in a statistical production assessment diagram Q 2008 - European Conference on Quality, Rome, July 2008 19

Statistics Quality Control: the experience of the Banco de Portugal 3. Statistics Audit § Self Assessment Diagram (2) Self Assessment Diagram (Securities Issues Unit (SIET) Accessibility Metadata Accessibility Staff Functions Share 5 Relevance System Adequacy 4 Dissemination Clearness and Coverage User Satisfaction 3 2 Consistency with other Statistics Overall Relevance 1 Serviceability 0 Temporal Consistency Transparency Degree Dissemination Punctuality Integrity Overall Coverage Methodological Soundness Intermediary and Final Data Validation Reporting Compliance Methodological Soundness Degree Elementary Data Quality SIET Unit Accuracy and Reliability Q 2008 - European Conference on Quality, Rome, July 2008 1 - Very low 2 - Low 3 - Medium 4 - High 5 - Very high 20

Statistics Quality Control: the experience of the Banco de Portugal 3. Statistics Audit § Self Assessment Diagram (3) Self Assessment Diagram (Securities Issues Unit (SIET) versus Statistics Audit Unit (NAE)) Accessibility Relevance Metadata Accessibility Staff Functions Share 5 System Adequacy 4 Dissemination Clearness and Coverage User Satisfaction 3 2 Consistency with other Statistics Overall Relevance 1 Serviceability 0 Temporal Consistency Transparency Degree Dissemination Punctuality Integrity Overall Coverage Methodological Soundness Intermediary and Final Data Validation Reporting Compliance Methodological Soundness Degree Elementary Data Quality SIET Unit NAE Accuracy and Reliability Q 2008 - European Conference on Quality, Rome, July 2008 1 - Very low 2 - Low 3 - Medium 4 - High 5 - Very high 21

Statistics Quality Control: the experience of the Banco de Portugal 3. Statistics Audit § Overall assessment of the statistical audit operations (1) The overall assessment is quite significant: 1. The deep interest and good collaboration revealed by those intervening in the statistical production systems audited 2. These operations allow a detailed assessment on the current features of the statistical systems in place 3. The possibility of conveying upwards the desires and aspirations as well as the shortcomings faced by the responsibles of the units under auditing, which, otherwise, might encounter more constraints before being acknowledged Q 2008 - European Conference on Quality, Rome, July 2008 22

Statistics Quality Control: the experience of the Banco de Portugal 3. Statistics Audit § Overall assessment of the statistical audit operations (2) 4. The share in all Department of procedures and good practices in place 5. Most of the suggestions/ recommendations raised by the statistical auditors and accepted by the units under audit are already being implemented or are included in the unit’s planning for the years ahead 6. The relevance of the follow-up of the implementation of suggestions/ recommendations Q 2008 - European Conference on Quality, Rome, July 2008 23

Statistics Quality Control: the experience of the Banco de Portugal 4. Institutional Cooperation More efficient statistical production processes: v Facilitates data sharing and good practices’ diffusion; v Promotes the use of harmonized statistical methodologies - thus improving statistical data comparability; v Helps reducing reporting burden and data redundancy; and, v Enhances the clarification of responsibilities and areas of competence of the parties involved. Q 2008 - European Conference on Quality, Rome, July 2008 24

Statistics Quality Control: the experience of the Banco de Portugal 4. Institutional Cooperation New Statistical National Law (May 2008) Some highlights: v Banco de Portugal became a national statistical authority in the same level playing field as Statistics Portugal, without prejudice of their independence; v This new law emphasizes the importance of ensuring an efficient cooperation and the need for them to develop models of cooperation considered as adequate to fulfil their role in the NSS; and, v The principles of the European Statistics Code of Practice are also formally adopted in this new legal framework. Q 2008 - European Conference on Quality, Rome, July 2008 25

Statistics Quality Control: the experience of the Banco de Portugal 4. Institutional Cooperation In national terms (1) - In the framework of the High Council of Statistics (HCS) – the body on top of the Portuguese NSS; - Other bilateral/multilateral initiatives: 1) 1998 - Protocol with Statistics Portugal (SP) and Ministry of Finance (MF) in the field of IMF Special Data Dissemination Standard (SDDS); 2) 1998 - Protocol on the compilation of Portuguese National Accounts shared responsibility between SP (non financial accounts) and Bd. P (financial accounts) – updated in 2001 and 2007; 3) 1999 – Protocol with SP for the joint conduction from 2000 onwards, in partnership, of a Quarterly Survey on non Financial Corporations; 4) 2004 – Two protocols in the field of Balance of Payments, with SP and General Directorate of Tourism, focusing on a survey of cross-border movement of travellers and a survey on international tourist expenditure (the last was conducted in 2007); Q 2008 - European Conference on Quality, Rome, July 2008 26

Statistics Quality Control: the experience of the Banco de Portugal 4. Institutional Cooperation In national terms (2) 5) 2006 - Protocol with SP, (the first was signed in 2000) for the conduction of a Survey to the Households Indebtedness and Wealth; 6) 2006 – Protocol with SP and MF, on the compilation of General Government Statistics, Public Debt Statistics and analysis of Excessive Deficit Report; 7) 2006 – Joint work with SP for the development of a unique File of Statistical Units for the whole NSS; 8) 2006 – Protocol with SP, MF and Ministry of Justice to define a single electronic data submission with data from the financial statements of non financial corporations – IES (formally created by Decree-Law n. 8/2007 of 17 th January). Q 2008 - European Conference on Quality, Rome, July 2008 27

Statistics Quality Control: the experience of the Banco de Portugal 4. Institutional Cooperation At international level - With ECB and the other EU national central banks in the framework of ESCB; With Eurostat and the EU national statistical offices under the umbrella of the Memorandum of Understanding (Mo. U); In the framework of CMFB (currently the Head of the Bd. P Statistics Department is also part of the CMFB Executive Body); With other international organizations namely IMF, BIS, World Bank and OECD; Technical assistance – several activities in particular, but not exclusively, with the national central banks ant the national statistical offices of the Portuguese-speaking Countries. Q 2008 - European Conference on Quality, Rome, July 2008 28

Statistics Quality Control: the experience of the Banco de Portugal 5. Conclusions and Prospects Ahead The appraisal on the work developed so far is quite significant. Further improvements: v Define a set of structured quality indicators, to be compiled on a regular basis, on the quality of the statistical production; v Improve the already existing quality manuals, exploring further these documents both in a user’s oriented perspective and in a statistical quality control framing; v Enlarge the feedback information currently produced and returned to the reporting institutions; v Define mechanisms to have an adequate feedback from statistics’ users (on statistical data already currently available and on the users’ future needs). Q 2008 - European Conference on Quality, Rome, July 2008 29

Statistics Quality Control: the experience of the Banco de Portugal Thank You for your Attention Q 2008 - European Conference on Quality, Rome, July 2008 30