European Conference on Quality in Survey Statistics

European Conference on Quality in Survey Statistics

Legislation, Codes of Practice and the Quality of Official Statistics Len Cook New Zealand Len Cook, New Zealand Legislation and quality in official statistics Cardiff, April 2006

The role for society of the statistician "The statistician keeps his fingers on the pulse of humanity, and gives the necessary warning when things are not as they should be. " Quetelet

The place of official statistics in timely and decisive action ? Official statistics Research Theory Policy Framework Case studies Experience Anecdote Myth Stereotype

How much is confidence in official statistics in a state of change? • Increased expectations • Changing World • Greater use • Privacy • Quality demands • Technology • Accessibility • Advances in methods and practice

Prospects and trade-offs for the Global statistical system Competitive & influential Being able to be decisive and timely in actions Prospects Sense of Place of Official Statistics Delivery Performance Trade-offs & risks in reputation & quality Trade-offs

Summary of key elements of statistical legislation Unlikely to be in legislation Strengthened by legislation Usually in statistical legislation Quality standards Roles of Head of National Statistical Institute, and responsible Minister Authority to obtain individual information Relevance Accessibility Protection of individual responses from disclosure Responsiveness Managing respondent load Objectivity of methods Government/ community Co-ordination across balance government (survey approval) Impartiality of results, and selection of measures Efficiency Capacity to link records Definition of official statistics

Responsiveness of statistics and the nature of the professional/political boundary x service agreement (key users) x structured user engagement/interaction x political consensus x community partnerships (ethnicity) RESPONSIVENESS x manage obsolescence (resources) x benchmarking, reviews x performance targets x quality framework x other legislation x international standards x political oversight x comprehensive statistical legislation x codes of practice PROFESSIONAL POLITICAL

The diminishing capacity to make quality assessments x service agreement (key users) x structured user engagement/interaction QUALITY INFORMATION x political consensus x community partnerships (ethnicity) x manage obsolescence (resources) x benchmarking, reviews x performance targets x quality framework x other legislation x international standards x comprehensive statistical legislation x political oversight QUALITY MEASURES x codes of practice PROFESSIONAL POLITICAL

The gap between capacity to measure quality and expectations x service agreement (key users) x structured user engagement/interaction QUALITY INFORMATION x political consensus x community partnerships (ethnicity) x manage obsolescence (resources) USER EXPECTATIONS OF QUALITY x benchmarking, reviews Knowledge gaps x performance targets x quality framework x other legislation x international standards x comprehensive statistical legislation x political oversight QUALITY MEASURES x codes of practice PROFESSIONAL POLITICAL

The growing gap between expectations and quality measures QUALITY INFORMATION USER EXPECTATIONS OF QUALITY Knowledge gaps QUALITY MEASURES PROFESSIONAL POLITICAL

Legislation and quality x service agreement (key users) Statistics released without being undermined Objective selection of measures x political consensus x structured user engagement/interaction Access to administrative records Increased record linkage Strong survey response rates x community partnerships (ethnicity) Continually robust survey frames Widen unit record access for research x manage obsolescence Expand small area statistics (resources) Reduce risk of accidental disclosure x benchmarking, reviews x performance targets x quality framework x other legislation x political oversight x international standards x comprehensive statistical legislation x codes of practice PROFESSIONAL { Authority to act Disclosure protection Objectivity of methods Impartiality of results POLITICAL

Parliament is the place to recognise when ministerial authority has been transferred Ministers are elected to Parliament to make decisions for their peoples. If the authority to act in one sphere of responsibility, official statistics, is to be constrained, it should be recognised by Parliament, in the legislation of Parliament. Without this recognition, deciding what accountability will remain with Ministers may lack consensus, clarity, or consistency. Often statistical legislation will give a clear delineation in the responsibilities of the country’s chief statistician and the respective minister in the government responsible for it. This explicit relationship can be taken as that which applies to all Ministers and to all official statisticians. Comprehensive statistical legislation reduces expectation that each statistical decision or statistical event can be managed to have a good political outcome.

The place of legislation in the protection of confidentiality • Obligation • International security • Protecting archives • Data-matching and increased register use • Challenging perceptions • Quality of protection • Increased capacity to access records from – Citizens (high response rates) – Business (tax records and market sensitive information) – Government (administrative records) • Capacity to protect (court and police requests) • Limit power of situational ethics • Limit spread of less relevant oversight

The place of legislation in ensuring the objectivity of methods • • • • Selection of methodology by statistician Highly political interventions should be by legislation or judicial process Fitness for use in unknown situations outside design criteria International coherence Open peer review of practice Access to unit record data Immediacy of linkage to politically sensitive outcomes places a greater constraint on the reflective, consultative processes of the statistician Need high standing of role of National Statistician High rate of obsolescence of methods and conceptual reorientation External drivers grown tax liability, benefit eligibility, public funding, EU comparability, international quota Growth in international frameworks, Demand for cohesiveness across independently managed statistical sources Emphasis on residuals and balances from large gross flows Performance targets selected by Ministers

The place of legislation in impartiality of results and practices • publish information about quality without approval • independent publication of compendia on any topic • have sufficient finance for assessing quality • judgement on early access and immediacy of policy statements • choice of comparisons • obligation of consistency • respondent relevant description and concepts • Indigenous recognition, language use • No intervention in senior appointments • Limit damage caused by early access • Limit juxtaposition of statistical release and policy reaction • Independence of selection of comparisons

The place of legislation in impartiality of results and practices (contd) • Release of information about the quality of results can affect confidence in policies based on them • Timing of release and quality standards brings tensions that can result in belief in political intervention • Need to publish methods and available evidence used in judgements even if confidential for other reasons • Reliance on quality not being a sensitive concern

Can legislation advance quality, given that it is a matter of culture and method? Quality focus in statistical offices sometimes drives major strategies - statistical modernisation (UK) But mostly where staff have a quality focus they make sure that: • Innovation often results from quality failure • Value added by users does not damage the contributing products • Avoidable limitations to quality are fixed

Innovation often results from quality failure • Unanticipated revisions • Delay in applying benchmark series revisions to rebased series • User enquiries cannot be answered • Users find errors • Users get inconsistent answers on different occasions • Derived measures use sources inconsistently • Users find inconsistencies across series that require revisions • Users fail to receive statistics as ordered • Instability and inconsistency from results of related series from different sources • Known policy applications delayed for statistical faults

Value added by users may reduce trust in NSI • Users place statistical results into standard databases • Users add metadata to series • Users add to coherence with related series • Users add explanations of series volatility • Users need to check unusual results

Avoidable limitations to quality get fixed • Inadequate range of quality measures • Expected revisions delayed • Insufficient length of series • UK or only some of England, Wales and NI • Inconsistent results across related series from different sources • Inexplicable delays in getting results

What is essential alongside statistical legislation? • • • • Strong formal user involvement Rich publishing culture High degree of analysis, including complex derived statistics such as productivity, poverty and fiscal incidence Access to unit record datasets Large number of processes for providing transparency Regular publication of measures of cohesiveness/ confrontation across statistical sources Strong peer review processes Regular cross country benchmarking activity High degree of linkages across the collection and integrating frameworks produced by the system (Nat accounts, demography) Parliamentary ownership of system Strong methodology teams Engagement in cross country developments of statistical frameworks and standards International exchange of staff from both professional and operational areas of statistics

Codes of Practice and quality frameworks • We may be overstating our capacity to measure quality • We may not have sufficient resources to systematically assess quality through a comprehensive quality framework • Hard to measure statistics that are very important may never be assessed • We could give more attention to assessing the measurability of key phenomena, where measures greatly influence public life • We may be undervaluing our judgements about quality in difficult uses of statistics by giving emphasis to tangible quality measures

Conclusion • Legislation plays a key role in quality, but cannot substitute for user engagement, transparency and judgements on measurability. • Legislation affirms the fundamental obligations on the statistician, and the authority to obtain information that is central to the job • The coming decades will see great change in our capacity to retain confidence in the quality of our work. • We will give more confidence through the certainty of our processes • The accessibility of statistical sources will be the most significant source of innovation, once technological transformations have been achieved

European Conference on Quality in Survey Statistics