National Immunization Survey Data Quality and Public-Use Data

National Immunization Survey: Data Quality and Public-Use Data Files Meena Khare, National Center for Health Statistics Michael P. Battaglia, Abt Associates Inc. July 16, 2002

Outline v National Immunization Survey v Quality of the data v Confidentiality issues v Public Use Data Files (PUF) v Summary 2

National Immunization Survey (NIS) v Large ongoing RDD survey, conducted by CDC since 1994 v Measures vaccination coverage among children aged 19 -35 months at National, State, and Urban area levels (78 IAP areas) v Monitors Healthy People 2000 and 2010 goals of immunization coverage u >90% Coverage: 4 DTP, 3 Polio, 1 MMR, 3 Hep. B, 3 Hib, and 4: 3: 1: 3 series v Monitors introduction of new vaccines v <4% of households in the U. S. contain child of age 19 -35 months

NIS Operations

NIS Data Collection v Household CATI Screener and Interview u Parent/Guardian (most knowledgeable person) u Socio-demographic information: mother and child u Shot card use or memory recall u Vaccination dates (shot card only) u Provider’s contact information with consent v Provider Record Check Study (mailed IHQ) u Provider’s office (e. g. , staff, nurse, manager) u Mail, Fax, Telephone u Completed IHQ or copy of medical records u Matched on DOB, Gender, Name u Vaccination history u Provider’s information

Goals of Quality Control in the NIS v. Evaluate and improve quality of the information collected v. Reduce/eliminate discrepancies v. Improve quality of the estimates v. Reduce potential bias in official estimates of vaccination coverage rates

Quality Control Procedures used in the NIS v Extensive monitoring and QC review of the data collected from CATI and mailed IHQs v Automated edit software for processing combined household and provider data v Manual ‘Matching Sheet’ review of discrepant cases v Resolution by using guidelines developed by expert immunization program managers

Review of Data Errors in the IHQ v Data Entry Validation u Double data entry u Approximately 400 forms reviewed/quarter u Error rates ranged from 0. 23% - 0. 67% (~30, 000 fields with data, NIS Q 4/1999 -Q 4/2001) v ~350 -1000 manual Matching Sheet reviews/quarter for discrepant data v Most errors in dates are found in the original provider-reported records v <1% children had data errors

Observed Discrepancies in Dates v DOB u Between household (HH) and provider(s) u Between multiple providers for a child v Shot dates u Between HH and provider(s) u Within a single provider u Between multiple providers for a child

Matching Sheet Review, 2000 NIS v DOB and Shot dates u 122 children classified as age-ineligible u 69 IHQs filled out for the ‘wrong’ child u 21 children had changes in best DOB u 2163 children had shot dates before DOB or vaccination interval < 1 month, or other shot date edits v Hepatitis B ‘birth’ dose u 361 children had missing provider ‘birth dose' of Hepatitis B and providers had checked the ‘given at birth’ box on IHQ l l 34 children had imputed dates from the household shot card 327 children had imputed dates from the distribution of provider-reported dates for the birth dose

Results of Quality Control Evaluation v No single data source is 100% complete and accurate v Some discrepancies in DOB and vaccination dates/number of doses could not be verified and remained after edits v Number of discrepancies increased if more than one immunization provider per child (1. 35 providers per child in 1999 NIS ) was identified v Errors in Original Immunization records Locating records for the correct child Transcribing shot dates to the IHQ Illegible dates, transposed dates, missing day/month, incorrect dates (especially year) u Shot date interval< 1 month, or shot dates<DOB u Missing ‘first’ Hepatitis B dose given at birth u u

‘Best’ Vaccination Value Research v Currently, coverage estimates from the provider data are used as official estimates v The goal of ‘best’ value research is to obtain the most complete and accurate immunization data possible from two sources u Supplement incomplete provider data with household data for children who are UTD from ‘shot card’ u Substitute completely missing provider data with household data for children who are 4: 3: 1: 3 UTD from ‘shot card’

NIS 2000 Sample v 35, 960 households with eligible children 19 -35 months v 33, 477 (93. 1%)of eligible households with completed interviews v 34, 087 age-eligible children with completed interviews v 22, 958 (67. 4%) children with completed interviews and ‘adequate’ provider data

Sources of Immunization Histories, 2000 NIS Source Household data from Shot Card ‘Adequate’ Provider data Yes n NO % n Total % n % Yes 11, 963 73. 8 10, 995 61. 5 22, 958 67. 4 No 4, 249 6, 880 11, 129 32. 6 Total 16, 212 47. 6* * row percents 26. 2 38. 5 17, 875 52. 4* 34, 087 100. 0

Assessment of Provider-Reported 4: 3: 1: 3* Coverage Estimates among Children with Household Data, 2000 NIS Source of HH reports and measures of quality Shot Card ‘Memory’ Recall** 11, 963 5, 526 Sensitivity 67. 1 44. 8 Specificity 61. 8 65. 3 Misclassification (prov) (HH UTD)/ Prov NUTD) 7. 4 8. 7 Overall agreement 66. 1 49. 9 Overall disagreement 33. 9 50. 1 Net Difference (HH Cov - Prov cov) -19. 1 -32. 7 Sample Size *4 DTP/3 Polio/1 MCV/3 Hib ** Other 5, 469 non-shot card children have missing 4: 3: 1: 3 UTD status

Comparison of Vaccine-specific Coverage Rates by Source of Immunization Histories, 2000 NIS Best Value: Combined Provider and household ‘shot card’ data Revised Best Value: After adjusting Best Value for missing provider data among non-shot card children 17

NIS Data Files and Estimates v In-house analytic files v Public-use data files (PUF ) v National, State, and Urban area level estimates of vaccination coverage u Internet: www. cdc. gov/nip/coverage u MMWR u Journal Articles u Articles in Proceedings u Presentations at National and State Conferences u Publications are listed on NIS Web site

Confidentiality Issues v Legal obligation to protect confidentiality of respondents and reduce risk of disclosure v Sample size (4 quarters) u u ~ 440 children per IAP with interview data ~ 290 children per IAP with provider data u u u State, Urban Areas Census Region Smaller geographic areas (e. g. , Zip code, county) u u u Age, gender, race/ethnicity WIC participation, Poverty status Minimum population size>5 in unique cells size v Only telephone households (90% HHs with telephone) v Rare population (children 19 -35 months) v Geographic identifiers v Socio-demographic details v Analytic cells and cell sizes (within IAP area) v Date of birth, shot dates, interview dates v Clearance by the NCHS Disclosure Review Board (DRB)

The NIS PUF v Public-use data files (1995 -2000): Child-level records with 78 IAP area (state and urban) identifiers (2001 NIS PUF by Fall 2002) v Approximately 35, 000 age-eligible children with household interview data v Approximately 23, 000 children with household interview and ‘adequate’ provider data; on average 295 children per IAP area v PUFs released on the Internet and CD-ROMs u Internet : www. cdc. gov/nis (1995 -2000) u CD_ROMs No 1 -5

Contents of The NIS PUF v Household Interview data items u Unique sequence ID, age at interview (months), gender, race/ethnicity, … u Mother’s age group, education, marital status, … u WIC participation, income, poverty status, … u Vaccine specific immunization history (#doses, up-to-date status) u IAP area and State identifiers, Census Region v Provider data items u Vaccine-specific Immunization History: age at vaccination(days/months), number of doses, up-to-date status (UTD) u Providers’ information (e. g. , facility type, VFC participation)

Contents of NIS PUF (cont. ) v Composite vaccine-specific and vaccine series flags with up-to-date status v Sampling weights and key analytic variables u Sampling weights: Children with household interview (HY_WGT) Children with Provider data(W 0) u Stratum + PSU identification variables

Summary v Complicated Childhood Immunization Schedule v No single source is 100% complete or accurate v <1% children were observed with discrepancies in household and/or provider reported data v Data reported from written ‘shot card’ are of high quality; Shot card use varies by State/IAP area v High vaccine-specific agreement in the UTD status between HH ‘shot card’ and provider report(s) when HH reported UTD

Summary v Substantial underreporting in vaccination coverage rates from the household reports v Strongly supports the use of provider-reported immunization histories in computing vaccination coverage estimates v Official estimates from provider reports may slightly underestimate coverage rates due to missing provider data; however, provider data produce comparable estimates across IAP areas

Visit the Web Sites v. For information on the NIS PUF and coverage estimates uhttp: //www. cdc. gov/nip/coverage