Developing NAEP Frameworks A Look Inside the Process

Developing NAEP Frameworks: A Look Inside the Process Mary Crovo Deputy Executive Director National Assessment Governing Board November 17, 2007

NAEP Frameworks: Governing Board Responsibilities in Current NAEP Legislation ® select the subject areas to be assessed ® develop assessment objectives and test specifications that produce an assessment that is valid and reliable, and are based on relevant widely accepted professional standards ® develop a process for review of the assessment which includes the active participation of teachers, curriculum specialists, local school administrators, parents, and concerned members of the public (P. L. 107 -279)

Governing Board Policy: Framework Development ® WHAT - comprehensive, inclusive, and deliberative process to determine the content and format of NAEP subject area assessments ® WHO - educators, testing experts, state and local policymakers, researchers, business representatives, military representatives, members of the public; key “users and consumers” in the subject area ® HOW – Steering and Planning Committees; broad-based input and review from NAEP stakeholders; input and guidance from ADC and the Board

Overview of the NAEP Framework Proce Procurement Contract Award Board Input Planning Document & Committee Composition Framework and & Specifications Framework Specifications Development Framework Review Process Board Action Framework & Specs to NCES

NAEP Framework Components WHAT ® ® Content areas and objectives for each NAEP grade Process or cognitive dimension HOW ® ® Types of test questions Distributions ® ® ® by content area by cognitive dimension by item type Administration procedures, delivery mode Sample test questions HOW MUCH ® ® Sample scoring rubrics Preliminary achievement level descriptions

NAEP Specifications ® Audience ® NCES and NAEP assessment development contractors ® Purpose ® to provide a detailed blueprint for constructing the assessment ® Framework and Specifications are both documents public

Developing the Framework & Specifications ® Charge to the project Steering Committee ® Issues Paper ® Committee meetings ® ® ® Consideration of research and key documents Drafting of Framework and Specifications Ongoing ADC and Board review, input, guidance

Framework Process Balances Consideration of: Research and Best Practice Research on student learning, cognitive development and assessment ® Current instructional and measurement issues ® Curricular reform efforts ® Standards and Assessments ® ® State standards and assessments International standards and assessments (e. g. , PIRLS, PISA, TIMSS) ® Professional association standards (e. g. , national content standards)

Framework Review Process ® Framework reviews in diverse venues across the U. S. ® External review panel ® Board action on Framework and Specifications ® Framework & Specifications sent to NCES for assessment development ® Framework dissemination strategies

Framework Process Balances Consideration of (continued) Other considerations Key documents having national or international significance ® Preparedness reports and research (grade 12) ® Previous NAEP Frameworks ® Following vs. leading the field (given 5 -6 year NAEP development cycle) ® ® Assessment content Assessment format/delivery Resources

Lessons Learned Participants Committee membership – cast a wide net ® Framework review – use broad outreach with diverse venues and audiences ® Often 1, 000+ individuals involved per Framework ® Process Project timeline – 18+ months ® Scope of the project, focus on: ® ® ® What NAEP can do well What NAEP is not designed to do Realities, constraints, and timing ® Resolving difficult issues – trust the process ® ® iterative, not linear

New Challenges in NAEP Framework Process ® Assessing 12 th grade preparedness ® Considering the role of technology ® Maintaining trend lines ® Increasing framework dissemination ® Funding for new frameworks

NAEP Related Resources For further information regarding the Framework process, policies, and publications, visit these websites: ® ® ® http: //nces. ed. gov/nationsreportcard/frameworks. asp http: //nces. ed. gov/nationsreportcard/frameworkhow. asph http: //nces. ed. gov/nationsreportcard/tdw/item_development/20 00_2001/framework. asp http: //www. nagb. org/policies/pl-index. htm http: //www. nagb. org/publications/frameworks. htm