Accountability Getting it Right in Early Childhood Kathy

Accountability: Getting it Right in Early Childhood Kathy Hebbeler Early Childhood Outcomes Center at SRI International Presented at the Measuring Child and Family Outcomes Meeting, Baltimore 2008

Where are you? Early Childhood Outcomes Center 2

Today we hope to… n n n Begin a conversation about what an ideal EC accountability system would look like Describe some recent developments in EC Accountability Support you as state leader in shaping the EC accountability system in your state. Early Childhood Outcomes Center 3

What is happening out there? n n Incredible expansion of EC programming at the state level Accountability is not going away How best to do accountability is still a topic for hot conversation in early childhood Essential that persons knowledgeable about IDEA programs be part of those conversations Early Childhood Outcomes Center 4

For what do we want to be held accountable? a) b) c) High quality services provided through 619 and Part C Good outcomes for children with special needs and their families Both Early Childhood Outcomes Center 5

And who are “we? ” a) b) c) Administrators and others associated with Part C and Part B 619 programs? Advocates concerned about how young children in our society are faring? Both Early Childhood Outcomes Center 6

Good Accountability requires n Intentionality n Being proactive n Having a vision So what is your state’s vision for your accountability system? Early Childhood Outcomes Center 7

Good Accountability requires. . n n Seeing accountability as a system…. …A group of interacting, interrelated, or interdependent elements forming a complex whole Early Childhood Outcomes Center 8

EC System (or not) Public Health home visiting Child Care Head Start Part C State Pre-K Early Head Start Early Childhood Outcomes Center 619 9

System Under Construction State Pre-K Child Care Part C Transition Accountability 619 Head Start Early Childhood Outcomes Center 10

Components of IDEA Service System n Service Delivery n n n Child Find System entry – eligibility determination Service provision Transition Etc. Supports to Service Delivery n n n Policy, legislation, regulation Professional Development Accountability Etc. n Early Childhood Outcomes Center 11

n n The Accountability System is a component within the Service System within a system Early Childhood Outcomes Center 12

Two Recent Reports Early Childhood Outcomes Center 13

State Early Childhood Accountability and Improvement System A system of standards-based assessments of (a) children’s development and learning and (b) program quality, designed to inform state policy decisions, investments, and improvement efforts for early education programs for three- and four-year-old children, linked to a continuum of kindergarten through third grade standards, curriculum, assessments, and program improvement efforts. National Early Childhood Accountability Task Force Early Childhood Outcomes Center 14

Ch ild O ut co m es State Early Childhood Accountability and Improvement System A system of standards-based assessments of (a) children’s development and learning and Program information (b) program quality, designed to inform state policy decisions, investments, and improvement efforts for early education Purpose programs for three- and four-year-old children, linked to a continuum of kindergarten through third grade standards, curriculum, assessments, and program improvement efforts. Linkag e National Early Childhood Accountability Task Force Early Childhood Outcomes Center 15

Three Primary Building Blocks n n n System Infrastructure Assessment/Program Improvement Approaches Steps toward Coherent Pre. K-Grade 3 Accountability Efforts National Early Childhood Accountability Task Force Early Childhood Outcomes Center 16

Infrastructure …. An underlying base or foundation especially for an organization or system …. The basic facilities, services, and installations needed for the functioning of a community or society National Early Childhood Accountability Task Force Early Childhood Outcomes Center 17

Four Parts of the EC Accountability Infrastructure ü ü Early learning and program quality standards Program rating and improvement system Professional development system Data management and reporting system National Early Childhood Accountability Task Force Early Childhood Outcomes Center 18

Assessment: Importance of Purpose n n n Make purpose of assessment explicit and public Match the assessment strategy to the purpose Administrators, teachers, and assessors should be able to articulate the purpose of an assessment NRC Report - Early Childhood Assessment: Why, What, and How? Early Childhood Outcomes Center 19

Assessment system is part of EC larger system n n n n Standards Assessments Reporting (data base) Professional development related to standards and assessment Procedures to assess opportunity to learn (Program quality) Fair assessment for all children Sufficient resources to implement Ongoing monitoring and evaluation NRC Report - Early Childhood Assessment: Why, What, and How? Early Childhood Outcomes Center 20

Relationship between these two reports n Task Force Accountability Report – n Focus: EC Accountability n n Assessment is a component of an accountability system NRC Report on Assessment n Focus: EC Assessment n Accountability is one purpose for assessment Early Childhood Outcomes Center 21

Components of an Accountability System n n Vision (related to Vision for Service System) Purpose Early Childhood Outcomes Center 22

Articulating purpose for state Accountability System n n Probably goes beyond “Reporting data for the SPP/APR” Needs to be more specific than “accountability and program improvement” Note: Purpose can vary with level; OSEP can have different purposes from state’s purpose(s) as can local programs. Early Childhood Outcomes Center 23

Possible purposes q q q Demonstrate effectiveness of program for policymakers (governor, legislature) Identify local programs in need of improvement Determine whether services are equally effective for different sub-groups, e. g. , q q Children with health problems Low income families, etc. Determine whether services are equally effective across different outcome areas. ____________________ Check all that apply Early Childhood Outcomes Center 24

Communication is key n Purpose(s) needs to be n n In writing n Readily available n n Clearly articulated Widely disseminated No searching. No secrets. No suspicions. Early Childhood Outcomes Center 25

How to Achieve Purpose(s)? 1. 2. Build evidence base so state can reach valid conclusions about service system. Use evidence base to accomplish stated purposes. Early Childhood Outcomes Center 26

Build the evidence base n n FMA* What does state want to know about what? *Findings, Meaning, Action Early Childhood Outcomes Center 27

Evidence about what? n Accountability Task Force: A system of standards-based assessments of (a) children’s development and learning and (b) program quality n Note: This assumes the service system has standards for children’s development and learning and standards for program quality so they can be assessed Early Childhood Outcomes Center 28

Standards for Children’s Learning and Development: Early Learning Guidelines n n n Knowledge, skills, habits, attitudes, dispositions children are expected to acquire by a given point in time Almost every state has them for 3 -5 year olds Many states already have them for 0 -3; others with them under development. Early Childhood Outcomes Center 29

Early Childhood Outcomes Center 31

Early Learning Guidelines n n Developed for multiple purpose Considerable variations in content, format, specificity across states Stand as public statements of what we want children to do at various ages Public policy statement of developmental expectations Early Childhood Outcomes Center 32

Early Childhood Outcomes Center 33

Early Childhood Outcomes Center 34

What we know (WWK) n To paraphrase Thomas Jefferson, “We hold these truths to be evidence-based…. Early Childhood Outcomes Center 35

WWK: Children’s development proceeds according to a general timeline n n n Competencies are acquired within an age range; developmental milestones Some skills precede other skills Developmental timeline is the basis for screening and assessment tools in early childhood; also provides foundation for ELGs Early Childhood Outcomes Center 36

WWK: Current developmental status predicts future developmental status n n Current functioning is a statement about where a child is Also, a prediction or odds for where the child will be in the future. Basis for developmental trajectories Goal of intervention is to change the odds. Early Childhood Outcomes Center 37

WWK: There is a lot of variation in children’s development/learning but some children’s development/learning is problematic n n Some is “just variation” Some is troublesome – because it bodes ill for the child’s future (bad odds) Critical that we can distinguish Policy response: n n Early intervention and early childhood special education State criteria for delay – the point at which the delay is so substantial, the child is not likely to “outgrow” it. Eligibility for IDEA services. Early Childhood Outcomes Center 39

Thinking About Age Expectations Ageexpected skills and behavior Movement away from age-expected Movement toward ageexpected Early Childhood Outcomes Center 40

Inequality at ECLS-K data from Inequality at the Starting Gate

inequality ECLS-K data from Inequality at the Starting Gate

WWK: There is a lot of variation in children’s development/learning but some children’s development/learning is problematic n Policy response: n n State Pre-K n n Head Start Child Care Quality Standards Goal: Alter the trajectory. Change the odds Early Childhood Outcomes Center 43

WWK: There is a lot of variation in children’s development/learning but some children’s development/learning is problematic n n n Policy response: Early learning guidelines [Hopefully] Grounded in what we know about development, ELGs lay out developmental expectations Provide guidance to keep children “on track” or get them back on track so they arrive at kindergarten with a certain set of skills Early Childhood Outcomes Center 44

Understanding the difference between odds and a sure thing n n Need to understand how to interpret data on children’s current functioning Not wise to obsess on slight degrees of variation Not wise to ignore substantial deviation (“grow out of it”) A child “off track” is a prediction begging to be proven wrong – if effective intervention is provided (WWK: Early intervention works. ) Early Childhood Outcomes Center 45

Application of ELGs to children with special needs n Do ELGs apply? Are they useful? n n They are statements of what is expected at given ages Parallels with the application of standards for children with disabilities in K-12? n n n At a individual child or classroom level? At a state level? Have fought for students’ rights to be held (and compared) to the same standards as all students Standards provide access to higher expectations for students with disabilities Could ELGs be a common currency in moving toward unified EC system? Early Childhood Outcomes Center 46

Application of ELGs to children with special needs n n n DEC Recommended Practice: A 41. Professionals monitor child progress based on past performance as the referent rather than on group norms. This RP is speaking to norm-referenced tests. Implication is that the only thing we want to look at is child compared to him’/herself. ELGs are not group norms – but they are group standards. Can ELGs provide directions to Part C and 619 programs for age expectations? Early Childhood Outcomes Center 47

Lessons ECO has learned n EI and 619 providers need to know more about typical child development n n n And so do child care providers, etc… Some lack of clarity over how to determine age expectations for OSEP reporting purposes Provider discomfort over how to talk to parents about current level of functioning when it is low Early Childhood Outcomes Center 48

n n n Early Childhood Outcomes Center PA’s red flags “…help adults gain a sense of whether additional screening or evaluation should be sought if a particular skill has not been acquired within a specific developmental age span. ” Help adults watch for possible delays. 49

What kind of evidence about children’s learning and development? n Task Force Recommendation: n n Measure progress and status. OSEP reporting does both! n Checks on status at 3 years and 5 years. n Looks at progress over time in program. Early Childhood Outcomes Center 50

Evidence about what? n Accountability Task Force: A system of standards-based assessments of (a) children’s development and learning and (b) program quality n Recommended expansion: Include family outcomes. Early Childhood Outcomes Center 51

What do we want to know about programs? n n Both ATF and NRC reports speak to looking at program quality Possible addition: Reframe to be program characteristics and program quality n Program characteristics: n n n Hours, months of operation Program model Services offered Curriculum Program quality measures Early Childhood Outcomes Center 52

What do we want to know about programs? n Important difference in “program” for C/619 compared to general EC n n n Some program measures apply to a group of children (e. g. , measures of classroom quality) Some are individualized (e. g. , receipt of speech therapy) Important implications for measurement Early Childhood Outcomes Center 53

What do we want to know about programs? n What is the child’s program? n n Services Minutes of services Location of service provision What is the quality of the child’s program? n How should we assess quality, especially of the individualized component? Early Childhood Outcomes Center 54

Program quality measures n n Examples: Early Childhood Rating Scale (ECERS), Early Language and Literacy Classroom Observation (ELLCO), Classroom Assessment Scoring System (CLASS) See NRC report for a comprehensive review of program quality tools Early Childhood Outcomes Center 55

Evidence about what? Possible Modifications to ATF Recommendations n Child and family outcomes n Program (Group, individual) n n n Characteristics Quality Infrastructure supporting the program n Legislation policies n Professional development, etc. Early Childhood Outcomes Center 56

System for Producing Good Child and Family Outcomes High quality services and supports for children 0 -5 and their families Good outcomes for children and families

System for Producing Good Child and Family Outcomes Adequate funding Good Federal policies and programs Good Local policies and programs Good State policies and programs Strong Leadership Prof’l Development • Preservice • Inservice High quality services and supports for children 0 -5 and their families Good outcomes for children and families

Ecological Model of Child Development Child Family Early Childhood Outcomes Center 59

Ecological Model of Child Development Child Family Community Early Childhood Outcomes Center 60

Ecological Model of Program Implementation n n Multiple forces impact how programs are implemented Important to know how your programs look Also, important to know what leads them to look like this Forces acting on programs provide levers for program improvement Early Childhood Outcomes Center 61

Concluding Caveat n n n NEVER ever reach a conclusion about program effectiveness based on child outcome data alone Best practice (at all levels of decisionmaking): Multiple sources of information Conclusion: Accountability system needs evidence about outcomes AND evidence about program Early Childhood Outcomes Center 62

Components of an Accountability System n Vision (related to Vision for Service System) n Purpose n Evidence Base Early Childhood Outcomes Center 63

How to get the evidence Good measurement procedures = procedures capable of producing high quality date 1. n Good system design Fidelity of implementation = procedures are implemented as intended 2. n Good quality assurance Early Childhood Outcomes Center 64

Components of an Accountability System n Vision (related to Vision for Service System) n Purpose n Evidence Base n Use of evidence to accomplish purposes Early Childhood Outcomes Center 65

Use of evidence* n Reaching valid conclusions (M) n Taking effective action (A) *Addressing the purposes Early Childhood Outcomes Center 66

Taking the fear out of accountability What are we afraid of? n Findings/Evidence n Meaning/Conclusions n Action n Continuum of actions n What are the options on the continuum? Early Childhood Outcomes Center 67

Accountability Nightmare n n Meaning – reaching the wrong conclusion Action – leading to actions that will ultimately harm children and families Early Childhood Outcomes Center 68

Avoiding the nightmare n Type of wrong conclusion n False negative –program is ineffective when it is effective n n Overall For some types of children In some areas False positive – program is effective when it isn’t Early Childhood Outcomes Center 69

Alignment of Conclusion and Reality Effective Early Childhood Outcomes Center Effective Correct! False Positive Ineffective Conclusion Ineffective False Negative Correct! 70

Avoiding the nightmare n Good evidence n Credible n Multiple pieces of evidence n n Evidence about child outcomes Evidence about program factors (quality)* *See Accountability Task Force Early Childhood Outcomes Center 71

Need for outcomes AND program data n Avoid both false negatives and positives by looking at multiple pieces of evidence n n n Triangulation Do multiple sources of evidence point to the same conclusion? The outcomes data; the program data; data on who is being served, etc. Early Childhood Outcomes Center 72

Avoiding the nightmare: Solid Infrastructure for an Accountability System n n n Clear vision Clear articulation of purpose and public statement as to how the data will be used Credible evidence (drill down here…) Early Childhood Outcomes Center 73

Clear statement of possible future actions What will be the consequences of a negative finding? n Sanctions? ? n The ultimate worst nightmare is the prospect of a false negative combined with sanctions n Never consider sanctions based on child outcomes data alone (or any single piece of evidence) Early Childhood Outcomes Center 74

Clear statement of possible actions n Supports to improve weak programs or weak program areas n n n Site visits Individualized technical assistance Professional development plan Etc. Note: If the wrong program is identified (false negative) and provided resources, no harm comes to children and families in that program but it is a waste of resources. Early Childhood Outcomes Center 75

Clear statement of possible actions False positives n n n Sort of where we are now: programs, investments assumed effective. . but some of them probably aren’t Consequences n n Short changes children and families Some children and families will not experience the kinds of outcomes they could have with a quality program. Early Childhood Outcomes Center 76

Sidebar thought n n Why are we more worried about false negatives than false positives? It would be damaging to children and families to defund an effective program but it also is damaging to have children and families receiving ineffective services…. Early Childhood Outcomes Center 77

Sidebar thought #2 n n n What about comparing states [programs]? What kinds of conclusions will OSEP reach? What are the actions/consequences that will follow? OSEP has no intention of comparing states on outcomes Early Childhood Outcomes Center 78

What about comparing? n n Can’t afford to be naïve Once data are out there, anyone can make comparisons (state to state; within state, local to local) Comparing is not the problem; invalid conclusions (especially false negatives) are the problem Using comparisons to identify weak programs is what accountability and program improvement is supposed to do Early Childhood Outcomes Center 79

Best protection against false negatives (and false positives) n Good evidence base n Multiple sources of information on: n n Children and families Programs Infrastructure Credible, well implemented measurement strategies Early Childhood Outcomes Center 80

Putting Good Accountability to Work 1. 2. 3. 4. Build the infrastructure for producing a body of evidence that will allow valid conclusions Compile the evidence and carefully reach conclusions Act on those conclusions to build a better system of services Repeat 2 and 3 indefinitely. Early Childhood Outcomes Center 81

Good EC Systems n Build on what we know (WWK) n About children’s learning and development n n n Multiple influences on development About adult learning About innovation, implementation Etc. (this is another way of saying they are evidenced-based) To provide high quality services and supports Early Childhood Outcomes Center 82

Accountability Task Force Recommendation n States should develop a single unified and coherent system of standards, assessments, data and professional development efforts across early childhood categorical programs and funding streams. **Alignment** **Systems Early Childhood Outcomes Center Integration** 83

Turning an EC Non-System into a System n n What resources exist in your state to promote good outcomes for children birth to 5? Which of these resources work together? n How? n n n Share a vision of common outcomes? Build a seamless system for families? Joint professional development? Shared vision for transition? Unified accountability system? Early Childhood Outcomes Center 84

Producing Good Outcomes for Young Children with Disabilities n n Proposition: Good IDEA services are not enough to effectively change the odds for some(? )/many(? ) young children with special needs. Need the entire EC System to be functioning effectively. Early Childhood Outcomes Center 85

Producing Good Outcomes for Young Children with Disabilities n n n Need more forces at work to change the odds and help children meet developmental expectations. Need good child care (and an accountability system) Need good Pre. K and Head Start programs Early Childhood Outcomes Center 86

Producing Good Outcomes for Young Children with Disabilities n n Need the EC services to work together Need an accountability system that can look at the entire EC system Early Childhood Outcomes Center 87

For what do we want to be held accountable? a) b) c) High quality services provided through 619 and Part C Good outcomes for children with special needs and their families Both Early Childhood Outcomes Center 88

And who are “we? ” a) b) c) Administrators and others associated with Part C and Part B 619 programs? Advocates concerned about how young children in our society are faring? Both Early Childhood Outcomes Center 89

References n Task Force on Accountability (2007): Taking Stock: Assessing and Improving Early Childhood Learning and Program Quality n n http: //www. pewtrusts. org/our_work_detail. aspx? id=102 National Research Council; Snow & Van Hemel (2008): Early Childhood Assessment: What, Why, and How? n http: //www. nap. edu/catalog. php? record_id=12446 Early Childhood Outcomes Center 90

References Web site on state Early Learning Guidelines n n n http: //www. ccsso. org/content/PDFs/State_ECstandards. MA TRIX 11. 27. 07. pdf Research on State’s ELGs –Search on “Catherine Scott Little” on Google Scholar Lee, V. E. , & Burkam, D. T. (2002). Inequality at the starting gate: Social background differences in achievement as children begin school. Washington, DC: Economic Policy Institute. Early Childhood Outcomes Center 91