Evidence-based Professional Development 2011 SPDG Regional Meeting Feb

Evidence-based Professional Development 2011 SPDG Regional Meeting Feb 2011 Washington, D. C. Michelle A. Duda, Ph. D. Melissa Van Dyke, LCSW Chris Borgmeier, Ph. D. Susan Davis, NC SPDG Margaret Mc. Glinchey, MI SPDG

A Few Questions: § What do we mean when we use the term, “professional development? ” § What outcomes are we attempting to achieve? § What do we know about the professional development strategies that are likely to achieve particular outcomes? § How can we monitor and improve the quality of professional development over time?

Common PD Challenges Overcoming Barriers with § § Common language Common frameworks Defined approach (best practices) Common measures (commitment to continuous quality improvement)

Overcoming Barriers : COMMON LANGUAGE

Training § “the process of bringing a person to an agreed standard of proficiency, by practice and instruction” http: //www. thefreedictionary. com/training § “the acquisition of knowledge, skills, and competencies as a result of the teaching of vocational or practical skills and knowledge that relate to specific useful competencies. ” http: //en. wikipedia. org/wiki/Training

Adult Learning Defined § “a collection of theories and methods for describing the conditions under which the processes of learning are optimized (Merriam, 2001; Trotter, 2006; Yang, 2003). ” Trivette, Dunst, Hamby, O’Herin, 2009, p. 1

Professional Development "Professional development. . . goes beyond the term 'training' with its implications of learning skills, and encompasses a definition that includes formal and informal means of helping teachers not only learn new skills but also develop new insights into pedagogy and their own practice, and explore new or advanced understandings of content and resources. ” Modified from http: //www. ncrel. org/sdrs/areas/issues/educatrs/profdevl/pd 2 prof. htm

Professional Development (cont. ) “. . . [This] definition of professional development includes support for teachers as they encounter the challenges that come with putting into practice their evolving understandings about the use of new skills and technology. ” Modified from http: //www. ncrel. org/sdrs/areas/issues/educatrs/profdevl/pd 2 prof. htm

Reflection: Language Barriers § How has issues with terminology, definition or language impacted your work?

Overcoming Barriers : COMMON FRAMEWORKS

Implementation Science Longitudinal Studies of a Variety of Comprehensive School Reforms Effective Interventions Actual Supports Years 1 -3 Outcomes Years 4 -5 Every Teacher Trained Fewer than 50% of the teachers received some training Fewer than 10% of the schools used the CSR as intended Every Teacher Continually Supported Fewer than 25% of those teachers received support Vast majority of students did not benefit Aladjem & Borman, 2006; Vernez, Karam, Mariano, & De. Martini, 2006

What Works § Implementation Drivers § Common features of successful supports to help make full and effective uses of a wide variety of innovations

rs mp ete nc Core Implementation Components Institutional capacity to support staff in implementing practices with fidelity e riv Co D on ati niz ga Or Staff capacity to support students/families with the selected How: practices rs Program/Initiative (set of practices) riv e What: Improved Outcomes for. . . y D Why: Leadership Capacity to provide direction/vision of process © Fixsen & Blase, 2008

D on riv e y D ati niz ga How: Or Professional Development rs Program/Initiative (set of practices) rs Core Implementation Components Institutional capacity to support staff in implementing practices with fidelity e riv mp ete nc What: Improved Outcomes for. . . Co Why: Leadership Capacity to provide direction/vision of process © Fixsen & Blase, 2008

Improved Outcomes for. . . Program/Initiative (set of practices) Performance Assessment (Fidelity) s riv e y D mp ete nc Co Integrated & Compensatory Facilitative Administration er riv Selection D on ati niz ga Training Systems Intervention Or rs Coaching Decision Support Data System Leadership Adaptive Technical © Fixsen & Blase, 2008

Why Focus First on Teachers’ Behavior? § § In education, the Teacher IS THE INTERVENTION § Create the conditions under which teachers can do the right thing for the right reason at the right time to maximize positive results § Wide ranging inputs (individuals with all their past history and current realities) § Build alignment of implementation practices and outcomes on what needs to happen at the point of the “learning exchange” Build supports in relationship to what the Teacher needs to be competent

Participants of Professional Development • Think about those who are the participants in (receivers of) your professional development activities… • What features would you suggest be in place around the Competency, Organization and Leadership drivers so that professional development is provided in a “host environment” to support sustainability and promote fidelity? Improved outcomes for students Participants with improved knowledge and skills around identified practice(s)

Providers of Professional Development • Think about those who are the providers of your professional development activities… • What features would you suggest be in place around the Competency, Organization and Leadership drivers so that your providers conduct quality professional development? Participants with improved knowledge and skills around identified practice(s) Professional development providers

The NCSIP Process: Focus Of Work Research. Based Practice Staff Training On-site Fidelity Checks On-site Program Reviews Student Progress Evaluation

Overcoming Barriers: DEFINED APPROACH “Best Practices”

The BIG QUESTION: What outcomes are you hoping to achieve?

Use training “alone” strategies if… § You want to disseminate new information § § You want to create “buy in” § You are talking with a broad audience You want to clarify “truths” and dispel “myths”

Use training “alone” strategies … § To increase knowledge about the effective program or practice related to. . . Ø underlying theory of change Ø intervention components Ø rationales of key practices § To increase familiarity with the use of new skills § To increase awareness of what it feels like to begin to use new skills and to receive expert feedback

Training “Best Practices” § § Theory grounded (adult learning) § Trainers have been trained and coached Skill-based Data-based (pre and post testing) Feedback to Selection and Feed Forward to Supervision

SAMPLE: Reading Foundations Training Pre and Post Test Items § Some tips to consider: § Pre-post test items should be linked to training/teaching objectives § Data from pre-post tests should be used for training revisions, developing coaching service delivery plans, etc. § Presentation of pre-post test should emphasize the non-punitive nature of this type of assessment and that it will be shared back with the participants § Be a good model of using data to get better

Training “Best Practices” § § § Theory Wh grounded (adult learning) at d Skill-based of t o w hes e k Feedback to Selection and Feed now e Supervision Forward to “b abo est pra ut e § Data-based (pre and post testing) ctic ach § Trainers have been trained and es? ” coached

Adult Learning “Best Practices” § The most effective training includes learner experiences related to planning, application, and deep understanding § Use a diverse array of Adult Learning methods (“where 5 or 6 adult learning method characteristics were used, the average effect size was almost 1. 25”) § Learners need to be engaged actively in the learning process § Multiple learning experiences, large doses of learner selfassessment and reflection, instructor facilitated learner assessment § Small numbers of participants, multiple occasions Trivette, Dunst, Hamby, O’Herin, 2009, p. 10 -11

Adult Learning “Best Practices” Trivette, Dunst, Hamby, O’Herin found best practices in each of the following categories: § Introducing Information § Illustrate/demonstrate § Practicing § Evaluation § Reflection § Mastery Trivette, Dunst, Hamby, O’Herin, 2009, p. 6 - 8

Skill-based “Best Practices” § § § Behavior Rehearsals (vs. Role Plays) Knowledgeable Feedback Providers Practice to Criteria

Data Based “Best Practices” § Develop and use pre/post tests to determine to what extent knowledge and skill levels are being improved § § Outcome data collected analyzed Fidelity measures collected analyzed

Staff Training Collins, S. R. , Brooks, L. E. , Daly, D. L. , Fixsen, D. L. , Maloney, D. M. , & Blase, K. A. (1976)

Feedback “Best Practices” § § Feed Forward of pre/post data to Coaches/Supervisors Feedback of pre/post data to Selection and Recruitment Performance Assessment Training Selection Coaching

Feedback “Best Practices” Of course, there are even more opportunities to benefit from feedback… Performance Assessment Training Selection Coaching

Improved Outcomes for training participants Effective Training Practices Performance Assessment (Fidelity) s riv e y D mp ete nc Co Integrated & Compensatory Facilitative Administration er riv Selection D on ati niz ga Training Systems Intervention Or rs Coaching Decision Support Data System Leadership Adaptive Technical © Fixsen & Blase, 2008

Implementation-informed PD § If Professional Development is in support of a well-defined, effective intervention. . . . then Professional Development includes § Training § Coaching § Performance assessment § and Selection

Training, plus coaching § Some well-defined interventions (EBPs) will require additional support to teachers posttraining: § Joyce and Showers (2002) data » » § With coaching = 95% use in classrooms Without = 5% use in classrooms Rogers, Wellens, & Conner (2002) data » § About 10% of what is taught in business workshops actually is put into practice Significant data across domains that feedback improves performance

Coaching “Best Practices” § Design a Coaching Service Delivery Plan § Use multiple sources of information for feedback – Direct observation is critical § Provide regular feedback to all “Drivers” § Develop accountability structures for Coaching – Coach the Coaches! § § Regular review of adherence to Coaching Service Delivery Plan Multiple sources of information for supervisor feedback

Performance Assessment Best Practices § Transparent Processes – Orientation § § What, When, How, Why Use of Multiple Data Sources § § Content § § Context Competency Tied to positive recognition – not used ‘punitively’

Selection “Best Practices” § Job description clarity about accountability and expectations § Pre-Requisites are related to “new practices” and expectations (e. g. basic group management skills). § Interactive Interview Process: § Behavioral vignettes and Behavior Rehearsals § Assessment of ability to accept feedback § Assessment of ability to change own behavior

EXAMPLE: Qualifications of Potential SIP II Foundation Trainers ü Successfully complete the Reading Foundations course including all of the tasks (level 2) and meet the passing criteria (80%) on the Structure of Language Test. ü Demonstrate mastery of the content of the Reading Foundations course as well as the ability to deliver this information to others. Mastery of content also includes knowledge of the texts identified as basic to the course. ü Provide appropriate review and feedback of tasks completed by participants in the Reading Foundations course.

Enhancing Professional Development § Promotes adult learning and changes adult behavior § § Empowers individuals to improve their craft § Integrates selection, training, coaching, and performance assessment to promote the development of a highly effective workforce § Supports the creation of a “hospitable environment” to allow for these new behaviors to be used and improved Ensures that new skills are used and/or current skills are refined

D on riv e y D ati niz ga How: Or Professional Development rs Program/Initiative (set of practices) rs Core Implementation Components Institutional capacity to support staff in implementing practices with fidelity e riv mp ete nc What: Improved Outcomes for. . . Co Why: Leadership Capacity to provide direction/vision of process © Fixsen & Blase, 2008

Discussion Question To what extent are current approaches for professional development designed to develop the necessary competence in participants to skillfully use the most effective educational approaches to accomplish the goal of improved outcomes for students?

Overcoming Barriers : COMMON MEASURES “Commitment to Continuous Quality Improvement”

Integrating All We Know § § § A well-defined and effective intervention § § § An effective approach to training § Use of selection “best practices” Clarity about the “active ingredients” Clear criteria to assess performance (that correlates with the desired outcomes) An effective coaching system An effective performance assessment process

Improvement Cycles

Improving PD Effectiveness § Collect and use data (pre/post tests to determine to what extent knowledge and skill levels are being improved) § Develop the capacity of the trainers (establish clear performance criteria, based on “what works, ” then select, train, coach, and assess trainers to criteria) § Improve the ways in which training participants are “prepared” for training (create readiness)

Questions to Consider 1. What do we mean when we use the term, “professional development? ” 2. What outcomes are we attempting to achieve? 3. What do we know about the professional development strategies that are likely to achieve particular outcomes? 4. How can we monitor and improve the quality of professional development over time? 5. What are some resources and tools we can use to support sustainable and aligned professional development?

For More Information Michelle A. Duda duda@unc. edu Melissa Van Dyke melissa. vandyke@unc. edu Chris Borgmeier cborgmei@pdx. edu

For More Information State Implementation and Scaling up of Evidence-based Practices (SISEP) Dean Fixsen, Karen Blase, Rob Horner, George Sugai www. scalingup. org § Concept paper § Annotated bibliography § Data on scaling up § Scaling up Briefs

www. implementationconference. org

Evidence-based Implementation Research: A Synthesis of the Literature Fixsen, D. L. , Naoom, S. F. , Blase, K. A. , Friedman, R. M. & Wallace, F. (2005). Implementation Research: A Synthesis of the Literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network (FMHI Publication #231). Download all or part of the monograph at: http: //www. fpg. unc. edu/~nirn/resources/detail. cfm? resource. ID=31

Thank You for your Support § Annie E. Casey Foundation (EBPs and cultural competence) § National Institute of Mental Health (research and training grants) § William T. Grant Foundation (implementation literature review) § § Substance Abuse and Mental Health Services Administration (implementation strategies grants; national implementation awards) Juvenile Justice and Delinquency Prevention (program development and evaluation grants § Office of Special Education Programs (scaling up capacity development center) § Administration for Children and Families (child welfare reform; capacity development) § Duke Endowment (child welfare reform) § Centers for Disease Control & Prevention (implementation research)