Data-Driven Decision-Making for Quality Control The Power of

Data-Driven Decision-Making for Quality Control: The Power of a Relational Database Vicki L. Cohen, Ed. D. Marlene Rosenbaum, Ed. D. Joshua Cohen Fairleigh Dickinson University Teaneck, NJ 07666 Presented at the Annual AACTE Conference New Orleans, February 2008

This session will: n n n Describe how using a relational database becomes the driver of a quality control system; Describe the development and utilization of a relational database; Show data is leveraged to support student learning, program revision, and outcomes based assessment.

The School of Education (SOE) at Fairleigh Dickinson University Comprised of aprox. 1, 000 students n Multiple programs that apply for state certification: n 5 -Year Accelerated QUEST program n MAT n LD n Educational Leadership n Reading Specialist n

The SOE at a Glance n n n On two campuses (Teaneck and Madison) Located at 3 Community Colleges throughout the state of New Jersey 15 Full-time faculty members Aprox 35 part-time faculty Place approximately 120 candidates into student teaching/year Place a total of approximately 700 candidates into clinical field experiences/year.

FDU at a Glance n n FDU has aprox 12, 000 students Largest private university in State of NJ SOE is part of University College on Teaneck campus QUEST program: n n n 45 candidates at Teaneck 200 Madison 75 CC

SOE Is a Complex Program! Went for TEAC accreditation

Preparation for TEAC n n n School of Education (SOE) needed to collect accurate information on its claims Started gathering data on programs and student performance Recognized need to access, organize and analyze data in meaningful ways Developed a relational database This would become the “driver” of our Quality Control System

The Quality Control System (QCS) Every institution and program has a set of procedures and policies to ensure quality in hiring, admissions, curriculum, program design, and student learning. n Together, these procedures and structures function as a Quality Control System (QCS). (TEAC) n

Need for Valid and Reliable Data QCS must yield valid and reliable evidence about the program’s practices and results, which influences its policies and decision making.

What is Valid Evidence? Are we measuring what we intended to measure? n Are we sure that our evidence is pointing us in the right direction? n How confident do we feel about the data we collected? n

“Am I measuring what I think I am measuring? ”

What is Reliable Evidence? Yields results that are accurate and stable n Collected in a consistent way n Confident we we are making the right decision. n

SOE TEAC Process 1. Developed a QCS that ensured we are collecting valid and reliable data on our claims and cross-cutting themes n n Instruments (rubrics, observation forms, surveys) were validated through external panel of “experts” Inter-rater reliability is being established 2. Developed infrastructure and system to collect the data

SOE TEAC Process 3. Analyzed data: aggregated and disaggregated 4. Determined strengths and weaknesses with total faculty involvement 5. Analyzed what revisions needed • • • Programs Curriculum Processes and policies 6. Currently making revisions based upon evidence.

SOE Assessment Philosophy We use multiple sources of data that are designed to assess the performance of teaching candidates as they progress through our program. n Collect data in three areas. n

1) Throughout the Program Continuous assessment of teaching candidates throughout the program from entrance, midpoint, and exit n n n Grades GPA Praxis scores Rubrics Reflections

2) In Clinical Field Experiences Assessment of pedagogical knowledge and skills that occurs during clinical field experiences; Placement n Observation forms n

3) Completion of Program Perceptions of teaching candidates and alumni after they have completed their program, which is used for program improvement n n n Alumni Surveys Exit Surveys Focus Groups

Traditionally…. Data Deficient Schools of Education have not been collecting data systematically n Infrastructure not set up n Not able to access multiple sources of information n

Traditionally…. . Data Dummies What data do we want to collect? n How can we manage it? n What do we do with data? n How do we organize it? n Access it? n Make sense? n

Currently…. Data-Driven Systematically collecting data n Infusing into our faculty culture n Meeting regularly to assess evidence n Making decisions based upon evidence n Beneficial process due to TEAC n

Data-Driven Decision-Making ct se fle vi Re. Re & Pl lyze Ana a Dat Im pl em en t an Collect Data

Example of Data Collection: Praxis Results 2005/2006 Program FDU SOE Pass Rate NJ Pass Rate Elementary Ed Content Knowledge 99% 84% English Lang Lit Content Knowledge 100% 71% Social Studies Content Knowledge 100% 64% Secondary: Math Content 100% 92% Middle School Math 100% 65% General Science Content Knowledge 100% 78% Spanish Content Knowledge 100% 87%

Wanted: Database Administrator New Job Description: full-time professional staff n Resources n Support from administrators n Ensure candidate had appropriate knowledge and skills to design database n

What Is a Database System? n A collection of data organized in tables, which can be accessed and manipulated, without having to restructure the tables n Elements of a Database System • A storage system • Data structures • Manipulation tools

Database Advantages n n Analyze sophisticated correlations easier because relationships are established between data sets Make decisions based on information derived from data Streamline business operations Organize data and eliminates: • Inconsistent data • Missing data • Redundant data

Problem #1: Field Placement Office Staff was overwhelmed with managing Clinical Placements n In 2007 800+ letters were mailed to 424 schools asking for placements n Previously, these letters were individually prepared in MS Word documents n

Problem #1: Field Placement Office (cont’d) n Clinical Placements must be coordinated with • School • Student • Supervisor n Difficult to aggregate data • What districts have most confirmed/declined rates? • What trends are we seeing? • What kind of schools are we sending our candidates to?

Solution #1: Streamlined Field Placement Office n Developed a data collection system for Clinical Placements n Data is: • Entered on 2 campuses • Used to create personalized communications to schools, students, and supervisors • Used to manage Clinical Placements • Confirmed/Pending/Declined • Supervisor Assignments

Solution #1: Streamlined Field Placement Office (cont’d) n Gives us the ability to aggregate data, look at trends, and make decisions Confirmed / Declined distribution by district n Analyze demographics of cooperating school districts n

Solution #1: Infrastructure

We start with the person record from the university system

Data on the Clinical Placement is entered

Solution #1: Generating letters

Generate standardized reports for “master lists”

Solution #1: Payroll

Problem #2: What Type of Districts Do We Place Our Candidates Into? The District Factor Group (DFG) is a socioeconomic indictor used for comparative test reporting of school districts for New Jersey’s statewide programs.

Problem #2: What Type of Districts Do We Place Our Candidates Into? (cont’d) n DFG Factors: n n n n % of adult residents failed to complete high school % of adult residents who attended college Occupational status (laborers, service workers, farm workers, professionals, etc. ) Population density Income Unemployment rate Poverty rate

Problem #2: What Type of Districts Do We Place Our Candidates Into? (cont’d) n Eight DFGs have been created based on the 1990 United States Census data n Range from A (lowest socioeconomic district) to J (highest) n A, B, CD, DE, FG, GH, I, J

DFG: State Distribution

DFG: Apprenticeship Teaching Distribution

DFG: Alumni Distribution

Solution #2: Share Data, Discuss, Revise n n n Evident discrepancy between where alumni get jobs and where candidates are placed We share evidence with faculty and key stakeholders They discuss and make appropriate decisions

DFG: In the future Will have full record of where candidates performed clinical experience n Will have record of where they are working n Can correlate accordingly n

Data Needed n A state database of teacher employment Difficult to track alumni as they move from school to school n Unique teacher & school identifier n n State database needs to integrate with University and commercial marketing data systems

Problem #3: Data Systems Not Integrated n SOE’s recordkeeping is not integrated with the University system Student information is entered into SOE system manually n Limits power of reporting n Duplicate person records may exist if Student ID is not entered correctly n

Problem #3: Data Systems Not Integrated (cont’d) n. A “live” data connection to University system is not possible n Technology is not in place n Support for integration is needed

More Efficient and Effective Use of Resources Relational database assists with: n Streamlined SOE business operations n n Generate mail merge letters Provide reports Automate payroll Leverages existing data to create information for program improvement

Started Slowly Started with trying to code and track our students properly n Administrative assistant created rudimentary Access Database n When she left, we hired a consultant to manage database n He totally redesigned and reorganized it n

Skill Set for Database Administrator n n n n Problem-solving Relational database design skills SQL proficiency Knowledge of structured programming language Excellent communication skills Work with faculty and technical staff “People-skills”

Conducted Extensive Search Advertised n Set up search committee n Interviewed many different applicants n Required each applicant to take a test n Presented problems to candidates to assess problem-solving abilities n Found many could enter data, but not design or problem solve n

Working with the University Collaborating with the Arts and Sciences n Establishing knowledge-base in content areas and general education n Addressing NJCCCS and Professional Standards in discipline n Aligning content courses and standards in matrices n

Working with the University(cont’d) n n n Meeting with individual departments Establishing long-term relationships Shifting to new paradigm--Learning Outcomes Assessment Working with A&S to collect data Database Administrator playing key role in collection of data across college

Leveraging TEAC Across the University n n n n Establishing the need for LOA: Middle States Educating the A&S faculty: LOA process Addressing resistance of A&S faculty Establishing a relational database system for university: program, college Creating the infrastructure to collect data Collecting multiple sources of data for A&S Getting various groups to communicate and plan.

Conclusion: The Power of a Relational Database Data-driven Decision-making requires an integrated system of collecting data from many different sources.

Systemic n The whole institution needs to be vested in the collection of data n Data needs to be collected on faculty, students, courses, grades, scores, rubrics, observations n University and Colleges need to ensure data collection systems are in place early.

Integrated All data sets need to be connected so relationships can be established n Queries made n Reports generated n Correlations and relationships analyzed n

Benefits for Total University n n n Middle States Accreditation Nursing Engineering College of Business Program improvement Student learning

School of Education Leads the Way n Ultimate goal is to improve teacher quality and impact achievement for all students. n Data provides the means to do this. n Relational database is the engine that makes this possible.

For more information please contact: Vicki L. Cohen cohen@fdu. edu Director, School of Education Marlene Rosenbaum rosenbau@fdu. edu Associate Dean, University College Joshua Cohen cohenjp@fdu. edu Database Administrator