Where s the Data Conducting a First-Year Data Audit

Where’s the Data? Conducting a First-Year Data Audit Karen Paulson Asheville, North Carolina – July 2002

The Data Audit Toolkit n Developed Through a Partnership > The Policy Center on the First Year of College • > John Gardner, Betsy Barefoot, Randy Swing, Mike Siegel, and Marc Cutright NCHEMS • Karen Paulson and Peter Ewell n Generously Supported by: > The Pew Charitable Trusts > The Atlantic Philanthropies

Pilot Institutions n Augustana College (IL) n Blue Ridge Community College (VA) n Lynchburg College (VA) n Northeast State Technical Community College (TN) n Ohio University n Santa Fe Community College (FL) n The University of Texas at El Paso n University of Cincinnati n University of Minnesota Duluth n Washington State University

Data Audit “The process of identifying data resources and uses wherever they may be within an institution and gathering them into a useable information system. ” NCHEMS Administrative Rationale manual, page 15

Basic Premise I n Most of the Data Needed for Meaningful Analysis Already Reside in Institutional Data Systems > Admissions > Registration/Transcripts > Student Surveys > Assessment Databases > Individual “Service Office” Records

Basic Premise II n The Major Task Is: > To Organize These Data in Useful Ways > To Make Access and Analysis as Flexible and Straightforward as Possible

The Data Audit Process n Is NOT Itself an End n Is a Means for Understanding and Improving the Use of Data and Information on Campus Key to Improving the First Year of College

“…we tend to assume that all first-year programs were implemented as planned and that the experiences of all students were uniform. ”

In Reality There Are Three First-Year Experiences* n The “Official” One (Designed and Published Plans) n The “Delivered” One (Actual Institutional Actions) n The “Experienced” One (Reality Including Student Choice) * Adapted from Joan Stark

The Main Question…. n How can your institution analytically disentangle the many elements of the first college year and provide evidence about the effectiveness, or lack thereof, of related first-year programs, policies, and procedures?

Some Preliminary Questions n What Is Unique About the First Year of College at Your Institution? n What Is it Like to Walk in a Student’s Shoes at Your Institution? n How Many Surveys, Tests, or Assessments Does a Student Really Experience During the Year? n Does Any Information from These Data Collections Get Communicated Back to the Students? How? n How Are These First-Year Data Used by Faculty and Administrators?

A Quick Self-Assessment n What Are the Primary Sources of Data About Students, Curricula, and Programs—Specifically Related to the First Year of College—Currently in Place at Your Institution? n Are Any of These Underutilized? Why? n What’s Missing About Which You Would Like Information?

Why a Data Audit? n Identify and Inventory Data Sources n Identify and Inventory Data Needs n Support Assessment of. . . > What Happened > What Mattered n Foster a Culture of Data Use

Primary Data Audit Activities The Supply Side Conducting a Campus-Wide Examination of Existing Data Sources The Demand Side Determining Which Data Are Most Needed for Evaluation, Assessment, and Decisionmaking

Key Components of a Data Audit n Identifying Sources of Data n Inventorying These Data Sources n Compiling Information About Data Sources n Identifying Gaps in Data Sources n Assessing What Data Users Need and Want to Know n Determining Which Analyses (Existing or Suggested) Have Utility for Stakeholders of Your Institution or Unit

Organizing a Data Audit The Data Audit Leadership Team n Institution-Wide Committee > IR, Student Affairs, Academic Affairs, Advisors, Orientation Staff, Faculty, Registrar Staff, Students, etc. n Team Members > People Directly Involved with the Institution > People Directly Involved with Data > First-Year Professionals not Normally “Data Connected” > Leader(s) with Broad Campus Support and Respect

The Right Attitude n Fresh Perspective n Open Attitude n Collaborative Approach Not a “Gotcha” Mentality

Visiting Potential Data Sites n Reasons for Physical Visits > Honors Unit Personnel on Their Turf > Builds Relationships > Allows You to Read Reactions > Allow You to Do Immediate Follow-Up and Collect Artifacts > Discovers Hidden Databases

Types of Data n Official Data > Registrar Systems > Admissions > Institutional Research Offices > Assessment Offices n “Unofficial” (Guerrilla) Databases > Unit Records Kept in Local Computers > Viewed as “Single” Purpose/Disposable Records

Supply Side Questions n Types of Data? n Who Collects Data and Why? n How Complete Are Data? n Where Do Data Go? n “Walking the Process”

Supply Side Questions to Ask n What Kinds of Records or Data Do You Keep on First Year Students? n How Complete Are Data Collected? n How Are Data Entered? n What Schedules Govern Your Data Collection? n How Are Data Updated?

Finding Existing Data n Follow Student “Footprints” > Time-Based Investigation n Who Collects Data? > Structures-Based Investigation n Why Are Data Collected? > Functions-Based Investigation

Supply Side: Items to Collect n Actual Forms and Questionnaires n Data Element Dictionaries n Data Element Definitions n Database Structures and File Formats

Demand Side Questions n Who Are the Key Constituencies? n What Are the Existing Reports and Requirements? n To Whom Are Data and Information Reported? n What, and When, Are the Decision Cycles? n How Current and Accurate Does the Information Need to Be? n What Are the Gaps in Existing Data?

Demand Side Questions to Ask n To Whom Do You Report Data and Information? n What Information Do You Need or Wish You Had? n What Are the Gaps in Data? n How Are Data Used By Others?

Demand Side: Items to Collect n Representative Reports to Constituents n Copies of External Data Reporting Requirements

Looking for. . . n Completeness of Data Gathered n Availability of Data n Integrity of Data n Consistency of Data Definitions n Who Coordinates Data Collection? n Who Coordinates Use of Data? n Who Controls Data and Data Processes?

Data Audit Output n Synthesized and Coherent Picture of Existing Data > Data Element Lists > Data Definitions > Data Locations and Locus of Responsibility > Data Collection Timetables > Plan for Common Data Collection/Sharing > Identification of Unmet Data Needs

Design of Recommended Data Structure n Define Core Data Elements n Responsibilities and Data Flows n Core Indicators and Calculational Routines

Data Leverage Points n Problem Identification > > > n Context Setting > > > n Achievement by Gender or Race Patterns of Student Performance Entering Student Characteristics To Inform Discussion > n Persistence Rate Violations of Prerequisite Sequencing Student/Advisor Ratios Overall Patterns Rather than Anecdotal or Single Selling Decisions > Creating Buy-In for Action

Using Evidence to Stimulate and Manage Change n Start with Obvious Discrepancies > > n > > Perceptions and Reality Designs and Delivery Expectations and Results Among Different Constituencies and Stakeholders Recognize There Will Always Be “Errors” Use Triangulation and Multiple Indicators Know How Good Is Good Enough Use Information in “Layers” > > > n • • • Avoid “Perfect Data Fallacy” > n Between: Avoid Excessive Complexity Package Data Around Problems Disaggregate as Needed in Response to Questions and Opportunities Use Information to Start Discussions, not “Give Answers” > > Develop Multiple Interpretations Establish “Data Dialogue”

The Data Audit Toolkit (coming fall 2002) n Administrative Rationale > A non-technical overview and rationale for a data audit. n Technical Manual > A detailed guide, listing data elements and data, as well as step-by-step plans for a data audit.

Thank You! If you have questions or want to discuss the data audit and analysis project further, please contact: n Karen Paulson, 303. 497. 0354, Karen@nchems. org n Mike Siegel, 828. 877. 6009, siegelmj@brevard. edu