The National Benchmarking of an Assessment Tool to Measure Outcomes in Administrative Areas National Community College Benchmarking Conference Terri M. Manning, Ed. D Central Piedmont Community College Charlotte, North Carolina
Some history…. CPCC is the largest community college in both the Carolinas with approximately 60, 000 enrolled students. u We have over 100 instructional program areas including for-credit, continuing education and literacy u We have over 30 administrative units u
What’s Going on Nationally u The Spellings Commission Draft Report calls for these things in higher education (across the country). • the complete overhaul of the entire student aid system, together with substantial increases in needbased aid; • creation of an overall measurement of an institution's "bottom line, " including measures of institutional costs and performance that let parents and policy makers view institutional results; • a mandate that institutions measure student learning outcomes, disseminate the results to students, and report them publicly in the aggregate;
Spellings Commission Report • development of a national student unit-record database to follow the progress of each student; • establishing a national accreditation framework that includes comparable performance measures, and making the findings of reviews easily accessible to the public; • removal of barriers to the transfer of student credits; and • creation of financial incentives for institutions to lower costs through technological advances. http: //www. ed. gov/about/bdscomm/list/hiedfuture/reports/0714 -draft. pdf
What May Happen More emphasis on outcomes for all that must be transparent and available to the public. u Overhaul of financial systems and funding formulas that is related to student success and outcome based criteria. u Our spending justification under the microscope (especially administrative overhead which is huge for most). u A complete overhaul of accreditation. u
What We Have and Don’t Have u Have • Good national tools to assess student Satisfaction u Engagement u u Don’t have: • Good tools to measure administrative units or “all-college functions”. • No one has really thought about it.
Institutional Effectiveness and the Southern Association of College and Schools u u SACS is the oldest and strictest accrediting agency, with North Central coming right along behind it ---- but the others are not yet quite as strict. At the heart of SACS’ philosophy is the concept of institutional effectiveness (IE). IE involves a process of planning, evaluation, assessment and use of results. Under the new core requirements and comprehensive standards, there are two “mandates” that specifically address program or unit reviews and the assessment of outcomes.
Under the New Core Requirements and Comprehensive Standards Core Requirement 2. 5 states “The institution engages in ongoing, integrated, and institution-wide research-based planning and evaluation processes that incorporate a systematic review of programs and services that (a) results in continuing improvement, and (b) demonstrates that the institution is effectively accomplishing its mission. ” Comprehensive Standard 3. 3. 1 States “The institution identifies expected outcomes for its educational programs and its administrative and educational support services; assesses whether it achieves these outcomes; and provides evidence of improvement based on analysis of those results. ”
What We Need In a Nut Shell u A planning and evaluation process for instructional programs, student services and administrative areas that is: • • Ongoing Integrated Institution-wide Systematic Research-based Shows continuous improvement Proves effectiveness (meeting mission)
The Pieces we Need Planning and Evaluation Process u Program Review Process u Define and Measurement Outcomes u Use Results (with evidence) for Improvement u
What is Evaluation Not research but research-based. u A process of defining purpose, identifying process, outcome or impact objectives and measuring if those were met. u Using the results to: u • Improve what we are doing • Make changes in program/purpose
Why this Process? u u u The evaluation of teaching and educational effectiveness is a top priority for most institutions However, the evaluation of business and administrative units is often undervalued and overlooked Evaluating the effectiveness of services (not just customer satisfaction) makes good business sense
Why This is Important u We want to answer a few questions through our program evaluation and assessment process. • Is this unit effectively serving the college? • What needs does the college have that this unit can fulfill? • Is this unit working efficiently? • What are the administrative outcomes for this unit and are they meeting them? • How can we demonstrate continuous improvement as a result of planning and evaluation?
IE at Central Piedmont CPCC established an IE committee in 1998 and an IE website in 2000 (http: //inside. cpcc. edu/IE) (this website gets an average of 60 hits a day – most from outside the college) u Committee members were from various instructional departments, student services and administrative/business offices u The committee established an IE plan for the college u
Visual of the IE Plan
Four Major Overlapping Pieces of the IE Plan u General Education Assessment Program/ Unit Review u u Annual Goal Setting Cycle College Assessment Process u Annual goal setting and follow-up required by all units (Strategic Plan) Program/unit review on a 3 -5 year cycle College assessment plan involving surveys, assessment and data analysis The evaluation of general education
Developing the Process We wanted to create a meaningful program/unit review process u We wanted programs to complete the process having learned something valuable (not a document to set on a shelf) u We wanted the process to be “outcome-based” to stand the test of time u
Developing the Process The first review process was created for instructional programs u During the 1998 -99 year, 18 programs were reviewed (approximately 100 programs over five years) u The perception at the beginning was that “this is just another academic exercise” u But the results were very different than any review process previously completed u
Identifying Outcomes u We focused on three types of outcome measures: • • • u Program Outcomes Student Learning Outcomes Administrative or Managerial Outcomes "Outcomes" are benefits for people: changes in knowledge, values, position, skills, behavior or status. More simply stated, outcomes are typically what service providers hope recipients achieve once they complete a program or receive services. This is not the “what” but the “why” of education.
Outcomes u u Student learning outcomes are outcomes related to the learning that takes place in the classroom. We measure improvements in writing, speaking, understanding the scientific method, etc. Program outcomes are the benefits to a student who receives an associate degree in Nursing or completes a certificate in Network Administration? Typical outcomes might be licensure exam scores, job placement, etc. Administrative outcomes are the benefits to the college as a result of the direct services provided or the function of a unit. Outcome objectives are just objectives that relate to the identified outcomes.
Program Outcome Model INPUTS ACTIVITIES OUTPUTS Resources Services Products or Results of Activities Staff Buildings Facilities State funds FTE Education (classes) Services Counseling Student activities Numbers served FTE (input next year) # Classes taught # Students recruited # Who participated Constraints Laws State regulations
Program Outcomes Model INPUTS > ACTIVITIES > OUTPUTS > OUTCOMES Benefits for People *New knowledge *Increased skills *Changes in values *Modified behavior *Improved condition *Altered status *New opportunities
Organizational Stages of Outcome Evaluation Stage 4 Stage 5 Acceptance & adaptation Challenge & competition Catalyst - Proactive Depression - exhaustion Compliance - Passive reactive Stage 3 Bargaining - no time/no money Seek outside sources Stage 2 Anger and antagonism Resistant & Reactive Stage 1 Disbelief & Denial Paralysis - Passive resistance
Instructional Program Review u The review process contained five sections: • III. • IV. • V. The Program Profile Program Content Student Learning Outcomes Need for Change Future Issues
Instructional Program Review u Tasks accomplished through review: • Defined their program/unit responsibilities • Detailed faculty and staff (credentials, prof. dev, accomplishments) • Unit mission or purpose • Link unit mission to the college mission • Defined the content of their function for the college • Defined the population served • Set outcome objectives • Performed some means of assessment • Analyzed results • Determined strengths and weaknesses • Created strategies for improvement • Determined future needs (including curricular change, needed resources, staff and space) • Wrote a one-year follow-up to make sure they closed the loop
An Example from Instruction u Workplace Basic Skills • This program is a literacy initiative that goes directly into the worksite and teaches ESL classes, GED prep and GED classes. During their review, they surveyed both employers and students. u This was the first time they had even done this. u
What They Learned u Employers said: • 43. 8% of employers reported increases in employee performance as a result of participation in the program. • 31. 3% reported a reduction in absenteeism by participants. • 87. 5% said classes improved the morale of their employees • 37. 5% said participants received raises • 50% said communication had improved.
What Students Said u u u 70. 2% reported being able to fill out job forms better 35. 5% said they could now help their children with their homework 91. 1% said they felt better about themselves 44. 4% said they had received a raise, promotion or opportunity as a result of the courses 86. 3% said their ability to communicate in the workplace had improved
What Has Happened Since Their assessment data has shown up in their marketing brochures to employers. u Their enrollment has grown dramatically. u They have received funding and marketing support from Charlotte Reads (considered a model adult literacy program). u
What We Learned From the Entire Process…. u u u The first group was dragged kicking and screaming through the process Faculty had very little time for the details Faculty had trouble identifying “student learning outcomes” The results produced great marketing materials Once they saw the results, faculty embraced the process
Assistance from the Website The IE website: u http: //inside. cpcc. edu/IE u • Explanation of IE process • Templates and forms for review • “Perfect” examples for clarification • Schedule for review
What Happened Deans used the results to make a case for resources u The administration became interested in what was learned through the review process u The college began to fold other activities into program review (retention assessments, program revitalization, etc. ) u Faculty knew their programs were working u
Unit Review – Lessons Learned u To do this well you need several critical pieces: 1. Support from the top (President or Chancellor, Vice Presidents or Vice Chancellors) 2. Buy in from the grassroots level (participation in the development of the process) 3. Across-the-college participation (no one is exempt) 4. Technology to make it easy (web page and review templates) 5. Technical support from institutional research 6. Effective tools or measures (assessments)
IE Committee Decision u u u Administrative units helped created an environment conducive to learning at the college and supported the learning process Administrative units should go through a similar process We created a committee to draft a similar process for the administrative areas of the college Two representative from each VP area participated in drafting the new review process We spent approximately six months creating a workable administrative review process
The Administrative Unit Review Design I. The Unit/Program Profile A. The Mission/Purpose 1. Role unit plays in the college mission 2. Unit/program goals as they relate to the college’s mission B. The Staff 1. Professional and administrative staff (since the last review) a. Position description/duties b. Credentials (full and part-time, if any) c. Accomplishments (if applicable) d. Service to college, community and nation
The Administrative Unit Review (continued) B. The Staff (continued) e. Professional development activities 2. Classified Staff a. List of names and positions b. List of required credentials (if any) C. The Customer/Client Served 1. Breakdown of students/faculty or staff by type or demographic information (thorough explanation of who is served)
The Administrative Unit Review (continued) II. Definition of Services or Program A. Definition of day-to-day duties of the unit B. Innovations, new projects, new initiatives, local, state-wide or national efforts C. Required functions of unit (description and status of compliance) 1. 2. 3. 4. SACS "must" statements State mandates Federal mandates Other
The Administrative Unit Review (continued) III. Administrative Objectives and Student Outcomes (where appropriate) A. Administrative Objectives (2 -3 objectives) B. Outcomes (or status if incomplete) of innovations, new projects, new initiatives, local, state or national efforts C. Assessment explanation (what was assessed, who, when, how many) D. Results of Administrative Objectives Based on Assessment
The Administrative Unit Review (continued) IV. Need for Change A. Strengths identified by external sources, faculty, staff and students B. Weaknesses identified by external sources, faculty, staff and students C. Recommendations by faculty, staff, external sources and students to improve the unit's services and programs D. Strategies for change (based on input from Sections A, B & C) - closing the loop E. A one-year follow-up brief report to the Unit VP reporting on the progress of D above (due at the end of the year following review)
V. The Administrative Unit Review (continued) Future Issues - Resources needed for future efforts A. Market trends within the broad service unit or program area (based on bestpractices, the literature or training received) B. Anticipated future changes and needs (based on market trends) C. Resources, equipment, space, staffing and work load changes needs for future growth or continuation D. Future plans of unit
But How do You Assess the Outcomes of Administrative Units? u u u We created an evolving and revolving faculty/staff survey (three year cycle). The survey is an online product and easy to administer. We spent time training units on “what outcomes were” rather than just satisfaction questions. They could also use it for needs assessment questions. Satisfaction questions can answer administrative outcome objectives though.
Assessment of Administrative Units During the year of program review, the “Annual Faculty/Staff Survey” contained questions from those units being reviewed. u Results were given to each unit and broken out by campus and job type http: //www. cpcc. edu/planning u u There’s a link to survey results.
Administrative Units at CPCC ü ü ü Administrative Technology Services Instructional Technology Services Compliance & Audit Inventory Control Basic Skills Compliance & Reporting Facilities Maintenance Facilities Design and Construction Institutional Advancement/Foundation Planning and Research (IR/IE) Resource Development Security
Administrative Units at CPCC ü ü ü ü ü Equal Opportunity Employment Human Resources Occupational Health and Safety Technology Help-desk Email Services The Book Store Vending/Food Services Parking Services Campus Printing Distribution Services (mail/moving)
Administrative Units at CPCC ü ü ü ü ü Payroll Financial Reporting/Budgeting Purchasing/Procurement Accounts Payable Cashiering Community Relations Marketing Services Graphic Design Professional Development Faculty Development (CTL)
New Unit (entrepreneurial) u The Services Corporation: • The Center for Applied Research • The CPCC Press • The Harris Conference Center • The Performing Arts Center u Will have to add new questions
Where Do All These Units Fall INPUTS ACTIVITIES OUTPUTS Resources Services Products or Results of Activities Staff Buildings Facilities State funds FTE Education (classes) Services Counseling Student activities Numbers served FTE (input next year) # Classes taught # Students recruited # Who participated Constraints Laws State regulations
When You are an Input – What are Your Outcomes INPUTS > ACTIVITIES > OUTPUTS > OUTCOMES Benefits for the College *New Efficient Approaches *Enhanced Services *Motivating Environment *Supportive Behavior *Improved Conditions *Altered Expectations *New opportunities
Facilities Services u Outcomes • Classrooms conducive to learning • Students to be safe and secure • Faculty to be able to work in assigned space u Survey questions were developed and asked around these themes
Facilities Services u Survey respondents gave high rankings (in the 80 -90% range) for: • Newly constructed or newly renovated facilities being of high quality and meeting needs • Classrooms and common areas being generally clean • Lighting levels being adequate • A safe and secure environment being provided for all members of the College community • Knowing procedures to follow if a medical emergency on campus
Areas Needing Improvement u Only 73% of respondents were pleased with new office spaces. • To address this concern, the Facilities Standards Subcommittee of Facilities Partners will review the adopted standards for offices and furnishings and make recommendations for revisions. u Only 50% of faculty and staff were pleased with the temperatures in classrooms (affected teachinglearning process). • The Central Energy Plant began operating in the summer of 2003; when connected to the main body of Central Campus, the back-up capabilities of this facility should minimize the disruption of cooling. The College is also developing a plan for the future pairing of boilers to provide back-up heating for Central Campus buildings. At the suburban campuses, central plant approaches are being used to link buildings, thereby providing back-up capability.
Areas Needing Improvement u Only 57% of the respondents expressed satisfaction with the cleanliness of bathrooms. • The College entered into a new housekeeping contract as of July 1, 2003 which employs a new vendor and has enforceable cleanliness standards that must be met. Additionally, Facilities Management is addressing the appearance of some of the older restroom facilities on Central Campus as part of a cyclical rehabilitation program.
Planning and Research u Outcomes • Be perceived as an effective and efficient unit of the college • Be perceived as contributing to the effectiveness of the college • Provide adequate assistance with survey design • Provide helpful guidance through the program review process • Have websites that are used and contain appropriate information
Planning and Research § Survey respondents gave high rankings (in the 80 -90% range) for: § § § § § Responding quickly to their need for data/information. Producing enough reports annually to meet the planning and information needs of faculty and staff. Making a significant contribution to the College. Contributing to the effectiveness of CPCC. Assisting in the area of survey development/design. Assisting with program/unit review. Having a quality Institutional Effectiveness Website. Having a quality Institutional Research Website. Having a quality Fact Book (available online).
Security u Outcomes • For students and staff to be able to work and learn. In order to do that they must: Feel safe on campus u Handle emergencies correctly u Gather safely in groups (students) u Perceive safety in their jobsite (employees) u
Questions Asked (level of agreement) u u If I see suspicious behavior, I always contact security/campus police. Insufficient Lighting is a significant safety concern within my work environment. Personal Security is a significant safety concern within my work environment. Safety in Gathering Places (crowd control) is a significant safety concern within my work environment.
Information Technology u Outcomes • Faculty and staff to be able to use reliable technology to get their jobs done more effectively and efficiently. • For the college not to be late on any state or federal reporting generated out of IT or assisted by IT. u Needs assessment • How many are using remote email service
Information Technology § Survey respondents gave high rankings (in the 80 -90% range) for: • Courteous and helpful staff in answering questions and resolving issues with administrative systems access. • Providing regular scheduled reports accurately and on time. • Reliability of systems and availability especially during heavy registration. • Not experiencing significant down time as a result of office computers not working.
Purchasing Department u Outcomes • Provide adequate training so departments can purchase what they need. • The purchasing card program will save time and money. • Manuals and information will provide adequate information in a readable format.
Vending u Needs Assessment • Do you like Coke or Pepsi products • Do you like 12 oz. cans or 20 oz. bottles u Outcomes • Faculty and staff can find healthy food and drink choices in the vending machines and through food services.
Book Store u Outcomes • To enhance learning, students should have books by the first day of class. • The communication between the book store and the faculty should be adequate and timely. • Faculty will understand the process of requesting special products for classes.
Need for Senior Administration Response Units spend a lot of time working on the program review. u If we want them to take it seriously, we have to take it seriously. u Once reviews are reported at the end of the year, their Dean or Vice President/Chancellor need to: u • 1. Read them • 2. Respond to them
From Personal Experience Sit down with the unit (group meeting) u Give them attention u Discuss what was learned u Discuss their major issues u Discuss what they need to make improvements u Actually attempt to direct resources to them to make those improvements u Then they do not see it as an academic exercise u
Overall Benefits u u #1 No recommendations in the area of Institutional Effectiveness from SACS during the October 2002 re-accreditation visit The college became change-oriented • Units had to define strategies for change • Didn’t have to be perfect but rather making continuous progress • Strategies for change helped identify needed resources for units • Units had to “close the loop” with the one-year follow-up (couldn’t promise and not deliver)
Overall Benefits u Units became empowered to perform their functions in an optimal manner and to ask for what they needed (no one noticed them before, now they do) • Created data to support needs • Accomplishments were reported to major administrative groups across the college (President’s Cabinet, Planning Council, etc. )
Overall Benefits The college community understood the purpose and function of every unit u Senior administration realized the benefits and became strong supporters u • Data are reported annually from program/unit review • VPs became stronger advocates for their units making changes to improve services
“Best” Results of “Best” Practice u u u Better use of data across the college We have become more client/customer focused Review gives direction for goals and needed changes Departments are empowered to do their jobs (can’t slip through the cracks and be unnoticed) Problems must be resolved (there is no hiding and no excuses) Surveys provide needs assessment data as well as evaluation which gives departments direction
What We Want to Do Work with 10 -20 colleges this year. u Collect outcomes from their administrative units. u Fine tune the assessment tool (needs to match the outcomes). u Distribute it to all the colleges in pieces (probably 3). u Nationally norm it. u Potentially – use it for benchmarking. u
What We Want to Do Give the colleges their individual reports. u Give them comparison data for the entire group. u Please sign up if you are interested. u
For a copy of this presentation Contact Terri Manning terri. manning@cpcc. edu u Download or print presentation: u http: //www. cpcc. edu/planning u • Click on “studies and reports” • NCCBC Admin Survey
Скачать презентацию The National Benchmarking of an Assessment Tool to
51146390efc8fb28a238aaaad9420eac.ppt