Скачать презентацию Accountability Reporting for California Community Colleges Patrick Perry Скачать презентацию Accountability Reporting for California Community Colleges Patrick Perry

6e66767e66d5c5986f6af043a2e7cd39.ppt

  • Количество слайдов: 58

Accountability Reporting for California Community Colleges Patrick Perry Vice Chancellor of Technology, Research, & Accountability Reporting for California Community Colleges Patrick Perry Vice Chancellor of Technology, Research, & Info. Systems CCC Chancellors Office 1

Data Preamble u “Information is the currency of democracy. ” -Thomas Jefferson u “Get Data Preamble u “Information is the currency of democracy. ” -Thomas Jefferson u “Get your facts first, then you can distort them as you please. ” -Mark Twain u “In the twenty-first century, whoever controls the screen controls consciousness, information and thought. ” -Timothy Leary 2

The CCC System u 109 campuses, 72 districts, all locally governed u 2. 6 The CCC System u 109 campuses, 72 districts, all locally governed u 2. 6 million students (annual unduplicated) u 1. 1 million FTES (annual) u 35% white; half over age 25; 70% part-time u No admissions requirements u $20/unit; 40% get fees waived u Highest participation rate of any CC system in US; 25% of all CC students are CCC 3

Participation (and Fees) 4 Participation (and Fees) 4

CCC Chancellor’s Office u Weak authority; powers vested locally u Unitary MIS data collection CCC Chancellor’s Office u Weak authority; powers vested locally u Unitary MIS data collection (1992) Student, faculty, course, section, session, grade level detail l Data collected end of term, 3 x/yr l Used for IPEDS, apportionment, accountability, research, online data mart l 5

History of CCC Accountability u Simple reporting, fact books until 1998 u 1998: State History of CCC Accountability u Simple reporting, fact books until 1998 u 1998: State provides $300 m ongoing in exchange for accountability reporting l “Partnership for Excellence” was born CCC developed report in isolation u CCC allowed to determine “adequate progress” u “Contingent funding” never triggered u l Used 5 metrics to measure system and collegelevel performance 6

PFE Metrics u Annual volume of transfers to CSU/UC u Annual volume of awards/certificates PFE Metrics u Annual volume of transfers to CSU/UC u Annual volume of awards/certificates u Rate of successful course completions u Annual volume of Voc. Ed. Course completions u Annual volume of basic skills improvements (lower to higher level) l 4 of 5 are volume metrics, only 1 rate 7

The State Said: u Your metrics allow for no adequate college comparisons u Your The State Said: u Your metrics allow for no adequate college comparisons u Your method of determining “adequate progress” is suspicious u You only look good because you are growing u Partnership over (2001), but keep reporting, (until 2004) l we have to spend your money buying energy from Enron 8

What Happened Next u Gov. Gray Davis: recalled for spending money buying energy from What Happened Next u Gov. Gray Davis: recalled for spending money buying energy from Enron u Replaced by “The Governator” 9

The Governator u Likes Community Colleges Comes from a country that has European “academic The Governator u Likes Community Colleges Comes from a country that has European “academic bifurcation” (Austria)university vs trade paths l Attended Santa Monica Community College l u Took ESL, PE, bookkeeping, micro/macroeconomics u Transferred to U. Wisconsin-Superior 10

And Arnold Said: u We shall haves deez accountabeelity seeztem for de community collegez. And Arnold Said: u We shall haves deez accountabeelity seeztem for de community collegez. u. A bill was passed to create the framework, and eventually the framework was enacted. u Named: Accountability Reporting for Community Colleges (ARCC). 11

Arnold Said: u There shall be no pay for performance, but there will be Arnold Said: u There shall be no pay for performance, but there will be the ability to compare performance. 12

We Said: u Some metrics will be system only; others will be at college-level We Said: u Some metrics will be system only; others will be at college-level u College metrics will be rates (to mitigate size for comparison) u No rankings—we will compare colleges against their “peers” u No $$$=ARCC is a “dashboard” accountability report. 13

Arnold Said: u Colleges need to address their performance annually to the State. 14 Arnold Said: u Colleges need to address their performance annually to the State. 14

We Said: u Colleges are more responsive to their local district Board; annual requirement We Said: u Colleges are more responsive to their local district Board; annual requirement to take local ARCC results to local Board and submit minutes to State u Colleges must submit 500 word response, which becomes a part of the final report. 15

Arnold Said: u The report shall be done in collaboration with the State, not Arnold Said: u The report shall be done in collaboration with the State, not in isolation. 16

We Said: u The Dep’t of Finance, Leg Analyst, and Secretary of Education shall We Said: u The Dep’t of Finance, Leg Analyst, and Secretary of Education shall be a part of the technical advisory committee (along with CCC researchers and stakeholders). u We will either succeed or fail together. u This was a really smart move. 17

ARCC u The l Model: Measures 4 areas with 13 metrics: u Student Progress ARCC u The l Model: Measures 4 areas with 13 metrics: u Student Progress & Achievement. Degree/Certificate/Transfer u Student Progress & Achievement. Vocational/Occupational/Workforce Dev. u Pre-collegiate improvement/basic skills/ESL u Participation l “Process” is not measured 18

Student Prog. & Achievement: Degree/Cert/Xfer u College: Student Progress & Achievement Rate(s) (SPAR) l Student Prog. & Achievement: Degree/Cert/Xfer u College: Student Progress & Achievement Rate(s) (SPAR) l “ 30 units” Rate for SPAR cohort l 1 st year to 2 nd year persistence rate l u System: Annual volume of transfers l Transfer Rate for 6 -year cohort of FTF’s l Annual % of BA/BS grads at CSU/UC who attended a CCC l 19

Student Prog. & Achievement: Voc/Occ/Wkforce Dev u College: l Successful Course Completion rate: vocational Student Prog. & Achievement: Voc/Occ/Wkforce Dev u College: l Successful Course Completion rate: vocational courses u System: Annual volume of degrees/certificates by program l Increase in total personal income as a result of receiving degree/certificate l 20

Precollegiate Improvement/Basic Skills/ESL u College: Successful Course Completion rate: basic skills courses l ESL Precollegiate Improvement/Basic Skills/ESL u College: Successful Course Completion rate: basic skills courses l ESL Improvement Rate l Basic Skills Improvement Rate l u System: l Annual volume of basic skills improvements 21

Participation u College: l None yet…but coming. u System: l Statewide Participation Rate (by Participation u College: l None yet…but coming. u System: l Statewide Participation Rate (by demographic) 22

Major Advancements of ARCC u Creating a viable alternative to the GRS Rate for Major Advancements of ARCC u Creating a viable alternative to the GRS Rate for grad/transfer rate. u Finding transfers to private/out of state institutions. u Doing a wage study. u Geo-mapping district boundaries. u Creating peer groups. 23

Defining Grad/Transfer Rate u Student Progress & Achievement Rate (SPAR Rate) l IPEDS-GRS for Defining Grad/Transfer Rate u Student Progress & Achievement Rate (SPAR Rate) l IPEDS-GRS for 2 -yr colleges stinks: u No part-timers u How do you define degree-seeking? u Tracking period too short u Outcomes counting methodology terrible AA/AS/Cert counted before transfer v Transfer to 2 -yr college is counted v 24

SPAR Rate u Defining l the cohort: Scrub “first-time” by checking against past records SPAR Rate u Defining l the cohort: Scrub “first-time” by checking against past records (CCC, UC, CSU, NSC) 25

SPAR Rate u Define “degree-seeking” behaviorally for CC populations l Not by self-stated intent; SPAR Rate u Define “degree-seeking” behaviorally for CC populations l Not by self-stated intent; this is a poor indicator u Behavior: did student ever attempt transfer/deg-applicable level math OR English (at any point in academic history) l Students don’t take this for “fun” 26

Defining Degree-Seeking Behaviorally u Separates out remedial students not yet at collegiate aptitude l Defining Degree-Seeking Behaviorally u Separates out remedial students not yet at collegiate aptitude l Measure remedial progression to this threshold elsewhere u Creates common measurement “bar” of student aptitude between colleges l Same students measured=viable comparison 27

SPAR Rate-Unit Threshold u CCC provides a lot of CSU/UC remediation Lots of students SPAR Rate-Unit Threshold u CCC provides a lot of CSU/UC remediation Lots of students take transfer math/Eng and leave/take in summer l Should not count these as success or “our” student l u Set minimum unit completed threshold (12) for cohort entrance l Any 12 units in 6 years anywhere in system 28

SPAR Denominator: u First-Time (scrubbed) u Degree-seeking (at any point in 6 years, attempt SPAR Denominator: u First-Time (scrubbed) u Degree-seeking (at any point in 6 years, attempt transfer/degree applicable math or English) u 12 units (in 6 years) u This represents about 40% of students in our system 29

SPAR Numerator u Outcomes l l l Earned an AA/AS/certificate; OR Transfer: to a SPAR Numerator u Outcomes l l l Earned an AA/AS/certificate; OR Transfer: to a 4 -yr institution; OR Become “transfer-prepared”; OR u l l Completed 60 xferable units Became “transfer-directed”: u l the State wants: Completed both xfer level math AND English No double-counting, but any outcome counts SPAR Rate=51% 30

Tracking Transfers u SSN-level matches with CSU, UC u Nat’l Student Clearinghouse for private, Tracking Transfers u SSN-level matches with CSU, UC u Nat’l Student Clearinghouse for private, proprietary, for-profit, out of state Match 2 x/yr, send all records since 1992 l Update internal “xfer bucket” l u Works great for cohort tracking u Needed method for “annual volume” 31

Tracking Transfers u Annual l CSU/UC: they provide these figures based on their criteria Tracking Transfers u Annual l CSU/UC: they provide these figures based on their criteria u We l Volume of Transfers didn’t want to redefine this Private/Out of State: NSC “cross-section” cut method u Validated source l against CSU/UC xfers from NSC Added another 30% to annual volumes 32

97 -98 98 -99 99 -00 00 -01 01 -02 02 -03 03 -04 97 -98 98 -99 99 -00 00 -01 01 -02 02 -03 03 -04 04 -05 05 -06 06 -07 FTF → → → → T FTF → → → → R FTF → → → A FTF → → → N FTF → → S FTF → → → F FTF → → E FTF → R FTF 06 -07 MIN 12 UNITS 33

Sector 01 -02 02 -03 03 -04 04 -05 05 -06 06 -07 CSU Sector 01 -02 02 -03 03 -04 04 -05 05 -06 06 -07 CSU 50, 473 50, 746 48, 321 53, 695 52, 642 54, 391 UC 12, 291 12, 780 12, 580 13, 211 13, 462 13, 874 ISP 17, 070 15, 541 18, 100 18, 365 17, 840 18, 752 OOS 10, 762 10, 540 11, 150 11, 709 11, 726 11, 825 Total 90, 596 89, 607 90, 151 96, 980 95, 670 98, 842 34

Transfer: Sector of Choice % to Instate % to Out % to UC % Transfer: Sector of Choice % to Instate % to Out % to UC % to CSU Private of State White 17. 9% 60. 7% 11. 0% 10. 4% Afr. Am 11. 5% 51. 2% 18. 1% 19. 2% Hisp/Lat 15. 1% 67. 7% 12. 1% 5. 1% Asian 37. 0% 49. 9% 9. 2% 3. 9% 35

Demography of Transfer Demog FTF All XFER(06 -07) Stdents CSU XFERUC XFERISP XFEROOS Afr. Demography of Transfer Demog FTF All XFER(06 -07) Stdents CSU XFERUC XFERISP XFEROOS Afr. Am 9% 8% 5% 3% 11% 13% Asian 11% 12% 26% 8% 7% Hisp/ Latino 35% 29% 23% 16% 23% 13% White 29% 35% 37% 40% 44% 55% 36

The Rise of The Phoenix 96 -97 2, 166 97 -98 2, 829 98 The Rise of The Phoenix 96 -97 2, 166 97 -98 2, 829 98 -99 3, 374 99 -00 4, 194 00 -01 5, 055 01 -02 5, 586 02 -03 6, 515 03 -04 8, 222 04 -05 8, 585 05 -06 8, 134 06 -07 9, 216 37

Who Transfers to Phoenix? Ethnicity UC Asian 29. 3% 14. 2% 4. 6% 2. Who Transfers to Phoenix? Ethnicity UC Asian 29. 3% 14. 2% 4. 6% 2. 4% 5. 2% 16. 8% Hispanic/Latino 13. 6% 23. 8% 28. 6% White 39. 1% 43. 6% 37. 5% African American CSU Phoenix 38

Wage Study u What was the economic value of the degrees (AA/AS/certificate) we were Wage Study u What was the economic value of the degrees (AA/AS/certificate) we were conferring? u Required l data match with EDD Had to pass a bill changing EDD code to allow match 39

Wage Study u Take year all degree recipients in a given Subtract out those Wage Study u Take year all degree recipients in a given Subtract out those still enrolled in a CCC l Subtract out those who transferred to a 4 -yr institution l u Match degree wage data 5 years before/after 40

Wage Study u Separate out two groups: Those with wages of basically zero before Wage Study u Separate out two groups: Those with wages of basically zero before degree l Those with >$0 pre wage l u The result: The Smoking Gun of Success 41

42 42

Mapping Districts u CC Districts in CA are legally defined, have own elections, pass Mapping Districts u CC Districts in CA are legally defined, have own elections, pass own bonds u We did not have a district mapping for all 72 districts l So we couldn’t do district participation rates 43

Mapping Project u Get a cheap copy of ESRI Suite u Collect all legal Mapping Project u Get a cheap copy of ESRI Suite u Collect all legal district boundary documents u Find cheap labor—no budget for this 44

45 45

Peer Grouping u “Peers” l l l historically have been locally defined: My neighbor Peer Grouping u “Peers” l l l historically have been locally defined: My neighbor college Other colleges with similar demography Other colleges with similar size 46

Peer Grouping u Taking l peering to another level: Peer on exogenous factors that Peer Grouping u Taking l peering to another level: Peer on exogenous factors that predict the accountability metric’s outcome u Thus leaving the “endogenous” activity as the remaining variance l Cluster to create groups u We picked 6 clusters, with a min of 3 in a cluster l Each metric produces different factors, peers, clusters 47

Peer Grouping: Example u Peering the SPAR Rate: 109 rates as outcomes l Find Peer Grouping: Example u Peering the SPAR Rate: 109 rates as outcomes l Find data for all 109 that might predict outcomes/explain variance l Perform regression and other magical SPSS things l See how high you can get your R 2 l 48

Finding Data u What might affect a grad/transfer rate on an institutional level? Student Finding Data u What might affect a grad/transfer rate on an institutional level? Student academic preparedness levels l Socioeconomic status of students l First-gen status of students l Distance to nearest transfer institution l Student age/avg unit load l 49

Finding Data u We had to create proxy indices for much of these (142 Finding Data u We had to create proxy indices for much of these (142 tried) GIS system: geocode student zipcode/ZCTA l Census: lots of data to be crossed by zip/ZCTA l Create college “service areas” based on weighted zip/ZCTA values l u Different than district legal boundaries 50

51 51

Finding Data u The l Killer Predictor “Bachelor Plus Index”, or what % of Finding Data u The l Killer Predictor “Bachelor Plus Index”, or what % of service area population of college has a bachelor’s degree or higher u “Bachelor Plus Index” a proxy for: First gen l Academic preparedness l Socioeconomic status l Distance to nearest transfer institution l 52

Peering SPAR Rate u Exogenous Rate: factors that predict SPAR Bachelor Plus Index l Peering SPAR Rate u Exogenous Rate: factors that predict SPAR Bachelor Plus Index l % older students l % students in basic skills l u R 2 l =. 67 What’s left is implied institutional variance u Demo 53

Peering: What’s Bad u Its complex and somewhat confusing and labor intensive. u Colleges Peering: What’s Bad u Its complex and somewhat confusing and labor intensive. u Colleges traditional notion of “peer” is shaken u Multiple peers for multiple metrics; can change every year u You could do well vs. State average, increasing over time, but last in your peer group 54

Peering: What’s Good u Its complex and somewhat confusing u You will likely look Peering: What’s Good u Its complex and somewhat confusing u You will likely look good in some areas, OK in others, and low in others u Its not very likely anyone will be high or low in all 6 metrics u It eliminated rankings. 55

The ARCC Report u Is almost 800 pages. u Comes out every March. u The ARCC Report u Is almost 800 pages. u Comes out every March. u Takes 4 PY’s to complete (about 6 months/yr) u Is generally regarded highly in CA academic and Legislative circles. u DOF and LAO and Sec. of Ed love it. u Local Trustees/Boards love it. 56

The ARCC Collaboration u Has brought the system more money: $33 mil in basic The ARCC Collaboration u Has brought the system more money: $33 mil in basic skills l Increased noncredit reimbursement rates by $300/FTE l u Has brought about trust between system and State stakeholders. u Has educated both sides tremendously. 57

No More “Girlie-Man” Accountability! 58 No More “Girlie-Man” Accountability! 58