Foundation Science Student performance explained Insight gained at

Foundation Science Student performance explained. Insight gained at UKZN. Nicola F. Kirby

At UKZN Normal entry requirements into BSc LES stream • Pre 2009: APS = 34; HG E or SG B Maths & Science • 2009: APS = 28; level 4 Maths & Science Centre for Science Access Level 4 English and LO Two types of students: Augmented Programme 1. almost meet normal Faculty entry requirements Pre 2009: APS = 28, HG E or SG B Maths and Science, 2009 onwards: APS = 22, Maths (3), 1 Science subject level (3) Foundation Programme Pre 2009: APS = 20, SG F Maths and Science, 2. those who are considerably weaker 2009 onwards: APS = 16, Maths (2), 1 Science subject level (2)

Access to the CSA • come from disadvantaged schools. • particular selection procedure that applies to Foundation students • Maths and Science selection tests • Selection model score = weighted “M Score” + selection test scores

2008 APS Large effect size btw Foundation & ESL and EFL n = 13 (in 2007) n = 46 n = 106 n = 79

2008 V. small effect size btwn Foundation and each of others. (N. S. D btwn Foundation and EFL) First year Biology module

2009 - new kind of student, same trend V. small effect size btwn Foundation and each of others. (N. S. D btwn Foundation and EFL) n = 21 n = 43 First year Biology module n = 209 n = 117

Number of students proceeding from the Foundation Programme (2006 -2009) Year Original intake Attrition from Programme *1 Number of students proceed from Programme 2009 109 64 45 2008 87 42 45 2007 66 41 25 2006 97 62 35 Note. *1 Attrition includes a few students who withdraw each year for a variety of personal reasons, the balance are academically excluded either in June or November. Results reflect proceed rates after supplementary exams results released.

Foundation works, how can we increase the student numbers going into 1 st year? What factors are most important in determining the performance of Foundation students in their access year? • the challenges may be better understood, • selection mechanisms refined, and • curriculum development informed.

Classification and Regression Tree Analysis • non-parametric • diagnostic and predictive • avoids complexities and restrictive assumptions of generalized linear modeling techniques • reveal the explanatory variables which best divide the data into homogenous, dichotomous groups • The analysis exposes a hierarchy of context dependent effects of the explanatory variables clear picture of the interaction between factors • Ecology, medicine, education outside South Africa

Two cohorts(2008: n= 79) (2009: n=88) • • • gender home language school quintile matric score endorsement English as first or second language English APS/ “equiv. ” Maths APS/ “equiv. Physics APS/ “equiv. ” Biology done at school Biology APS/ “equiv. ” Grades (08) • Maths selection test score • Science selection test score • English SATAP score (Standardised Assessment test for Access and Placement) • Selection Model Score • “M score” • accommodation • financial support • travel (08 only) • Senior certificate (35)/ NSC (53) (09)

Equivalent? 2008 Norms (Umalusi) Old HG (400) New System Old SG (300) • A (320 – 400) 7 A (240 - 300) • B (280 – 319) 6 B (210 - 239) • C (240 – 279) 5 C (180 - 209) • D (200 – 239) 4 D (150 - 179) • E (160 – 199) 3 E (120 - 149) • F (136 – 159) 2 F (100 - 120) • FF (120 – 135) 1 (and below) FF G 1 (100 -119) G 2 (80 – 99) GG H G GG H

2008 Foundation Overall Mark • Selection Model Score, 56 • 30% • Surrogate Physics APS • NB: English SATAP and financial support • when considered, “M score” ½ as good as SM • “ M score” model revealed English SATAP and maths selection test effect

2009 Foundation Overall Mark Selection Model Score, 56 again

2009 To date, cut off has been 52, but many students accepted with not even that. Surrogate: Accommodation almost as good NB: Financial support and maths selection test when considered, “M score” did not feature.

2009 Foundation Overall Average Mark Inverse relationships with Science selection and English SATAP

Overall Average • Can predict 2009 using SM score only (correlation between predicted and actual 2009 mark r = 0. 450**, large effect size) • Can NOT predict 2009 using 2008 M score model (+ English SATAP and maths selection) (r= 0. 03 N/S) in spite of ability to describe in 2008 in absence of selection model score

• selection model score has great value • not been used to its best potential • selection model can be improved further • “M score” diminished from 08 to 09 (biology an exception) • maths selection test growing importance • financial aid important

Accommodation primary splitter For whole Tree, SM most important (accommodation 74. 6%) 2008 Foundation Proceed decision

Accommodation primary splitter For whole Tree, SM most important (accommodation 46. 2%) 2009 Foundation Proceed decision

• No single school history indicator of potential appears to exist • Maths selection tests increasingly important • “M score” appears to have lost some of its discriminatory value.

• Once selected, providing a place in residence will improve that student’s chances of proceeding from the Programme considerably. • Mechanisms allowing access to students, should be extended to include socio-economic issues. Thank you

Can we improve the Selection Model Score? 1. Make it simpler? Do away with selection tests? • What about M score only? (only Maths and Science Matric subjects) 2. Make it better? Add selection tests? • What about the English SATAP test?

Proceed decision • M is a surrogate but not strong enough to be reflected in Tree • Can predict 2009 using SM score and accommodation (r = 0. 500**) • In 2009, relative to SM, accommodation is even more important than it was in 2008. 2009 can be predicted well on basis of accommodation alone (r = 0. 37**, medium to large effect) • Can NOT predict 2009 using M score (and other variables important in absence of Selection Model score)

Whether a student met the augmented stream requirements or not… • Difference in overall Foundation mark, – those who had met the augmented requirements (M =53. 2, SE = 1. 13) and those who had not (M =55. 83, SE = 1. 53), t (86) = 0. 16, N/S ( very small effect, r = 0. 15) • Significant association in proceed decision, 2 (1) = 4. 632, p < 0. 05. • Of those foundation students that did not meet the augmented stream requirements (N= 41), 63. 4% proceeded from the Programme. • Of those foundation students that did meet the augmented stream requirements, (N = 47), only 40. 4% proceeded. • Based on the odds ratio, the odds of a student proceeding are 2. 54 times higher if they did not meet the augmented stream requirements.

To EXPLAIN 2008: Chemistry Physics Selection model score English SATAP Selection model score Maths Selection model score Travel and Financial support Biology English SATAP English APS Science English SATAP Communication English APS Overall Mark Selection model score Proceed or not Accommodation Selection model score

To PREDICT 2009: Chemistry Selection model score M score no good Physics Selection model score M score no good Maths Selection model score M score no good Biology English SATAP and M score Science English SATAP Communication English APS Overall Mark Selection model score M score no good Proceed or not 1. Accommodation+ 2. Selection model score M score no good

Foundation selection procedure favours Physics, Chemistry and Maths Not selecting students for Biology or Science Communication English language proficiency determines success in these modules (SATAP not School English)

The selection model (SM) score is a good predictor of performance in Physics, Chem and Maths It is also the best predictor for average overall mark. But not for proceed/exclude. Accommodation is a better indicator of proceed/exclude. When devising a new SM- work with Average overall mark instead

Selection Model score: 24. 32+0. 28(M score)+0. 22(Maths selection test)+0. 07 (Science selection test) Selection pool doesn’t allow selection procedure to reach its potential Auto-accept cut off for SM: score >or = 52 2008: 21/79 (27%) 2009: 13/88 (15%) Foundation performed better than Augmented and ESL students in First year module; performance in Foundation year of those who met augmented requirements. Widen the Access Body, increase the selection pool, rethink models of Access , use selection model cutoffs

• SM can be improved • Science test is problematic • English SATAP has predictive value for Biology and Science Communication • Linear regression used to devise new selection model

New Selection Model 08= 25. 406 + (0. 291*Mscore) +(0. 185*Maths selection test) + (0. 123*SATAP English)

• But New Selection Model 08 was not as effective in predicting 09 average mark or proceed desicion (0. 233*, 0. 226*; small to medium effect). • Devise New Selection Model 0809 by combining cohorts • 31. 858+(0. 229*Mscore) + (0. 207*Maths selection)

31. 858+(0. 229*Mscore) + (0. 207*Maths selection) New Selection Model 0809 can successfully predict 09 average overall mark (r= 0. 4)** Cut off of 54 for a passing average NO English SATAP NO Science Test

New Selection Model 0809 can successfully predict 09 proceed (r= 0. 34)** Cut off of 53 English is not helpful in establishing success for all students

2008 and 2009 cohorts Selection model best distinguishes students, but accommodation is most important (normalized independent variable importance of accommodation 100%) New 08 09 selection model not quite as good as original (as surrogate: 9. 702 improvement, association of 0. 8) BUT more succinct AND…

31. 858+(0. 229*Mscore) + (0. 207*Maths selection) New 0809 selection model best selects students to passas long as they are given a place in Residence. Accommodation still most important. New 0809 selection model does not include science selection test or English SATAP

Further curriculum development towards embedding English language proficiency and literacy in the Foundation Programme and beyond “Epistemological Access”