Breaking-down the Silos Working together towards solving the

Breaking-down the Silos: Working together towards solving the persistent problem of student success Marian Neale-Shutte, Qobo Qwaka, Andrea Watson & Kim Hurter SAAIR Annual Conference Windhoek, Namibia 24 -26 October 2017

Outline q q q q Institutional Profile Law Faculty Profile Collaboration and Aims & Objectives Sample and Methodology Results Descriptive Analyses Results Logistic Regression Analyses Implications and Reflections “Education is the most powerful weapon which you can use to change the world. ” - Nelson Mandela

NMU Institutional Profile

Nelson Mandela NMU Institutional Profile University Profile *Source: Office for Institutional Planning, Infographic Q 1

Nelson Mandela University Faculty of Law Profile – Nelson Mandela University Law Faculty Profile WHO ARE OUR LAW STUDENTS? ACADEMIC FTE STAFF: STUDENT RATIO (2016) BY FACULTY SUCCESS RATES OF ALL COURSEWORK MODULES *Source: Office for Institutional Planning, Infographic Q 2

Early warning, tracking & monitoring system • • • Under the auspices of the Siyaphumelela project. Pilot institutional E, W, M&T system. Draws in & integrates data from different databases. More integrated, real-time picture of students academic performance. Dashboard-like interface - will use indicators to flag students “in need of support. ” • Inform: • The indicators through research. • Interpretation of student data and trends. • “Just-in-time” targeted interventions.

• • HEMIS data Intelliweb data and reports Findings from Institutional Research studies Institutional and National Trends • • LAW Provide a broader view of students & student success. CAAR data Academic programme performance and interplay with biographical, school and Access Assessment variables Tell a more complete Story. • • Module marks Module performance, throughput, and anecdotal information Develop more comprehensive plans for Interventions. Encourage collaboration & capacity building. DATA RICH • • CAAR OIP Working together

Aim & Objectives CONNECT THE PARTS Investigate the relationship between pre-university demographic and education variables and academic success 1. Obtain a student description/profile. 2. Identify progression pathways & suggest benchmarks for risk identification. 3. Identify predictors of students at risk/ or “in need of support” FORMATION OF A WHOLE

Sample & Methodology CONNECT THE PARTS SAMPLE: n=725 • All 1 st time entering, first year students registered in the Law Faculty from 20112015 • Registered for: LLB EXT • Completed the CAAR access assessment • 1. Descriptive & pathway analyses: Obtain a student description/profile. 2. Identify progression pathways & suggest benchmarks for risk identification. • Logistic Regression analyses: 3. Identify predictors of students at risk or “in need of support” for LLB/LLB Extended FORMATION OF A WHOLE

Transforming data for the early warning, tracking & monitoring system WHY? WHAT’S BEST? HOW TO? VALUE WHAT? https: //kvaes. wordpress. com/2013/05/31/data-knowledge-information-wisdom/

Student Profile - LLB (n = 497)

Student Profile - LLB Extended (n = 228)

Pathway Indicator of Sequential Choice [PISC] Main objectives of the PISC: q Understand the pathways that students follow through the LLB and LLB Extended programmes q For the current presentation ü Identify the pathways followed ü Identify points for targeted intervention q For further investigation ü Identify student profiles for specific pathways followed ü Identify predictors of academic performance for the different pathway categories *Robinson, R. A. (2004). Pathways to Completion: Patterns of Progression through a University Degree. Higher Education, 47, 1, pp. 1 -20.

PISC – Graduated students Student progression to graduation occurred over a period of 4, 5 or 6 years with some stop-outs and transfers out of the faculty and then back into the LLB. 97 Graduated LLB/Ext 7 LLLL_G LLLL_S_L_G LLLLLL_G LTTTL_G XXXXXX_G

PISC- Continuing students Of the students still continuing we can immediately see where intervention, or at a minimum monitoring, might be important. 97 18 Continuing in LLB/Extended LL LLLLLL L_S_S_S_LL L_SL LL_S_LLL LL_S_S_L XX XXXXXX X_S_XXX 25 Continuing in BA(Law) L_S_S_LB LLBBB LLLBB XXXXBB

PISC – Transferring Students What are the pathways of students who start in the LLB or LLB Extended but end up somewhere else …? 97 Graduated Other 30 LL_S_TT_G LL_S_TTT_G LLTBB_G LLTTT_G LTTTT_G XTTT_G XXTTT_G Registered L_S_S_T LLT_ST LLT_S LLLTTT LT LTT LLTTTT LTTTTT XXTT XXXTT XT XXXTTT XXX_S_T XXTTT 40 Dropped-out LLLT_D LTTT_D X_S_T_D XX_S_T_D XXTT_D

PISC – Departed Students We tend to think of drop-out in simplistic terms, rather than in terms of persistence, time spent and resources spent before drop-out occurs … 97 34 Lost to the university and potentially to law Dropped-out L_D LLL_D LLLT_D LTTT_D XTT_D X_S_T_D XXX_D XX_S_T_D XXXX_D XXTT_D XX_S_X_D XLL_D Stopped-out L_S LLL_S LLLL_S LT_S X_S XB_S XXX_S XXXTT_S XXXT_S XTT_S XXXXT_S

Defining academic performance • Considered academic performance in three ways: Credits Passed Academic Average The total of a student’s The average percentage first semester / first year obtained across all credits passed. modules taken in the first semester / first year building in the relative course weight of the modules taken. % of Modules Passed The total number of modules passed in the first semester / first year divided by the total number of modules registered for in the first semester / first year (expressed as a percentage)

Determining patterns of academic performance q Looked at various patterns of academic performance across different kinds of groups of students. q The following groups of students performed differently: Different school performance in terms of the cumulative APS Had Maths as opposed to Maths Literacy Had NSC English Home Language as opposed to another school Home Language Were Directly Admitted (met minimum requirements) as opposed to those Tested Admitted (gained admission through access testing) – Males and Females – School Quintiles in terms of Quintiles 1 -3 as opposed to Quintiles 4 -5 and P/I – –

Logistic Regression - Variables LLB Extended Dependent Variables: First Semester Academic Average Independent Variables: • • Semester 1 modules: Introduction to Law (JLK 111) Law of Persons (JLP 111) Constitutional Law (JJT 111) Admission Point Score (APS) Maths / Maths Lit NSC Home Language Gender School Quintile Tested Admitted / Directly Admitted • • • Semester 1 modules: Introduction to Law (JLK 1 X 1) Law of Persons (JLP 1 X 1) APS Maths / Maths Lit NSC Home Language Gender School Quintile

Logistic Regression - LLB Model: Distribution : BINOMIAL, Link function: LOGIT Modeled probability that Academic Success_1 st Sem = lt_60% (i. e. mark not >= 60, therefore ‘in need of support’) Lower CL Upper CL Level of Degree of Estimate Standard Error Wald’s Chi-Square p-level Effect Freedom -95. 00% 1 23. 30473 3. 815897 37. 29879 15. 82571 30. 78375 0. 000000 APS_best_7_Sum 1 -0. 02583 0. 006552 15. 54890 -0. 03867 -0. 01299 0. 000080 JLK 111 T 1 1 -0. 07679 0. 015506 24. 52693 -0. 10719 -0. 04640 0. 000001 JLP 111 T 1 1 -0. 03323 0. 011362 8. 55393 -0. 05550 -0. 01096 0. 003448 JJT 111 T 1 Q 1 -3 Gender_F NSC_ENG NSC_ML TAS Scale 0 0 0 1 1 1 -0. 07812 0. 36182 -0. 26875 0. 37568 -1. 47033 -0. 20653 1. 00000 0. 010270 0. 432697 0. 313451 0. 373376 0. 379309 0. 376420 0. 000000 57. 86106 0. 69922 0. 73514 0. 99474 15. 02599 0. 30103 38. 78218 -0. 09825 -0. 48625 -0. 88311 -0. 36259 -2. 21376 -0. 94430 1. 00000 -0. 05799 1. 20989 0. 34560 1. 11395 -0. 72690 0. 53124 1. 00000 0. 000000 0. 403045 0. 391222 0. 318587 0. 000106 0. 583239 Intercept p < 0. 05

Logistic Regression - LLB Model: Distribution : BINOMIAL, Link function: LOGIT Modeled probability that Academic Success_1 st Sem = lt_60% (i. e. mark not >= 60, therefore ‘in need of support’) Level of Effect APS_best_7_Sum JLK 111 T 1 JLP 111 T 1 JJT 111 T 1 Q 1 -3 Gender_F NSC_ENG NSC_ML TAS p < 0. 05 0 0 0 Odds Ratio 0. 974497 0. 926081 0. 967315 0. 924853 1. 435939 0. 764331 1. 455986 0. 229850 0. 813405 Lower CL -95. 00% 0. 962063 0. 898359 0. 946012 0. 906423 0. 614927 0. 413496 0. 695873 0. 109289 0. 388953 Upper CL 95. 00% 0. 987091 0. 954658 0. 989098 0. 943658 3. 353115 1. 412835 3. 046379 0. 483407 1. 701045

Evaluation of the Model - LLB • The classification matrix for the model was examined: Logistic Regression: LLB Analysis Sample Odds Ratio: 21. 178427 Log odds ratio: 3. 052983 Predicted: It_60% Observed: It_60% 141 Observed: gteq 60% 32 Predicted: gteq 60% 31 149 Percent correct 81. 98 82. 32

Logistic Regression - LLB Extended LLB Ext Model: Distribution : BINOMIAL, Link function: LOGIT Modeled probability that Academic Success_1 st Sem = lt_60% (i. e. mark not >= 60, therefore ‘in need of support’) Lower CL Upper CL Level Degree of Estimate Standard Error Wald’s Chi-Square p-level of Freedom -95. 00% Effect Intercept 1 9. 64460 4. 173685 5. 33985 1. 46433 17. 82487 0. 020843 APS_best_7_Sum 1 -0. 01021 0. 009579 1. 13661 -0. 02899 0. 00856 0. 286370 JLK 1 x 1 T 1 1 -0. 01298 0. 021412 0. 36723 -0. 05494 0. 02899 0. 544519 JLP 1 x 1 T 1 1 -0. 07632 0. 016997 20. 16292 -0. 10964 -0. 04301 0. 000007 0 1 -0. 37070 0. 768045 0. 23295 -1. 87604 1. 13464 0. 629345 Q 1 -3 0 1 0. 47106 0. 465110 1. 02577 -0. 44053 1. 38266 0. 311154 Gender_F NSC_ENG 0 1 -1. 00121 0. 804836 1. 54750 -2. 57866 0. 57624 0. 213504 NSC_ML 0 1 -1. 07642 0. 464844 5. 36227 -1. 98750 -0. 16534 0. 020577 Scale 1. 00000 0. 000000 1. 00000 p < 0. 05

Logistic Regression - LLB Extended LLB Ext Model: Distribution : BINOMIAL, Link function: LOGIT Modeled probability that Academic Success_1 st Sem = lt_60% (i. e. mark not >= 60, therefore ‘in need of support’) Level of Effect Odds Ratio (unit change) Lower CL -95. 00 % Upper CL +95. 00 % APS_best_7_Sum 0. 989839 0. 971428 1. 008599 JLK 1 x 1 T 1 0. 987108 0. 946539 1. 029416 JLP 1 x 1 T 1 0. 926518 0. 896160 0. 957903 Q 1 -3 0 0. 690254 0. 153196 3. 110067 Gender_F 0 1. 601698 0. 643693 3. 985499 NSC_ENG 0 0. 367436 0. 075876 1. 779343 NSC_ML 0 0. 340813 0. 137038 0. 847603 p < 0. 05

Evaluation of the Model – LLB Extended • The classification matrix for the model was examined: Logistic Regression: LLB Ext Analysis Sample Odds Ratio: 12. 373737 Log odds ratio: 2. 515576 Predicted: It_60% Predicted: gteq 60% Percent correct Observed: It_60% 34 18 65. 38 Observed: gteq 60% 10 71 87. 65

Logistic Regression - Limitations & Future Research q Consider that other biographical or non-cognitive variables could play a role in predicting “academic success” for example, motivation, study habits, self-efficacy, perseverance, and so forth. Future studies should consider gathering and adding in such variables as predictors. q Future studies should consider adding in variables from Learning Management System data as predictors. q Future studies should investigate and analyse potentially significant predictors in relation to academic success for other faculties (whether models that successfully predict “academic success” in other faculties and programmes are similar, or whether they differ from these models for these programmes in the Law Faculty).

Logistic Regression - Implications q Build LR “models” into the early warning system: q Once the results for the first semester test for each relevant first semester, first year Law module for the LLB and LLB Extended programmes are available, the data for the biographical and school-based predictors for each new intake of first year Law students, could then be drawn in. q The automated system could then use the logistic regression equations to calculate which students are predicted to be potentially academically unsuccessful in the first semester exams, and who are thus “in need of support. ” q It would then automatically generate a list (report) identifying those students q In this way “students in need of support” in the LLB and LLB Extended programmes can be timeously identified in the first semester - while support is still possible. q These students could then be contacted by lecturers and referred to the Faculty academic advisor, who would work with the lecturers and CTLM staff, to point them towards the sorts of interventions that may best support them. q Interventions can be undertaken by Law Faculty lecturers & university support services for these identified students. q These students’ further academic progress in the semester would be monitored, and their uptake of support tracked via the system.

Breaking the silo’s Tells a more complete Story. FORMATION OF A WHOLE Provides a broader view of students and student success. Informed the Siyaphumelela project & development of the institutional RADAR system. Created space for exploration of new research techniques, thus contributing to professional development for all involved. Provided a clearer description of typical law student trajectories & important predictive variables. Can support & inform CAAR practices around access to the Law Faculty programmes for students who do not meet direct entry requirements. Informs understanding of the student body. Can inform decisions around the development of academic support interventions to assist students in the Faculty. Points to possible areas for consideration for recurriculation/ targeted interventions within the curriculum. Empirical evidence to support the development & use of RADAR in the Law Faculty. Permits more comprehensive plans for support Interventions. KNOWLEDGE LAW CAAR OIP Working together

Working together WHY? WHAT’S BEST? HOW TO? - Breaking the silo’s VALUE WHAT? https: //kvaes. wordpress. com/2013/05/31/data-knowledge-information-wisdom/

Marian Neale-Shutte marian. neale-shutte@mandela. ac. za Qobo Qwaka qobo. qwaka@mandela. ac. za Andrea Watson andrea. watson@mandela. ac. za Kim Hurter kim. hurter@mandela. ac. za

www. mandela. ac. za