Conducting programme evaluations/reviews with a view to sustainable quality promotion Jan Botha University of Stellenbosch South Africa Keynote Address Oman Quality Network Conference Muscat, 28 October 2008
Key questions • Can we apply the 80/20 principle to programme evaluations? • What are universities learning from the evaluation of (formative) programmes? 2
Outline of the paper 1. Suggestions for the academic leader 2. QA in higher education context 3. Misconceptions about QA in higher education context 4. Programme evaluations and the 80/20 principle 5. What are we learning from programme evaluations 3
Suggestions for the academic leader
The predicament of the academic leader • My research … • • I have become “one of them” In the middle of the sandwich From critic to custodian I have to “manage”… “quality”
Academic Leadership Capabilities • “Learning Leaders in times of change” • Geoff Scott, Hamish Coates, Michelle Anderson (May 2008)
Four suggestions for the Academic Leader 1. Use quality management as instrument to achieve your vision and goals for your School / Department / Programme 2. Quality management can be a powerful change mechanism 3. Synchronise and integrate evaluation activities 4. Use an academic and research-informed approach to quality management
QA as an Academic Project • QA to be approached as a form of research • • • an intellectual enterprise driven by curiosity, creativity and pragmatics underpinned by social-scientific inquiry accompanied by experimentation and reflection informed by theorising resulting in application and publication
A Principle of Design Specifications are continually moving upwards Opportunity for incremental changes Specification s Calculate this cost Poor design Good design Poor design
Quality Assurance in Higher Education Context
“Living with ideology in the university” • Ronald Barnett* The “idea of the university” as “a site of reason” • Universities are beset by many ideologies Foucault: ideology is “that which is opposed to truth” • Competition and Entrepreneurialism • Quality • Managerialism • Research • Access, inclusivity, multicultiralism 2. Turn the power of ideology onto itself, move from ideologies to ideaologies – recall the idea of the university * Beyond all reason. Living with ideology in the university. 2003. Open University Press.
The idea of the university? “Athens” Knowledge for the sake of knowledge; educated citizen; wisdom (e. g. Philosophy Department) Pectora Roborant Cultus Recti “Berlin” All-round humanist education; transmission of culture; research in basic disciplines (e. g. Physics Department) “New York” Market, entrepreneurialism; professions (Business School) “Culcutta” Focus on problems and needs of the community; applied research (Nursing School, Engineering, Agriculture) 12
What are universities for? • Universities until recently believed their existence and value was self-evident • Pressure of governments: universities should address the needs of society (and become “skills factories”? ) • Universities state their vision, mission and objectives: • • • A place of quality, a place to grow Creating futures Your Knowledge Partner Innovation Generation To advance Learning and perpetuate it to Posterity • Sultan Qaboos University aspires to be an academic institution of excellence recognized both nationally and internationally, a center for learning where individuals can develop their capabilities in an environment that promotes academic achievement and research excellence; and a community where the highest moral and ethical values prevail for the purpose of both self-advancement and service to the community.
International snapshot • International Network for Quality Assurance Agencies in Higher Education (INQAAHE)… a worldwide association of some 200 organisations • Members include • • • Oman Accreditation Board Commission for Academic Accreditation, UAE Netherlands Quality Agency (NQA) Malaysian Qualifications Agency (MQA) Higher Education Commission, Pakistan CONEAU - Comision Nacional de Evaluacion y Acreditacion Universitaria, Argentina • National Accreditation Agency (NAA), Russian Federation
India • The National Assessment and Accreditation Council (NAAC) • accredits institutions of higher education • size of Indian system: assessing hundreds of general education programs in 12 000 institutions of higher education is not feasible • institutional accreditation for general education institutions has been the strategy
United States of America • Many decades of institutional and programme accreditation in the USA • Council on Higher Accreditation • Six regional accreditation organizations (MSA, SACS, NEASC, NWCCU, NCA, WASC) • Specialised accreditation agencies, e. g. ABET • 2, 800 programs at more than 600 universities, over 1, 500 dedicated volunteers participate annually in ABET activities • As many as 142 calendar days devoted in one year to visiting panels
South Africa • Council on Higher Education (CHE) • “just hang on, it will go away, as happened with the other acronymous bodies…” • Institutional Audits • Programme Accreditation • National Reviews of Programmes • Quality Promotion and Capacity Building • Three steering instruments in Higher Education
(Slide: Ian Bunting)
Misconceptions about Quality Assurance in Higher Education
Challenging the “management” of quality • Is “managerialism” not contradictory to the nature of a university ? • “Management” of quality used to cover for poor quality? • More management of quality results in less quality… • CHE publication • Academic Freedom, Institutional Autonomy and Public Accountability in South African Higher Education (August 2008) (www. che. ac. za)
Misunderstandings (1) • Point of departure: lecturers are doing poor quality work, • we need QA to ‘catch the lazy buggers out” • I am satisfied with 95% of the work of the people in my university/faculty/institution • therefore QA is not necessary • Just make sure that you meet the minumim standards and “keep them off your back” • Aim for the minimum • Comply…
Misunderstandings (2) • If good academics are appointed at universities, we do not need QA systems. If the wrong people are appointed, no document (however thick it is), will ensure quality. • If nobody steals, we do not need laws and police. If there are thieves, no law (however thick it is), will ensure that no theft takes place. • If nobody is ill, we do not need doctors. Since we know beforehand that everyone is in any case going to die some day, the whole medical profession is senseless.
The case for Quality Management • Quality management is a necessary condition for quality, however, • quality management is not sufficient for quality
Programme evaluations and the 80/20 principle
Acknowledgement • Co-researchers • Prof Carools Reinecke • Emeritus Vice-Chancellor, Potchefstroom University (South Africa) • Mr Mr Kamal Bhagwandas Jogibhai • University of the Witwatersrand, Johannesburg, South Africa • Research Assistant • Mr André Müller, University of Stellenbosch 25
The Pareto Principle • Vilfred Pareto (1848 -1923) • Wealth distribution in 19 th Century England is predictably unbalanced • Most income and wealth go to a minority of people • Consistent mathematical relationship between proportion of people and amount of income and wealth • Principle of least effort • Principle of Imbalance • Zipf’s Principle of Least Effort • Resources tend to arrange themselves to minimize work • 20 -30 % of resources accounted for 70 -80% activity • Jurian’s Rule of the Vital Few or the 80/20 Principle 26
The 80/20 Principle illustrated Effort Results R Koch 2007. The 80/20 principle. The secret of achieving more with less. London: N Breadly Publishing 27
Purposes and types of programme evaluations • Mouton & Babbie (The practice of social research. Oxford, 2001): • judgement-orientated evaluations • improvement-orientated evaluations • knowledge-orientated evaluations • Trow (Academic reviews and the culture of excellence. Stockholm 1994): • internal supportive • internal evaluative • external supportive • external evaluative • Evaluation outcomes used by different role players for different purposes 28
What is a programme? • A social intervention • A set of planned, structured learning experiences leading to a qualification • One qualification with various programmes leading to that qualification • BSc in Biodiversity and Ecology • BSc in Mathematical Sciences • One programme in a “field” with various qualifications • The “programme in mathematical sciences” consists of a Certificate, Diploma, Bachelor, Masters, and Ph. D 29
Unit of evaluation/review • An institution as a whole • An academic organisational units within an institution • department, centre, institute, bureau, school, faculty • Specific activities or processes within an institution • • 30 learning and teaching programmes research programmes community engagement programmes capital campaigns
Types of programme evaluations • Evaluation of professional programmes as part of the periodic programme accreditation process • conducted by professional councils • National reviews of programmes • conducted by the a national QA body • the same programme evaluated and (re)accredited simulatenously at all institutions • Internal programme evaluations/reviews (for different purposes: improvement/closure/consolidation etc. ) • conducted by institutions • professional and formative programmes • undergraduate and postgraduate programmes 31
Challenges in programme evaluation • Coherence within broader internal and external QA system • Formative programmes as unit of evaluation • BA, BSc, BComm, B Soc. Sc • Spread over many departments in Schools and Faculties • Conflating “evaluation” with “reporting the results of an evaluation process” • Too much effort and resources for outcomes? 80/20 • Administrative burden • Evidence 33
34
Expectations of Programme Evaluations • Sustainable quality promotion • An instrument for change management • Integrated with other forms of evaluation • Outcomes should justify the effort • for academics and evaluators and HE managers • Standard methodology used • self-evaluation based on criteria/standards, report, panel visit, report, improvement action • Applicable to formative programmes 35
A “Pareto Approach” to Programme Evaluation 1. Consider the ideal situation • refer to criteria / standards • clustered in 11 themes • simplified (and “translated”) 2. Formulate a key question (or questions) • focus, contextualise (for us) 3. Express the evaluation in a symbol (5 point scale) discipline to make a judgement 4. Motivate the evaluation symbol (brief narrative, with substantiating evidence) 5. Formulate strategies (in bullet points) 36
Template for the Self-evaluation Report 1. Programme rationale Criteria The programme is consonant with the faculty’s mission, planning and resource allocation. The design maintains an appropriate balance of theoretical, practical and experiential knowledge and skills. It has sufficient disciplinary content and theoretical depth at the appropriate level. The programme offers opportunities for community interaction. The design offers learning and career pathways to students with opportunities for articulation with other programmes within and across institutions, where possible. The 2003 formulation of the strategic importance of the programme was re-evaluated. 1. 2 Key Question 1. 3 Evaluation 1. 4 Motivation 1. 5 Strategy Reports limited (8, 9 pages). Strategies formulated as actions 37
Criteria/standards clustered in themes 1. 2. 3. 4. 5. 6. 7. 8. 9. Programme rationale Academic integrity Student recruitment, admission and selection Staff Learning facilitation Assessment Infrastructure and academic information sources Programme co-ordination Student success and academic support for student success 10. Service Learning and work-based learning 11. Programme Evaluation and Development 12. Research basis (postgraduate programmes) 38
Case study • Faculty of Agri-Sciences, Stellenbosch University • BSc. Agric in Animal Production Systems; Agricultural Economics; Wine Production Systems; Crop Production Systems; Forestry; Food Science • (BAgric at Elsenburg College) • Faculty of Sciences, Stellenbosch University • BSc in Physics; Chemistry, Mathematical Sciences, Earth Science, Biodiversity, Molecular Biology, Human Life Sciences, Sport Science, Education • Faculty of Humanities, University of the Witwatersrand (Johannesburg) 39
Rationale Academic integrity Student recruitment, admission, selection Staff Facilitation of learning Assessment Infrastructure and access to academic information Programme coordination Student success and student support Service learning and workbased learning Programme evaluation and development Agri-Sciences Programme 1 4 5 5 4 3 5 4. 45 Programme 2 4 3 4 4 4 5 3 5 5 4. 27 Programme 3 3 4 2 3 4 3 3 2 4 3 3 3. 09 Programme 4 3 4 4 3 5 4 3 2 4 2 3 3. 36 Programme 5 3 4 2 3 4 5 4 4 3 3. 64 Programme 6 3 3 2 4 3 4 2 4 3 3 4 3. 18 3. 33 3. 83 3. 33 3. 67 4. 00 3. 33 3. 83 40 3. 00 3. 67 4. 17
Rationale Academic integrity Student recruitment, admission, selection Staff Facilitation of learning Assessment Infrastructure and access to academic information Programme coordination Student success and student support Service learning and workbased learning Programme evaluation and development Agri-Sciences Programme 1 4 5 5 4 3 5 4. 45 Programme 2 4 3 4 4 4 5 3 5 5 4. 27 Programme 3 3 4 2 3 4 3 3 2 4 3 3 3. 09 Programme 4 3 4 4 3 5 4 3 2 4 2 3 3. 36 Programme 5 3 4 2 3 4 5 4 4 3 3. 64 Programme 6 3 3 2 4 3 4 2 4 3 3 4 3. 18 3. 33 3. 83 3. 00 3. 33 3. 67 4. 00 3. 33 3. 83 41 3. 67 4. 17
Rationale Academic integrity Student recruitment, admission, selection Staff Facilitation of learning Assessment Infrastructure and access to academic information Programme coordination Student success and student support Service learning and work-based learning Programme evaluation and development Sciences Programme 1 3 3 4 3 3 - - 2. 64 Programme 2 3 4 4 4 5 5 4 4 3 4. 00 Programme 3 5 4 3 3 4 4 5 4 4 2 4 3. 82 Programme 4 3 3 3 4 4 4 2 5 3. 64 Programme 5 4 4 3 4 4 4 4 3. 91 Programme 6 3 4 2 4 3 5 4 2 4 - 3 3. 09 Programme 7 5 5 4 5 3 4 5 4 4 - 5 4. 00 Programme 8 4 4 4 2 3 3. 36 Programme 9 4 3 3 4 4 4 5 4 3 - 4 3. 45 Programme 10 3 5 3 4 4 4 3 2 3 3. 45 Programme 11 4 3 3 3 2 2 4 3 3 3. 00 Programme 12 5 4 4 5 3 4 5 4 4 - 5 3. 91 3. 83 3. 33 3. 92 3. 58 3. 92 4. 00 3. 58 3. 67 1. 58 3. 50 42
Rationale Academic integrity Student recruitment, admission, selection Staff Facilitation of learning Assessment Infrastructure and access to academic information Programme coordination Student success and student support Service learning and work-based learning Programme evaluation and development Sciences Programme 1 3 3 4 3 3 - - 2. 64 Programme 2 3 4 4 4 5 5 4 4 3 4. 00 Programme 3 5 4 3 3 4 4 5 4 4 2 4 3. 82 Programme 4 3 3 3 4 4 4 2 5 3. 64 Programme 5 4 4 3 4 4 4 4 3. 91 Programme 6 3 4 2 4 3 5 4 2 4 - 3 3. 09 Programme 7 5 5 4 5 3 4 5 4 4 - 5 4. 00 Programme 8 4 4 4 2 3 3. 36 Programme 9 4 3 3 4 4 4 5 4 3 - 4 3. 45 Programme 10 3 5 3 4 4 4 3 2 3 3. 45 Programme 11 4 3 3 3 2 2 4 3 3 3. 00 Programme 12 5 4 4 5 3 4 5 4 4 - 5 3. 91 3. 83 3. 33 3. 92 3. 58 3. 92 4. 00 3. 58 3. 67 1. 58 3. 50 43
Academic integrity Student recruitment, admission, selection Staff Facilitation of learning Assessment Infrastructure and access to academic information Programme coordination 4 5 5 4 3 5 Programme 2 4 3 4 4 4 5 3 5 5 Programme 3 3 4 2 3 4 3 3 2 4 3 3 Programme 4 3 4 4 3 5 4 3 2 4 2 3 Programme 5 3 4 2 3 4 5 4 4 3 Programme 6 3 3 2 4 3 4 2 4 3 3 4 3. 33 3. 83 3. 00 3. 67 4. 17 3. 33 3. 67 4. 00 3. 33 3. 83 Student success and student support Service learning and work-based learning Programme evaluation and development Rationale Programme 1 Student recruitment, admission, selection Staff Facilitation of learning Assessment Infrastructure and access to academic information Programme coordination 3 3 4 3 3 - - 2. 64 Programme 2 3 4 4 4 5 5 4 4 3 4. 00 Programme 3 5 4 3 3 4 4 5 4 4 2 4 3. 82 Programme 4 3 3 3 4 4 4 2 5 3. 64 Programme 5 4 4 3 4 4 4 4 3. 91 Programme 6 3 4 2 4 3 5 4 2 4 - 3 3. 09 Programme 7 5 5 4 5 3 4 5 4 4 - 5 4. 00 Programme 8 4 4 4 2 3 3. 36 Programme 9 4 3 3 4 4 4 5 4 3 - 4 3. 45 Programme 10 3 5 3 4 4 4 3 2 3 3. 45 Programme 11 4 3 3 3 2 2 4 3 3 3. 00 Programme 12 5 4 4 5 3 4 5 4 4 - 5 3. 91 3. 83 3. 33 3. 92 3. 58 3. 92 4. 00 3. 58 3. 67 1. 58 3. 50 Programme evaluation and development Academic integrity Programme 1 Student success and student support Service learning and work-based learning Rationale 4. 45 4. 27 3. 09 3. 36 3. 64 3. 18
What are we learning from programme evaluations?
1. Programme rationale (69) • To enhance interaction with stakeholders (subject specific societies, industry, extraordinary lecturers, alumni) to broaden academic and industry specific networks (through a voluntary advisory committee) • To review/restructure the content/focus of the programme to ensure that module outcomes strongly reflect the programme specific and generic outcomes, reviewing programme structures annually and implementing student feedback and industry input, and developing a formal system to gather information from graduates on their appraisal of the course, and suggestions for improvement. • To adapt and better integrate courses, share expertise/reduce duplication and use resources more cost-effectively, reducing lecturing load by constructing generic courses across departments and including and designing modules to fill theoretical gaps and deepen theoretical knowledge or to add attractive career paths • To define service/experiential/work based learning within the programme and incorporate electives/optional subjects to better prepare students for the work environment, extending practical training/internship periods and awarding credits for lab or field work experience; To offer more opportunities for community interaction, especially for graduate students. 46
1. Programme rationale (continued) • To outcomes communicate clearly to contextualize lectures, and information about administrative and support services to students, staff and stakeholders (advertising student assistantships effectively) • To identify and support students at risk in small groups and give academic and non-academic help and to communicate the importance of class attendance; also addressing low/increased student numbers • To sharpen recruitment and marketing efforts, improving outreaches where scholars, prospective and current students can be informed about the programme and its strategic value. (Open days, information sessions. . . ) 47
2. Academic integrity (65) • To increase reseach and benchmarking opportunities for staff with international scholars, to ensure the programme remains at forefront of new developments, and encouraging research/teaching interaction; • To review the undergraduate programmes annually, continuously and coherently planning modules in terms of content, level of difficulty and credit value; constantly reviewing the curriculum, learning materials, learning methods and programme outcomes; moderating it and ensuring that the feedback from external moderators is taken into account • To increase collaboration between lecturers to improve programme cohesion, filling in theoretical gaps through new and adapted modules, and cutting out duplication; also working with timetabling administrators to free up module combinations; allowing greater articulation with cognate programmes, structuring the programme to expose students to the core themes, balancing practice and theory to ensure the relevance of prescribed modules that are presented by other departments from both within, and external to, the faculty • To identify and remove obstacles inhibiting the use of experential learning and increase laboratory time, re-evaluating the module composition and structuring of the programme in order to make place for a longer period of internship; to evaluate the efficacy of the practical portions of the modules and investigate coherent year-long practical modules at second and third year levels; To communicate the reason for the approach followed during the first year of study, and maintain a challenging learning environment for students despite low/increased numbers • 48
2. Academic integrity (continued) • To develop writing, information and computer skills and improve the teaching of practical skills through the use of continuous assessment for all modules from second year upwards, and to identify and adapt modules fairly early in the academic programme; To focus on improving student mathematic skills; To complete computer literacy and scientific communication skills modules in the first year • To manage or establish new research institutes/units/centres such as an Institue for Food Technology (IFT) or Institute of Food Science and Technology (IFST) • To recruit students from historically disadvantaged backgrounds • To recruit more international students 49
3. Student recruitment, admission and selection (93) • To involve industry stakeholders in recruitment and funding for undergraduate scholarships; To acquire additional bursaries from all sector role players; To expand facilities and staff; To increase support staff, especially in terms of popular media and marketing; To involve subject societies in recruitment actions; To increase interaction with bursary providers about the criteria for successful students and the problems that are experienced by current bursary holders; To investigate the roles professional bodies play in promoting the image of certain careers • To review admission requirements for mathematics and science related to forest science programme; To evaluate the tempo at which graduates are employed; To investigate existing recruitment actions and material in terms of success and relevance; To monitor the extended degree programme to ascertain whether it is increasing student diversity; To monitor student numbers at the beginning of the year and consider the adjustment of entrance criteria if necessary; To review policy on ranking on application form so as to admit students who desire to follow; To review academic support available to struggling students in all years • To adapt intake in terms of quantity, quality and equity of students. To align recruitment efforts of the university, faculty, programme-committee and departments; To develop and implement mechanisms, including assessment methods, to widen access, (e. g. summer school, bridging programme); To increase the diversity of the student body in terms of South African cultural groups and international students; To create a structured marketing and recruitment plan as top priority, buying in expertise if needed, consulting with SU marketing divisions, utilizing existing marketing mechanisms, negotiating for undergraduate bursaries and addressing the student diversity; To improve the administrative implementation of admission criteria; To reconsider admission requirements at first year level to curb the high failure rate, as well as at honours level (possibly); To re-assess the modules which have become “service courses”; 50
3. Recruitment, admission and selection (continued) • To make the bridging programme compulsory for students with a matric mark between 50%-56%; To visit underprivileged schools in our immediate vicinity; To sponsor prizes (e. g. book prizes) for the best Biology (now Life Sciences? ) student in Grade 12 at a few selected schools; To consider an orientation period including visits to departments as well as the experimental farm before registration of second-year students to allow informed choices on major subjects; To supply information on programmes at the Expo for Young Scientists and Olympiad candidates, as well as high school science teachers; To encourage third years to attend final years’ product development presentations; To focus on the recruitment of coloured students due to the demand from Agri businesses; • To initiate discussions with the govrnment on fragmentation of higher forestry education in South Africa; To broaden the communication base with students at first- and second year level; To develop a culture of uniqueness and excellence amongst staff members and students; To raise awareness of language issues in the academic programme • To implement an extended degree programme (and first year academy) to benefit students that have to overcome academic inadequacies due to historical barriers, and help students to overcome cultural and language difficulties. To prevent over-subscription to the course; Cap student numbers (no more laboratory space available); . ; To identify the optimum number of students that can be accommodated in the programme (estimated at between 40 and 50 students), keeping economic factors in mind; To market a BSc(Hons) as a career route – both commercially and in parastatal, particularly academic spheres; To pay careful attention to entrance requirements in the next few years as the new school leaving certificate is implemented. Aim to raise entrance requirements; To critically review failures and examine the reasons for these 51
3. Recruitment, admission and selection (continued) • To make better use of recruitment opportunities (e. g. Open Day, visits to schools); To recruit black students via industry contacts; To send promotional material (pamphlets) to all Secondary schools in the Western Cape; To represent the department at open days organized by IPS; To write an article promoting Conservation Ecology and Entomology in general and our department in particular for the popular science magazine Quest; To ensure the web-site “inspires” students; To continue with current promotions activities and transparent admission policies; To have greater flexibility within the language policy; To provide admission requirements into the program (especially as of 2009) on departmental and faculty web pages and brochures. ; To advertise the programme campus wide on notice board, emphasizing the financial support through industry bursaries, especially to students from disadvantaged backgrounds. ; To promote the need for a Faculty Open Day with a smaller, but more specific learner group (eg. top 10 learners within a grade with mathematics as school subject or learners from strong feeder schools) so that departments can participate more effectively. Our perception is that the University Open Day fails to recruit quality motivated students for the program. To improve the distribution of marketing material; To recruit talented learners with bursaries; To contact CPS to disseminate the information on courses in the programme, job opportunities and available bursaries; To give more students access to bridging programmes; To try to attract the best Afrikaans and English students; To attract black students with the Fundza Lushaka bursary scheme 52
4. Staffing (62) • To enhance continuous professional development; To make sure all newly appointed lecturers attend the University’s induction and training courses; To ensure on-going professional development, letting staff attend various courses (e. g Web-CT training) as the need arises. To deliver graduates who are in demand in the food and cognate industries; To develop C 2 and C 3 staff new skills and knowledge; To enable personnel to develop academically in their fields of specialization; To encourage research activity by giving all staff members an “off semester” with no undergraduate teaching commitments and additional training in assessment methods for large classes; To maintain good research profiles and to expose students and lecturers to current research and teaching issues and trends; To quantify time usage and check against workload document to enforce contract hours; To manage time more effectively by setting fixed consultation times • To beter utilize the annual performance evaluation process , Permanent staff members to be required to compile a personal work agreement and performance contract. This assists in quality assurance of teaching and learning, research and service; To have all teaching staff, with one exception, have a Ph. D; To review teaching load of staff through programme evaluation; To have positive departmental evaluations; To ensure an equitable distribution of the teaching load by a yearly review; To review and evaluate the deployment of technical staff for optimum technical support • To ensure more realistic work loads for academic staff ; To align specialisation and research focus of staff needs with programme. To ensure greater cooperation and less duplication 53
4. Staffing (continued) • To utilize personnel seconded from industry to the departments can assist in maintaining research capacity; To investigate mechanisms by which specialised technical / practical knowledge can be imparted to students. This may be linked to a specific academic staff member, specific tasks during the internship, and/or exposure during practicals; To ensure that student assistants are well -trained To better communicate the benefits of an excellent diversity profile and the spin-offs for the student body, who are exposed to a new experiences, approaches • and ideas. To inform the faculty through the departmental evaluations that a critical mass of academic staff and sufficient administrative support are not present within all departments in the program to effectively perform and excel in all 3 areas expected namely teaching, research and community service; To actively encourage students and lecturing staff to interact about the expectations of web presence of modules, particularly in smaller senior modules • To maintain quality despite the increase in student numbers; To manage part-time staff carefully; To have adequate facilities and personnel to allow students to do practical work and experiments • To address the perceived insufficient levels of remuneration to attract and retain young academics, especially for young black staff. The current personnel evaluation has shortcomings and financial incentives linked to excellence are insufficient to motivate academic staff. 54
5. Learning facilitation (70) • To get more clarity on the meaning of “student centered teaching” and its implications; To encourage participation by academic staff in the activities of Centre for Teaching and Learning; To utilize Web. CT to effectively communicate with large groups, but not replace lecturer-student interaction or class notes; To evaluate and compare learning material with international universities; To develop a policy on student centered teaching so that independent, enthusiastic and spontaneous learning consistently takes place; To incorporate fundamental knowledge much more overtly • To act on students’ feedback and evaluation ; To give a copy of each module evaluation to the programme coordinator (improve efficiency of the feedback); To request lecturers to complete module frameworks, literature bibliographies, goals, outcomes in accordance with the module framework requirements of Senate; To review the relevance of all units; To revisit the PBL process in terms of life long learning, critical thinking and professional reasoning; To review what is lectured, the links between problems and lectures and workshops and feedback; To review problems and evaluate their incremental demands over the four years; To review decision to make curriculum books only available to students in eformat • To align all study guides with the University’s policy requirements for study guides (specifically the inclusion of module-level outcomes in study guides); Investigate ways to make class experience more stimulating; To make more use of text books and journal publications, and less class notes; Powerpoint slides always to be provided to students; To employ a variety of assessment opportunities to evaluate the facilitation of learning; To use text books and scientific publications to a greater extent and in preference to class notes; To review the modules to ensure that they contain learning opportunities for the development of these skills, without unnecessary duplication; To relook at the orientation course in first year (preparation for PBL curriculum) and add a test to ensure the content is mastered and internalized 55
5. Learning facilitation (continued) • To investigate the option of providing credits for lab or field work experience • To adequately communicate the module outcomes to the students yearly by the chairperson and via the website; To advertise student assistantships & opportunities effectively; To expose second year students to the layout and cohesion of the programme; To involve students in more personal feedback (similar to this evaluation exercise) at a module and programme level; To highlight the fact that ability to work in a team is part of the supposed programme outcomes and establish where this is in fact done; • To assign tutors dedicated to the support of disadvantaged students; To inform departments of the need to apply computer literacy skills and gradually introduce sourcing and use of scientific literature earlier in the program (3 rd-year level). Basically, do not leave all tasks, seminars and research projects for the 4 th year; To discuss with computer literacy convenors options to allow mathematical science students to do fewer but more relevant modules within computer literacy; To investigate the possibility of introducing opportunities for students to improve and perfect their written and verbal communication skills at early stages in their studies; To develop oral presentation skills for senior students 56
6. Assessment (58) • To ensure that all staff members are aware of departmental policies and requirements for assessment and the publication and storage of marks; To use peer-reviewing within student/study groups as an effective supplementary method of assessment; To encourage continues assessment training of staff; To change fieldwork rubrics to be more user-friendly and precise (with student input); To review the number of assessment activities that contribute to the marks and activities; To review time needed to answer questions and consider the advisability of seen vs unseen questions; To analyze all exam questions according to Bloom’s taxonomy; To align assessment practices all course frameworks, study guides and assessment opportunities with the university’s requirements ; • To ensure that all tests and exams are aligned with the principles of the University’s Assessment Policy; To investigate “Turn-It-In” for electronic submission of assignments; To investigate ways to reduce the significant increase in administrative load associated with internal and external moderation; To have rigorous internal moderation, and external moderation for third year and honours modules; To establish, manage and oversee an accurate and reliable administrative system to record marks onto the SIS. To enhance strategies to eradicate plagiarism ; To make assessment challenging, in particular to test problem-solving abilities; To give more smaller tests rather than only a few major tests and an exam; To assess individuals fairly in a group assignment and CTL should be approached for assistance in this regard; To pursue a more systematic manner for collecting student feedback on each module; To improve module frameworks to include assessments details (dates, type of assessment as well as expected timeframe for feedback); To use a range of assessment methods such as a seminar, laboratory, written and oral examinations; • 57
6. Assessment (continued) • To clearly communicate the means by which problem-solving abilities will be assessed, i. e. the quality of the questions to be expected, the amount of insight that will be required; To update the assessment dates and weights on the website; To keep lecturing staff (and yearbooks) updated with regulations regarding assessment and moderation (internal and external) at departmental level; To communicate the different assessment methods of different modules clearly to the students; To provide reasons or motivations for giving a particular mark, especially for essay-type projects and similar essay-type exam questions • To conduct individual interviews with students scoring >30% in semester test, determine reasons, plan for support; To devise an early warning system for students who are struggling (more difficult with larger classes); To give attention to entrance requirements and streaming of students to preserve standards; To handle question papers with care to avoid corruption of the assessment process; • To monitor mechanisms. 58 individual student progress in terms of the First Year Academy’s
7. Infrastructure and academic information sources (55) • To keep teaching facilities to up to date; To improve, maintain and replace laboratory equipment; To challenge the insufficient budgets to support the running cost of undergraduate practical training in some courses (especially those with high student numbers); To expand laboratory space, personnel; To develop IT support services to personnel • To utilize excellent library services optimally , faculty librarian specialists and proper student training and regular purchases of new DVDs and material; To investigate the possibility of using NARGA for training and tests • To re-allocate facilities to departments so that lecture theatres, laboratories and the cellar can be in the same building; To arrange for continued access to academic & scientific information (library) once graduates are working in rural areas; To maintain the effective academic support offered by the Library, Sun. Media and Information Technology. To monitor access to and choice of electronic journals To identify one or more modules to be enriched in teaching of writing skills and use of library facilities and undergraduate students should be required to submit a literature review at some point in the academic programme to ensure that the desire to continue reading and learning beyond graduation is fostered. 59
7. Infrastructure and academic information sources (cont) • To improve the equipment for undergraduate practicals so that each student can complete practical assignments individually; To update the computer facilities and associated software; To optimize laboratory equipment for teaching purposes (more student-proof research equipment) • To optimally utilize the class space, modern equipment, laboratory facilities, and computer and library services • To maintain laboratory equipment; To monitor the lecture hall size : amount of students • To attend to the insufficient field laboratories; To prevent overshooting capacity of a maximum of 40 students in the programme; To challenge the tariffs quoted by the General Maintenance Section of the University for services are perceived to be inflated due to a sub-contracting system and is not seen to be consistently offering value and high quality service. 60
8. Programme coordination (61) • To integrate programme inputs from different stakeholders: into curriculum development; undergraduate, post-graduate, extraordinary appointments, alumni and industry; • To align teaching methods with international best practrice ; To state programme outcomes (generic and specific) for all lecturers and students involved in the programme through updated module frameworks To monitor/review service levels regularly on all levels for relevancy within the programme; To evaluate and implement final year • feedback; To monitor throughput of modules and support modules; To monitor student feedback earlier in the modules to report back; To consider student representation in the Programme Committee To institute ongoing actions to improve and evaluate the efficiency and quality of the teaching programme; To align individual • courses and course materials towards the main focus of the programme, especially for new members of staff; To improve communication between departments to avoid unnecessary overlap between courses and course material; 61
8. Programme coordination (contunued) • To ensure better communication between lecturers in different departments about the contents of programme; To introduce programme coordinator to 1 st year students & provide contact details; To ensure regular communication between teaching administration at the University, and teaching staff and students; To introduce the programme coordinator to first and second year students, and provide contact details in class and on the internet; To ensure that module frameworks exist for all modules, also on the website; To hold more frequent committee meetings and to create more regular opportunities for student input and participation; To explain lines of communication and authority to students To identify and support at risk students by tracking their records and analyzing it to help the department; To explore how the information in the • Student Oracle system can be utilized; To ensure that deadlines are met; • To gain better understanding of the reasons why certain students do not study successfully 62
9. Student success and academic support (60) • To provide more opportunities for individual face to face contact between lecturers and students; To develop presentation skills for lecturers and technical staff; To engages with CHSE to ensure academic staff have adequate educational background and facilitation and assessment skills; To give both new and experienced staff the opportunity to attend CTLD assessment and teaching courses; To adhere to University Teaching Assessment policy; To maintain classes on/below 40 students per class To improve the monitoring system for 1 st and 2 nd year students, To monitor student success more systematically; To review mentorship strategy • with a view of developing a formal approach, close contact student / lecturer; To monitor continuously that the pressure to improve the pass rate of students, especially at first year level, is not carried over on to the second- and third year level to finally compromise the integrity of the whole program; To monitor the success rate in every module in the department (and faculty) where it resides. To adapt evaluation techniques to maintain current standards; To make the existing mentoring program more accessible to students in the • programme. To evaluate and adjust the mentoring system in order to make it more effective and to ensure continuity from 1 st year onwards. To look afresh at entrance requirements and options for students who struggle To investigate the possibility of continuing the mentor system for support of students at second and third year levels, in addition to the current first year support. 63
9. Student success and academic support (continued) • Improve support to part time/sessional staff in their teaching role; To teach students to improve skills rather than just to supervise and critique them • To assist students to overcome the challenge posed by language. T-options for courses could be made more readily available; To remind lecturing staff that all assessments, including small tutorial tests, should be bilingual; To encourage language use accessible to all students To alert students to the available student support services ; To scrutinise students’study records in all their modules; To improve language, numerical and cognitive skills within the programme; To let students participate in the first year Learning Academy; To assist students with the challenges of the first-year of the program. The Extended degree program (also First Year Academy) is seen as an option to improve student success; To make better use of tutor demonstrators. To extend the successful tutor system to other modules if financial support can be obtained from the Faculty; To identify at risk students earlier and offer compulsory extra tutorial time; To identify good role models for students • • To provide much broader access to an “extended degree”, especially within the mathematical sciences. Come to terms with the fact that a 4 -year degree is the norm and use this to plan curricula. Consider a universal 4 -year programme with some students emerging with an honours; others with a BSc. 64
10. Service learning and work-based learning (43) • To cooperate with stakeholders to create opportunities for WBL interaction with industry for all students; To update lists of industry knowledge partners; To explore collaboration efforts with Mo. Us to clarify roles; To ensure that the commercial entity is suitable for receiving undergraduate students and that there is no family connection with the student To review and improve the WBL coordination responsibility guidelines and monitor the success and/or need for WBL through departmental evaluations; • To follow up past-graduates to see how feasible their recommendations are • To adapt programmes to integrate WBL modules ; To use experimental farms optimally; To lengthen period of (vacation) internships; To extend mentoring system to alumni in the industry • To • searches during internships; To formalize community interaction projects with SUSPI; To consider the possibility of site visits and voluntary internships ensure the commercial entity is fully aware of its WBL obligations to the student before placement (e. g. internship); To require literature 65
10. Service learning and work-based learning (cont) • To communicate to all parties the outcomes of the WBL and what the student should be achieving, To ask all module coordinators to discuss work opportunities in the context of their modules; • To monitor WBL risks continuously e. g. to provide mentoring, support and networking options for supervisors (employers); To overcome logistic difficulties of WBL in certain industries, e. g. the mining and petroleum industries in the Western Cape; encourage students to take initiative for finding WBL placement • To address any WBL misconceptions in the industry regarding • To the abilities of graduates and the balance of theoretical, practical and experiential teaching and training during the academic programme. To challenge lecturing staff to guard against modules being too theoretical given the context in which they are taught. To do an impact study to measure the effectiveness of the programme in preaparing community servers for practice 66
11. Programme evalution and coordination (44) • To maintain regular interaction with sector stakeholders and survey employers on the impact of the programme and levels of satisfaction; To improve liaison with industry / employers; To improve industry’s understanding of the purposes and outcomes of the programme; To review the mechanism used to give stakeholders information about curriculum changes based on their input To implement coordinated strategies to review programme content and outcomes annually; To obtain data on student recruitment generated during • departmental evaluations; To ensure that all departments obtain feedback from industries; To send evaluation forms to alumni one year after graduation; To evaluate all quality assurance practices used in the department To adapt courses to eliminate redundancy and streamline the learning process; To evaluate the relevance and efficacy of all modules, including the practical portions thereof To initiate a process of communication and interaction with departments at this university and at other institutions to ensure that we are offering modules of the • • highest standards and relevance; also consider the employability of exit level students; To encourage final year students to continue with post-graduate studies to better their qualifications and improve their employability and develop a society for past • graduates 67
Reflections and conclusions • Selectivity justified due to 80/20 approach? • “The 20% doctor”: all students must achieve all outcomes • Each programme must meet all the criteria/standards • Sampling and focus in evaluations • How to select evidence? • Just right and just enough • Superficial? 68
Reflections and conclusions • Strategies at different levels (Programme, Department, Faculty, Institution, System, and beyond) • Encourage creativity • Encourage debate, discussion across silo’s • Closing the loop remains one of the biggest challenges • Good systems, procedures • Good people (at all levels) 69
Скачать презентацию Conducting programme evaluations reviews with a view to sustainable
5864df225655e84b5da476be9842d322.ppt