project e-scape evidence-based web-portfolios - to empower learning - for reliable assessment Richard Kimbell Karim Derrick in association with …. TERU: Goldsmiths University of London TAG developments
e-scape system enables us to support and capture live performance • performance assessment • on task • in real time science to display the performance in web-portfolios • easily distributed • URL / log in / password geography to assess performance • for GCSE / Diploma • linked to Awarding Body systems • with very high reliability (0. 95) design
capturing performance evidence data and notes mind-maps audio reflections video presentation sketching (and collaborating) photos of the state of play
continuum of control • tight control of tasks • strict timings • controlled response modes • standardised portfolio formal examinations • learner autonomy of tasks • flexible timings • learner choice of tools • customised portfolio extended coursework projects client/user can choose the degree of control
the portfolio as a ‘big-picture’ of the activity … that can be dipped into for the detail
the assessment challenge
teachers / examiners review portfolios with ‘RED PEN’ different markers have separate ‘layers’ teacher speaks directly to the student teachers notes on student portfolio criteria displayed with the work and evidence Is noted
but there is another (quite different) possibility L. Thurstone (1927) • law of comparative judgement • judges’ personal standards cancel out A. Pollitt (2004) • comparative pairs methodology • for reliability studies between exam boards e-scape (2007 & 2009) • the 1 st and 2 nd complete data sets • the ‘pairs engine’ • for front line assessment
eye testing
teachers and assessment teachers are good at • relative judgements • ranking because their judging standard cancels out teachers are not good at • absolute judgements • scoring because ‘criteria’ become personalised
• teachers discuss criteria • teachers use them as ‘signposts’ to look into the portfolios • teachers identify strengths and weaknesses of portfolios …but then they make a relative holistic judgement
the ‘pairs engine’ manages the assessment process portfolio A portfolio B
the whole process is run on-line in the ‘pairs engine’ • judges in Ireland, Israel and across the UK • shared viewing of exemplar work • on-line marker training
the pairs engine dynamically generates the rank what emerges is the collective professional consensus of an expert group of judges
2009 sample 350 pupils awarding grades is a separate process multiple judges (28) multiple comparisons (17+) } reliability coefficient = 0. 95
“ Verbal and Mathematical Reasoning tests. . were developed for 11+ and similar purposes from about 1926 to 1976. These were designed to achieve very high reliability, by minimising inter-marker variability and maximising the homogeneity of the items that made them up. KR 20 statistics for those were always between 0. 94 and 0. 96. With e-scape, a level of internal consistency comparable to those reasoning tests has been achieved without reducing the test to a series of objective items. “ Alastair Pollitt: Dec 2008
identifying problems the engine identifies portfolios where judges disagree and individual judge profiles show their consensuality or ‘misfit’
reducing standard error • the first 10 judgements do most of the work • the second 10 fix the scale • thereafter the gain in accuracy per extra judgement is small
how long do judgements take? median: • for all 28 judges • 130 judgements each = 4 mins 6 secs = total of 8 hrs 45 mins
Judges’ reflections …. . the learner voice emerges much, much faster. . and less scary Portfolios displayed in this way have a huge advantage in that the big picture can be seen immediately. It’s very easy to get a feel for the project. . . The ability to dip in and out of the different sections enabled me to reinforce my holistic mental picture of the project. • would encourage learners’ understanding of capability • would encourage learning through critique • could support e. g. PLTs evidence The variety of media used within the portfolio seems to give everyone a fair chance to display their design thinking regardless of their ability. It was fairly easy to see that there was a range of performance evident in the portfolios. Surprisingly pupils’ attitudes were also often evident. It’s brilliant not having to carry around boxes full of portfolios/folders and practical work.
current implementations
new applications for e-scape & ‘pairs’ Edexcel Examination Board: UK 14 -19 Diploma in ‘Creative & Media’ West Australia Curriculum Council school leaving certificate examinations (engineering / applied IT / phy ed / LOTE ) Limerick University: Ireland engineering design project (mobiles) + assessment UK government agency writing assessments at age 11 Scotland SQA / LTS formative assessment in transition years (ege 11 -13) Bath Spa University & Astra. Zeneca Science Teaching Trust science/technology for age 11 -13
e-scape references and citations by authors outside the e-scape team
project e-scape e-solutions for creative assessment in portfolio environments Contact me at: r. kimbell@gold. ac. uk Goldsmiths University of London www. gold. ac. uk/teru
Скачать презентацию project e-scape evidence-based web-portfolios — to empower learning
f520d8efc0dca4e764a9f04f0eb57b0a.ppt