On Layers and Objects in Assessment Design Robert

On Layers and Objects in Assessment Design Robert Mislevy, University of Maryland Michelle Riconscente, University of Maryland

Evidence-Centered Design § Conceptual design framework § Broad applicability § Based on § Evidentiary reasoning principles § Demands of assessment production & delivery § Assures § Validity of assessment § Coordination among collaborators 2 (Mislevy, Almond, & Lukas, 2003)

Defining Assessment § Reasoning from the particular things students do, say, or make… § to inferences about their knowledge, skills & abilities. 3

Some Fundamentals § Evidentiary Reasoning § Assessment as Argument 4

Evidentiary Reasoning C unless on W account of B since so D A supports R 5 Toulmin (1958)

Example I C unless on W account of B since so D A supports R 6 Toulmin (1958)

Example II John caused the car accident and Marg was seriously injured. C W 1: Since an intoxicated driver is generally presumed to be at fault in an accident…. W 2: Since the impact of a 2000 lb. auto moving at 50 mph on a human will generally cause serious injury… on W account of B since B 1: State Code: drunk driving at. 10 BAC and common law doctrine of negligence per se. B 2: Emergency medical records at Wishard Hospital indicate this type of collision will result in serious condition. unless so D A Marg was also intoxicated… supports R • John’s BAC was. 13. • Marg was unconscious and bleeding. • John’s speech was slurred. • Marg had the right of way. 7 Toulmin (1958)

Assessment as Argument Inferences C W since D Observations § Inferences § Observations needed to ground them § Situations that will evoke them § Chain of reasoning connecting them Situations 8

Some Fundamentals § Evidentiary Reasoning § Assessment as Argument § Knowledge Representations (KRs) § Common Language 9

Knowledge Representations § Directions from Campus to the White House § http: //www. mapquest. com/directions/main. adp? go=1&do=nw&un=m& 2 tabval=address&cl=EN&ct=NA&1 tabval=address&1 y=US&1 a=&1 c=& 1 s=&1 z=&1 ah=NXXzj. Lrv%252 bg. Ge. W 7 Je. Q 4 f. Dr. UNya%252 b 4 Dt. Ksjxh e. MTb. VTh. QJ 3 l. E 9 Wr. Nx 7 DAJWKfg. ZBsx. DNU 8 KMBJ 3 MT 8 ws. LFk 85 Pi 6 j LRq. QHx. BFYMlklq 5 k 82 h. IODOVi. Bi. LYDx. Dz 3 sk%252 b. Xx. Gl. Uc. WB 2 Xe c 5 x 21 s. Agg. O 4 x. XFyg%253 d&2 y=US&2 a=1600+Pennsylvania+Av enue&2 c=Washington&2 s=DC&2 z=&2 ah=&idx=0&id=417 d 754100047 -01632 -400 c 2551&aid=417 d 7541 -00048 -01632 -400 c 2551 § OR ? 10

Some Fundamentals § Evidentiary Reasoning § Assessment as Argument § Knowledge Representations (KRs) § Common Language § Layers § Leveraging Varied Expertise § Common Structures 11

A Layered Approach § Leveraging Varied Expertise § e. g. , Housing Developments § Common Structures § e. g. , Architecture, Software Design § Iterate through Assessment Argument § narrative technical specifications Inferences Observations Situations 12

ECD Layers Domain Analysis Domain Modeling Conceptual Assessment Framework Assessment Delivery • What is important about this domain? • What tasks are central to proficiency • Represent key aspects in this domain? of the domain in • What KRsassessment terms of are central to. Whatdomain? this tools • argument. and materials do we need to implement this kind of assessment? • How do we move to implementation with actual examinees? 13

Domain Analysis § Valued work § Task features § Representational forms § Performance outcomes § Valued knowledge § Knowledge structure and relationships § Knowledge-task relationships 14

ECD Layers Domain Analysis Domain Modeling Conceptual Assessment Framework Assessment Delivery 15

Domain Modeling § Express the content of the domain analysis in terms of the assessment argument. § Takes narrative form Domain Modeling Inferences Observations Situations § PADI KR: Design Patterns 16

Design Patterns § Identify in narrative form: § Knowledge, skills, and abilities (KSAs) § Observations to support inference § Features of task situations that elicit target KSAs § Related content or inquiry standards § Do not provide a concrete design or implementation of an assessment task 17

18

ECD Layers Domain Analysis Domain Modeling Conceptual Assessment Framework Assessment Delivery 19

Conceptual Assessment Framework § Move from narrative to more technical specifications. § Express assessment argument in terms closer to implementation ECD Models Conceptual Assessment Framework 20

From Argument to Models STUDENT Inferences MODEL EVIDENCE Observations MODEL TASK Situations MODEL 21 (Mislevy, Almond, & Lukas, 2003)

From Argument to Models STUDENT MODEL What are we measuring? EVIDENCE MODEL How do we measure it? • Quality • Relevance TASK MODEL Where do we measure it? 22 (Mislevy, Almond, & Lukas, 2003)

Conceptual Assessment Framework § Move from narrative to more technical specifications. § Express assessment argument in terms closer to implementation ECD Models § Create pre-blueprints from which we will eventually generate blueprints and operational assessments. § PADI KR: Task Templates 25

Task Templates § Support the specification of technical details § Link CAF components § Serve as pre-blueprints: abstractions of multiple assessment tasks § Become task specifications when all template components are specified 26

27

28

Summing Up § ECD: assessment design as development of an assessment argument § Coordinate work across expertise § Use of layers § Knowledge Representations § Illustrated with PADI work § In addition: § Explication of reasoning behind assessment design decisions § Identification of reusable elements and pieces of infrastructure 29

An ECD “Flight Check” § Conceptual design framework § Broad applicability § Based on § Evidentiary reasoning principles § Demands of assessment production & delivery § Assures § Validity of assessment § Coordination among collaborators 30 (Mislevy, Almond, & Lukas, 2003)