Design Patterns: Supporting Task Design by Scaffolding the Assessment Argument Robert J. Mislevy University of Maryland Geneva Haertel & Britte Haugan Cheng SRI International DR K-12 grant #0733172, “Application of Evidence-Centered Design to State Large-Scale Science Assessment. ” NSF Discovery Research K-12 PI meeting, November 10, Washington D. C. This material is based upon work supported by the National Science Foundation under Grant No. DRL- 0733172. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
Overview § Design patterns § Background § Evidence-Centered Design § Main idea § Layers § Assessment Arguments § Attributes of Design Patterns § How they inform task design
Design Patterns § Design Patterns in Architecture § Design Patterns in Software Engineering § Polti’s Thirty-Six Dramatic Situations
Messick’s Guiding Questions § § § What complex of knowledge, skills, or other attributes should be assessed? What behaviors or performances should reveal those constructs? What tasks or situations should elicit those behaviors? Messick, S. (1994). The interplay of evidence and consequences in the validation of performance assessments. Educational Researcher, 23(2), 13 -23.
Evidence-Centered Assessment Design § § § Organizing formally around Messick quote Principled framework for designing, producing, and delivering assessments Conceptual model, object model, design tools Connections among design, inference, and processes to create and deliver assessments. Particularly useful for new / complex assessments. Useful to think in terms of layers
Domain Analysis What is important about this domain? What work and situations are central in this domain? What KRs are central to this domain? Domain Modeling How do we represent key aspects of the domain in terms of assessment argument. Conceptualization. Conceptual Assessment Framework Design structures: Student, evidence, and task models. Generativity. Assessment Implementation Manufacturing “nuts & bolts”: authoring tasks, automated scoring details, statistical models. Reusability. Assessment Delivery § Students interact with tasks, performances evaluated, feedback created. Four-process delivery architecture. Layers in the assessment enterprise From Mislevy & Riconscente, in press
Domain Analysis What is important about this domain? What work and situations are central in this domain? What KRs are central to this domain? Domain Modeling How do we represent key aspects of the domain in terms of assessment argument. Conceptualization. Conceptual Assessment Framework Design structures: Student, evidence, and task models. Generativity. Assessment Implementation Manufacturing “nuts & bolts”: authoring tasks, automated scoring details, statistical models. Reusability. Assessment Delivery § From Mislevy & Riconscente, in press Students interact with tasks, performances evaluated, feedback created. Four-process delivery architecture.
Domain Analysis What is important about this domain? What work and situations are central in this domain? What KRs are central to this domain? Domain Modeling How do we represent key aspects of the domain in terms of assessment argument. Conceptualization. Conceptual Assessment Framework Assessment Implementation Assessment Delivery § From Mislevy & Riconscente, in press Design structures: Student, evidence, and task models. Generativity. Manufacturing “nuts & bolts”: authoring • Assessment argument tasks, automated scoring details, • Design Patterns statistical models. Reusability. Students interact with tasks, performances evaluated, feedback created. Four-process delivery architecture. structures
Domain Analysis What is important about this domain? What work and situations are central in this domain? What KRs are central to this domain? Domain Modeling How do we represent key aspects of the domain in terms of assessment argument. Conceptualization. Conceptual Assessment Framework Design structures: Student, evidence, and task models. Generativity. Assessment Implementation Manufacturing “nuts & bolts”: authoring tasks, automated scoring details, statistical models. Reusability. Assessment Delivery § From Mislevy & Riconscente, in press • Psychometric models • interact with tasks, Students Automated scoring performances evaluated, • Task templates feedback created. Four-process • Object models delivery architecture. • Simulation environments
Domain Analysis What is important about this domain? What work and situations are central in this domain? What KRs are central to this domain? Domain Modeling How do we represent key aspects of the domain in • Authoring interfaces terms of assessment argument. Conceptualization. • Simulation environments • Re-usable platforms & elements Conceptual Assessment Framework Design structures: Student, evidence, and task models. Generativity. Assessment Implementation Manufacturing “nuts & bolts”: authoring tasks, automated scoring details, statistical models. Reusability. Assessment Delivery § From Mislevy & Riconscente, in press Students interact with tasks, performances evaluated, feedback created. Four-process delivery architecture.
Domain Analysis What is important about this domain? What work and situations are central in this domain? What KRs are central to this domain? Domain Modeling How do we represent key aspects of the domain in terms of assessment argument. Conceptualization. Conceptual Assessment Framework Assessment Implementation Assessment Delivery § From Mislevy & Riconscente, in press • Interoperable elements Design structures: Student, evidence, and • Generativity. task models. IMS/QTI, SCORM • Feedback / instruction / reporting Manufacturing “nuts & bolts”: authoring tasks, automated scoring details, statistical models. Reusability. Students interact with tasks, performances evaluated, feedback created. Four-process delivery architecture.
Toulmin’s Argument Structure Claim unless Alternative Warrant explanation since so Backing Data
Assessment Argument Structure Claim about student unless Warrant for assessment argument since so Data concerning performance Alternative explanations
Assessment Argument Structure Claim about student unless Warrant for assessment argument since so Data concerning situation Data concerning performance Alternative explanations
Assessment Argument Structure Claim about student unless Warrant for assessment argument since so Data concerning situation Warrant for task design since Data concerning performance Warrant for since scoring Student acting in assessment situation Alternative explanations
Assessment Argument Structure Claim about student unless Warrant for assessment argument since so Data concerning situation Warrant for task design since Data concerning performance Warrant for since scoring Student acting in assessment situation Alternative explanations e. g. , near or far transfer, familiarity with tools, assessment format, representational forms, evaluation standards, task content & context. Not in measurement models, crucial to inference. Other but information concerning student vis assessment situation
PADI Design Patterns § Structured around assessment arguments § Substance based on recurring principles, ways of thinking, inquiry, etc. § E. g. , NSES on inquiry, unifying themes § Science ed. & cog psych research
Some PADI Design Patterns § Model-Based Reasoning § Model Formation; Evaluation; Revision; Use § Model-Based Inquiry § § Design under Constraints Generate Scientific Explanations Troubleshooting (with Cisco) Assessing Epistemic Frames (in progress; with David Williamson Shaffer)
The Structure of Assessment Design Patterns ATTRIBUTE DESCRIPTION Focal Knowledge, Skills, Abilities The primary knowledge / skills / abilities (KSAs) targeted by this design pattern. Rationale How/why this DP addresses evidence about focal KSAs Additional KSAs Other knowledge/skills/abilities that may be required by tasks. Characteristic features of tasks Aspects of assessment situations that are needed to evoke evidence about the focal KSAs. Variable features of tasks Aspects of assessment situations that can be varied to shift difficulty or focus. Potential work products What students actually say, do, or make, to produce evidence. Potential observations Aspects of work products we might identify and evaluate, as evidence about students’ KSAs. Potential rubrics Ways of evaluating work products to produce values of observations.
How Design Patterns Support Thinking about the Assessment Argument ATTRIBUTE Focal Knowledge, Skills, Abilities unless Warrant for assessment argument Warrant for task design since How/why this DP addresses evidence about focal KSAs Other knowledge/skills/abilities that may be required by tasks. Aspects of assessment situations that are needed to evoke evidence about the focal KSAs. since so Data concerning situation Rationale Characteristic features of tasks Alternative explanations The primary knowledge / skills / abilities (KSAs) targeted by this design pattern. Additional KSAs Claim about student DESCRIPTION Data concerning performance Warrant for since scoring Student acting in assessment situation Other information concerning student vis assessment situation Variable features of tasks Potential work products Aspects of assessment situations that can be varied to shift difficulty or focus. What students actually say, do, or make, to produce evidence. Potential observations Aspects of work products we might identify and evaluate, as evidence about students’ KSAs. Potential rubrics Ways of evaluating work products to produce values of observations.
How Design Patterns Support Thinking about the Assessment Argument ATTRIBUTE Focal Knowledge, Skills, Abilities Claim about student unless Warrant for assessment argument since so Warrant for task design Alternative explanations DESCRIPTION The primary knowledge / skills / abilities (KSAs) targeted by this design pattern. Rationale How/why this DP addresses evidence about focal KSAs Additional Other knowledge/skills/abilities that may is KSAs organized be required by tasks. The design pattern Characteristic be Data around Focal KSAs. They will Aspects of assessment situations that concerning features of are needed to evoke evidence about situation performance involved in the Claim, although there KSAs. tasks the focal Variable Aspects may be other. Other KSAs that are included of assessment situations that features Warrant for in the targetinformation of inferencetasks of Model varied to shift difficulty or focus. (e. g. , can be concerning since scoring student vis assessment Formation—but what models, what students actually say, do, or Potential work What situation products make, to produce evidence. context? ). Student acting in assessment situation Potential Aspects of work products we might observations identify and Associated with evaluate, as evidence about students’ KSAs. Characteristic Features of Potential work products Tasks. Ways of evaluatingobservations. to rubrics produce values of
How Design Patterns Support Thinking about the Assessment Argument ATTRIBUTE Focal Knowledge, Skills, Abilities unless Warrant for assessment argument Warrant for task design since Other knowledge/skills/abilities that may be required by tasks. Characteristic features of tasks Aspects of assessment situations that are needed to evoke evidence about the focal KSAs. Potential rubrics Ways of evaluating work products to produce values of observations. since so Data concerning situation How/why this DP addresses evidence about focal KSAs Additional KSAs Alternative explanations The primary knowledge / skills / abilities (KSAs) targeted by this design pattern. Rationale Claim about student DESCRIPTION Data concerning performance Warrant for since scoring Student acting in assessment situation The Rationale provides background that Variable Aspects of assessment situations features of can be Focal KSAs, into the nature of thevaried to shift difficulty or focus. tasks and. Potential workof What students actually say, do, or the kinds things that people do in products what kinds of situations that make, to produce evidence. Potential Aspects of work to the evidence it. It contributes products we might observations identify and evaluate, as evidence Warrant in the assessment argument. about students’ KSAs. Other information concerning student vis assessment situation
How Design Patterns Support Thinking about the Assessment Argument ATTRIBUTE Focal Knowledge, Skills, Abilities unless Warrant for assessment argument Warrant for task design since How/why this DP addresses evidence about focal KSAs Other knowledge/skills/abilities that may be required by tasks. Aspects of assessment situations that are needed to evoke evidence about the focal KSAs. since so Data concerning situation Rationale Characteristic features of tasks Alternative explanations The primary knowledge / skills / abilities (KSAs) targeted by this design pattern. Additional KSAs Claim about student DESCRIPTION Data concerning performance Warrant for since scoring Student acting in assessment situation Other information concerning student vis assessment situation Variable Aspects of assessment situations that Additional be varied to shift difficulty or focus. features of can KSAs play multiple tasks roles. You need to think about Potential work What students actually say, do, or which ones you really DO want products make, to produce evidence. to include as targets products we might Potential Aspects of work of inference observations identify and evaluate, as evidence (validity) and which ones you about students’ KSAs. really DON’T (invalidity). Potential rubrics Ways of evaluating work products to produce values of observations.
How Design Patterns Support Thinking about the Assessment Argument ATTRIBUTE Focal Knowledge, Skills, Abilities unless Warrant for assessment argument Warrant for task design since How/why this DP addresses evidence about focal KSAs Other knowledge/skills/abilities that may be required by tasks. Aspects of assessment situations that are needed to evoke evidence about the focal KSAs. since so Data concerning situation Rationale Characteristic features of tasks Alternative explanations The primary knowledge / skills / abilities (KSAs) targeted by this design pattern. Additional KSAs Claim about student DESCRIPTION Data concerning performance Warrant for since scoring Student acting in assessment situation Other information concerning student vis assessment situation Variable Aspects of assessment situations that The Additional KSAs difficulty. DO features of can be varied to shift you or focus. tasks want to include as targets of Potential work What students actually say, do, or inference are part ofevidence. the claim. products make, to produce E. g. , Potential knowing of work products we might Aspects Mendel’s laws as observations being able to formulate a well as identify and evaluate, as evidence about students’ KSAs. model in an investigation. Connected with Variable Potential rubrics Ways of evaluating work products to produce valuesof observations. Features of Tasks.
How Design Patterns Support Thinking about the Assessment Argument ATTRIBUTE Focal Knowledge, Skills, Abilities unless Warrant for assessment argument Warrant for task design since How/why this DP addresses evidence about focal KSAs Other knowledge/skills/abilities that may be required by tasks. Aspects of assessment situations that are needed to evoke evidence about the focal KSAs. since so Data concerning situation Rationale Characteristic features of tasks Alternative explanations The primary knowledge / skills / abilities (KSAs) targeted by this design pattern. Additional KSAs Claim about student DESCRIPTION Data concerning performance Warrant for since scoring Student acting in assessment situation Other information concerning student vis assessment situation The Additional KSAs you Variable Aspects of assessment situations that DON’T want to include as focus. features of can be varied to shift difficulty or tasks targets of inference introduce Potential work What students actually say, do, or alternative explanations for poor products make, to produce evidence. performance. work products we might Potential Aspects of (Especially observations identify assessing special important forand evaluate, as evidence about students’ KSAs. Connected with Variable populations – UDL & Potential Ways of evaluating work products to Features of Tasks acommodations. ) & Work rubrics produce values of observations. Products.
How Design Patterns Support Thinking about the Assessment Argument ATTRIBUTE Focal Knowledge, Skills, Abilities unless Warrant for assessment argument Warrant for task design since How/why this DP addresses evidence about focal KSAs Other knowledge/skills/abilities that may be required by tasks. Aspects of assessment situations that are needed to evoke evidence about the focal KSAs. since so Data concerning situation Rationale Characteristic features of tasks Alternative explanations The primary knowledge / skills / abilities (KSAs) targeted by this design pattern. Additional KSAs Claim about student DESCRIPTION Data concerning performance Warrant for since scoring Student acting in assessment situation Other information concerning student vis assessment situation Variable features of tasks Aspects of assessment situations that can be varied to shift difficulty or focus. The What students actually say, do, or Potential work. Characteristic Features of products make, to produce evidence. Tasks help you think about Potential Aspects of critical data work products wethe concerning might observations identify and evaluate, as evidence about students’ KSAs. situation –what you need to get Potential Ways of evaluating work products to evidence about the Focal KSAs. rubrics produce values of observations.
How Design Patterns Support Thinking about the Assessment Argument ATTRIBUTE Focal The primary knowledge / skills / Variable abilities (KSAs) targeted by this design Knowledge, Features of Tasks also Skills, Abilities pattern. Claim about student unless Warrant for assessment argument Alternative explanations since so Data concerning situation Warrant for task design since Student acting in assessment situation help you think about data Rationale How/why this DP addresses evidence concerning the situation – but about focal KSAs now to Other knowledge/skills/abilities Additional influence difficulty …that KSAs may be requiredin or reduce or to bring by tasks. Characteristic features of tasks Data concerning performance Warrant for since scoring DESCRIPTION Other information concerning student vis assessment situation demand for Additional KSAs Aspects of assessment situations that are needed to evoke evidence about to avoid alternative the focal KSAs. explanations. Variable features of tasks Aspects of assessment situations that can be varied to shift difficulty or focus. Potential work products What students actually say, do, or make, to produce evidence. Potential observations Aspects of work products we might identify and evaluate, as evidence about students’ KSAs. Potential rubrics Ways of evaluating work products to produce values of observations.
How Design Patterns Support Thinking about the Assessment Argument ATTRIBUTE Claim about student unless Warrant for assessment argument Alternative explanations since so Data concerning situation Warrant for task design since Data concerning performance Warrant for since scoring Student acting in assessment situation DESCRIPTION Focal The primary knowledge skills Some Variable Features of/by this/design Tasks Knowledge, abilities (KSAs) targeted help you pattern. Skills, Abilities match features of tasks Rationale How/why / DP addresses evidence and backgroundthisknowledge / about focal KSAs characteristics of students: Additional Other knowledge/skills/abilities that Interests, may be required byprevious familiarity, tasks. KSAs instruction. Characteristic Aspects of assessment situations that features of tasks Other information concerning student vis assessment situation are needed to evoke evidence about the focal KSAs. Variable features of tasks Aspects of assessment situations that can be varied to shift difficulty or focus. Potential work products What students actually say, do, or make, to produce evidence. Potential observations Aspects of work products we might identify and evaluate, as evidence about students’ KSAs. Potential rubrics Ways of evaluating work products to produce values of observations.
How Design Patterns Support Thinking about the Assessment Argument ATTRIBUTE Focal The primary knowledge / skills / Knowledge, abilities (KSAs) targeted by this design Potential Work Products help you Skills, Abilities pattern. Claim about student unless Warrant for assessment argument Alternative explanations since so Data concerning situation Warrant for task design since Student acting in assessment situation think about whatthis DP want to you addresses evidence How/why about performance – capture from afocal KSAs Additional process, constructed that product, Other knowledge/skills/abilities KSAs may be required by tasks. model, written explanation, etc. Rationale Characteristic Aspects of assessment situations that features of are needed Can also call attention to to evoke evidence about tasks the focal demand for Additional KSAs. Data concerning performance Warrant for since scoring DESCRIPTION Other information concerning student vis assessment situation Variable Aspects of assessment situations that KSAs, & avoid alternative features of can be varied explanations (e. g. , Stella) to shift difficulty or focus. tasks Potential work products What students actually say, do, or make, to produce evidence. Potential observations Aspects of work products we might identify and evaluate, as evidence about students’ KSAs. Potential rubrics Ways of evaluating work products to produce values of observations.
How Design Patterns Support Thinking about the Assessment Argument ATTRIBUTE Focal Knowledge, Skills, Abilities unless Warrant for assessment argument Alternative explanations Warrant for task design since Rationale How/why this DP addresses evidence about focal KSAs Other knowledge/skills/abilities that may be required by tasks. since so Data concerning situation The primary knowledge / skills / abilities (KSAs) targeted by this design pattern. Additional KSAs Claim about student DESCRIPTION Data concerning performance Warrant for since scoring Student acting in assessment situation Other information concerning student vis assessment situation Potential Observations are Characteristic Aspects of assessment situations that possibilities for the evoke evidence about features of are needed to qualities of tasks the focal – i. e. , Work Products KSAs. the data Variable Aspects of assessment situations concerning be varied to shift difficulty or that the performance. focus. features of can tasks Potential work products What students actually say, do, or make, to produce evidence. Potential observations Aspects of work products we might identify and evaluate, as evidence about students’ KSAs. Potential rubrics Ways of evaluating work products to produce values of observations.
How Design Patterns Support Thinking about the Assessment Argument ATTRIBUTE Focal Knowledge, Skills, Abilities Claim about student unless Warrant for assessment argument Alternative explanations Warrant for task design since The primary knowledge / skills / abilities (KSAs) targeted by this design pattern. Rationale How/why this DP addresses evidence about focal KSAs since Additional And KSAs so Data concerning situation DESCRIPTION algorithms/rubrics/rules for Characteristic Aspects of assessment situations that features of evaluating are needed to evoke evidence about Work Products to get tasks the focal KSAs. Data concerning performance Warrant for since scoring Student acting in assessment situation Other knowledge/skills/abilities that Potential Rubrics are may be required by tasks. Other information concerning student vis assessment situation the Variable features of tasks data concerning the Aspects of assessment situations that can be varied to shift performance. difficulty or focus. Potential work products What students actually say, do, or make, to produce evidence. Potential observations Aspects of work products we might identify and evaluate, as evidence about students’ KSAs. Potential rubrics Ways of evaluating work products to produce values of observations.
For more information… § PADI: Principled Assessment Design for Inquiry § § http: //padi. sri. com NSF project, collaboration with SRI et al. Links to follow-on projects Bob Mislevy home page § § § http: //www. education. umd. edu/EDMS/mislevy/ Links to papers on ECD Cisco applications
Now for the Good Stuff … § Examples of design patterns with content § Different projects § Different grain sizes § Different users § How they evolved to suit needs of users § Same essential structure § Representations, language, emphases, and affordances tuned to users and needs § How they are being used
Use of Design Patterns in STEM Research and Development Projects Britte Haugan Cheng and Geneva Haertel DRK-12 PI Meeting, November 2009
Current Catalog of Design Patterns q ECD/PADI related projects have produced over 100 Design Patterns q Domains include: science inquiry, science content, mathematics, economics, model-based reasoning q Design Patterns span grades 3 -16+ q Organized around themes, models, and processes, not surface features or formats of tasks q Support the design of scenario-based, multiple choice, and performance tasks q The following examples show projects have used and customized Design Patterns in ways that suit their needs and users
Example 1 DRK-12 Project An Application of ECD to a State, Large-scale Science Assessment q Challenge in Minnesota Comprehensive Assessment of science: q How to design scenario-based tasks, technology-enhanced interactions, grounded in standards both EFFICIENTLY and VALIDLY. q Design Patterns support storyboard writing and task authoring q Designers are committee of MN teachers, supported by Pearson q Project focuses on a small number of Design Patterns for “hard-toassess” science content/inquiry q Based on Minnesota state science standards and benchmarks and the NSES inquiry standards q Design Patterns are Web-based and interactive
Design Pattern Observational Investigation q Relates science content/processes to components of assessment argument q Higher-level, cross-cutting themes, ways of thinking, ways of using science, rather than many finer-grained standards q Related to relevant standards and benchmarks q Interactive Features: q Examples and details q Activate pedagogical content knowledge q Presents exemplar assessment tasks q Provides selected knowledge representations q Links among associated assessment argument components
Design Pattern Observational Investigation
Design Pattern Observational Investigation (cont. )
Design Pattern Observational Investigation (cont. )
Interactive Feature: Details
Interactive Feature: Linking assessment argument components
Design Pattern Highlights Observational Investigation q Relates science content/processes to components of assessment argument q Higher-level, cross-cutting themes, ways of thinking, ways of using science, rather than many fine-grained standards q Interactive Features: q Examples and details q Activates pedagogical content knowledge q Presents exemplar assessment tasks q Provides selected knowledge representations q Relates relevant standards and benchmarks q Links among associated assessment argument components
Design Pattern Reasoning about Complex Systems q Relates science content/processes to components of assessment argument q Across scientific domains and standards q Convergence among the design of instruction, assessment and technology q Interactive Features: q Explicit support for designing tasks around multi-year learning progression
Design Pattern Reasoning about Complex Systems
Interactive Feature: Details
Interactive Feature: Linking assessment argument components
Design Pattern Highlights Reasoning about Complex Systems q Relates science content/processes to components of assessment argument q Across scientific domains and standards q Convergence among the design of instruction, assessment and technology q Interactive Feature: q Explicit support for designing tasks around multi-year learning progression
Example 2 Principled Assessment Designs in Inquiry Model-Based Reasoning Suite q Relates science content/processes to components of assessment argument q A suite of seven related Design Patterns support curriculum-based assessment design q Theoretically and empirically motivated by Stewart and Hafner (1994), Research on Problem-Solving: Genetics. In D. L. Gable (Ed. ), Handbook of research on science teaching and learning. New York: Mac. Millan Publishing. q Aspects of model-based reasoning including model formation, model use, model revision, and coordination among aspects of model-based reasoning q Multivariate student model: scientific reasoning and science content q Interactive Feature: q Support the design of both: q Independent tasks associated with an aspect of model-based reasoning q Steps in a larger investigation comprised of several aspects including model conceptualization, model use and model evaluation
Design Pattern Model Formation
Design Pattern Model Formation (cont. )
Interactive Feature: Links among Design Patterns
Design Pattern Highlights Model-based Reasoning Suite q Relates science content/processes to components of assessment argument q Facilitate the integration of model-based reasoning skills into any science content area q Serve as basis of a learning progression q Interactive Features: q Support the design of both independent tasks associated with an aspect of model-based reasoning and steps in a larger investigation that is comprised of several aspects including conceptualization of a model to its use and evaluation q Explicit supports (links among Design Patterns) for designing both investigations and focused tasks
Example 3 Principled Science Assessment Designs for Students with Disabilities Designing and Conducting Scientific Investigations Using Appropriate Methodology q Relates science content/processes to components of assessment argument q Guide refinement of science assessment tasks across multiple states by identifying and reducing sources of construct-irrelevant variance q Integrate six categories of Universal Design for Learning (UDL) into the assessment design process: q Perceptual, linguistic, cognitive, motoric, executive, affective q Interactive Feature: q Highlight relationships among Additional KSAs, Variable Task Features and Potential Work Products to reduce construct-irrelevant variance in a systematic manner
Design Pattern Designing and Conducting a Scientific Investigation Using Appropriate Methodology
Design Pattern Designing and Conducting a Scientific Investigation Using Appropriate Methodology (cont. )
Interactive Feature: Linking Additional KSAs and Potential Work Products
Design Pattern Highlights Designing and Conducting a Scientific Investigation Using Appropriate Methodology q Relates science content/processes to components of assessment argument q Integrate UDL in assessment design process rather than applying accommodations to an existing task q Supports the selection of task features that reduce construct-irrelevant variance and enhance the performance of all test takers q Particular attention to knowledge representation and executive processing demands q Further customization of Design Patterns to develop assessment tasks for students with particular disabilities q Interactive Feature: q Relate the perceptual and expressive capabilities required to complete an assessment task to that task’s features (Additional KSAs, Variable Task Features and Potential Work Products)
Example 4 Alternate Assessments in Mathematics Describe, extend, and make generalizations about geometric and numeric patterns q Relates math content/processes to components of assessment argument q Standards-based Design Patterns co-designed across three states to guide the development of statewide assessment tasks for students with significant cognitive disabilities q Integration of six UDL categories into the design process q Interactive Feature: q For logistical reasons, Word document used to create Design Patterns q Attributes visualized in accordance with the assessment argument resulting in increased efficiency and improved quality of argument q New arrangement now under development for use in online system
Design Pattern Describe, extend, and make generalizations about geometric and numeric patterns
Design Pattern Describe, extend, and make generalizations about geometric and numeric patterns (cont. )
Design Pattern Describe, extend, and make generalizations about geometric and numeric patterns (cont. )
Design Pattern Describe, extend, and make generalizations about geometric and numeric patterns (cont. )
Interactive Feature: Horizontal View Aligning Focal KSAs, Potential Observations and Potential Work Products
Interactive Feature: Horizontal View Aligning Additional KSAs and Variable Task Features
Design Pattern Highlights Describe, extend, and make generalizations about geometric and numeric patterns q Relates math content/processes to components of assessment argument q Deconstruction of NCTM expectations to identify KSAs that are less difficult or tasks that assess related cognitive background knowledge q Supports the principled alignment of task difficulty and scope with challenges to accessibility q Interactive Feature: q Use of multiple views of the Design Pattern to support understanding of the relationship of components of the assessment argument q Increased efficiency of design and validity of assessment argument
Summary q Design Patterns are organized around assessments and key ideas in science and math, as opposed to surface features of assessment tasks. q Support designing tasks that move in ways NSES and NCTM advocate in ways that build on research and experience q Design Patterns support task design for different purposes and different formats (e. g. , learning, summative, classroom, large-scale, hands-on, P&P, simulations). q Especially important for newer forms of assessment q Technology-based, scenario based tasks in Minnesota q Scenario-based learning & assessment (Foothill-De. Anza project) q Simulation-based tasks (network troubleshooting, with Cisco) q Games-based assessment (just starting, with Mac. Arthur project)
Summary q Design Patterns are eclectic—they are not associated with any particular underlying theory of learning or cognition; all psychological perspectives can be represented q Document design decisions q Represent hierarchical relationships among Focal KSAs, sequential steps required for the completion of complex tasks, or superordinate, subordinate, and coordinate relations among concepts q Re-usable; a family of assessment tasks can be produced from a single Design Pattern q Enhance the integration of UDL with the evidence-centered design process q Technology makes evident the relationships among Design Pattern attributes and their role in the assessment argument
Скачать презентацию Design Patterns Supporting Task Design by Scaffolding the
b436f882c21cfe6fa5211aa909b6c163.ppt