Storytelling as an Instructional Method Generating and Assessing

Storytelling as an Instructional Method Generating and Assessing Stories in Support of Instruction J. Michael Spector Florida State University, Tallahassee, FL USA mspector@lsi. fsu. edu Nov 7 -8, 2006 Mesa, Arizona Learning Systems Institute, Florida State University, Suite 4600 -C University Center, Tallahassee, FL 32306 -2540 (PHONE): (850) 644 -2570 (FAX): (850) 644 -4952 URL: WWW. LSI. FSU. EDU

Outline: Twice Told Tales – 2 Stories & 2 Ideas • • • Tell me a story When How Generating stories automatically Assessing learning with stories Learning Systems Institute, Florida State University, Suite 4600 -C University Center, Tallahassee, FL 32306 -2540 (PHONE): (850) 644 -2570 (FAX): (850) 644 -4952 URL: WWW. LSI. FSU. EDU

A Tale of Two Stories • A Game of India – H. A. Nielsen (Michigan Quarterly Review, 1978) – Experience first … then understanding – Humility … we know less than we are inclined to believe • A Confession – Lev Tolstoy, 1884 – Learning as disturbance – Quine & Ullian’s Web of Belief – The interconnectedness of experience Learning Systems Institute, Florida State University, Suite 4600 -C University Center, Tallahassee, FL 32306 -2540 (PHONE): (850) 644 -2570 (FAX): (850) 644 -4952 URL: WWW. LSI. FSU. EDU

Why • Natural inclination – It is what people do • Memory – efficiency of episodic memory (Anderson, 1983; Schacter, 1996) – efficient encoding, decoding and recall • Complexity – Confronting complexity indirectly – tacit knowledge Learning Systems Institute, Florida State University, Suite 4600 -C University Center, Tallahassee, FL 32306 -2540 (PHONE): (850) 644 -2570 (FAX): (850) 644 -4952 URL: WWW. LSI. FSU. EDU

Story-based Approaches Goal-based Scenarios Storytelling Case Studies Dynamic Stories Structure Case analyses – Verbal or developed over written; static time; static Case Generated documentation; situations, static problems, solutions Form Case, scenario story case Problem scenario Learning Approach Experiential learning Reflective learning Learning from examples Multi-faceted Aim Acquire specific knowledge and skills Enhanced understanding Acquiring case experience Deep insights into complex problems Application Education, training Organizational learning, leadership, etc. Medical and Wide variety business educ. , engineering Learning Systems Institute, Florida State University, Suite 4600 -C University Center, Tallahassee, FL 32306 -2540 (PHONE): (850) 644 -2570 (FAX): (850) 644 -4952 URL: WWW. LSI. FSU. EDU

When • With younger learners – widespread acceptance • With millennials … growing expectations • With what types of instructional goals and tasks? • With non-recurrent, ill-structured, complex, dynamic problem solving tasks! Learning Systems Institute, Florida State University, Suite 4600 -C University Center, Tallahassee, FL 32306 -2540 (PHONE): (850) 644 -2570 (FAX): (850) 644 -4952 URL: WWW. LSI. FSU. EDU

How • • Personal delivery … pacing, intonation, … Computer-based delivery Provide an overview – scenarios Generate interest Motivate inquiry Reflective exercises Computers generating stories Learning Systems Institute, Florida State University, Suite 4600 -C University Center, Tallahassee, FL 32306 -2540 (PHONE): (850) 644 -2570 (FAX): (850) 644 -4952 URL: WWW. LSI. FSU. EDU

Automatic Generation of Stories People tell stories. They tell stories to fit a situation or need. The use of stories in that sense is dynamic. Can an instructional computing system tell a story based on an underlying mathematical model and problem scenario or need? This possibility exists – an important research agenda worth pursuing. Learning Systems Institute, Florida State University, Suite 4600 -C University Center, Tallahassee, FL 32306 -2540 (PHONE): (850) 644 -2570 (FAX): (850) 644 -4952 URL: WWW. LSI. FSU. EDU

A Visual Representation Learning Systems Institute, Florida State University, Suite 4600 -C University Center, Tallahassee, FL 32306 -2540 (PHONE): (850) 644 -2570 (FAX): (850) 644 -4952 URL: WWW. LSI. FSU. EDU

Generating a Problem • Create and validate a system dynamics simulation model for a complex, dynamic system – a non-trivial task but many such simulation models already exist • Ensure that each variable, stock and constant are well documented • Given the current state of the simulation model and the learner’s experience, generate a problem situation from the model itself Learning Systems Institute, Florida State University, Suite 4600 -C University Center, Tallahassee, FL 32306 -2540 (PHONE): (850) 644 -2570 (FAX): (850) 644 -4952 URL: WWW. LSI. FSU. EDU

This depicts a system dynamics model – the highlighted flow rate is opened. There is a documentation tab in Power. Sim that allows each component to be described. This can be used to generate a task for someone interacting with the simulation model. Learning Systems Institute, Florida State University, Suite 4600 -C University Center, Tallahassee, FL 32306 -2540 (PHONE): (850) 644 -2570 (FAX): (850) 644 -4952 URL: WWW. LSI. FSU. EDU

Interaction and Feedback Learning Systems Institute, Florida State University, Suite 4600 -C University Center, Tallahassee, FL 32306 -2540 (PHONE): (850) 644 -2570 (FAX): (850) 644 -4952 URL: WWW. LSI. FSU. EDU

Assessing Learning with Stories How well do stories support learning in and about complex, ill-structured domains? How to determine? - standard problems with standard solutions are not available A developmental pathway - inexperienced problem solving and decision making towards expert-like problem solving and decision making Learning Systems Institute, Florida State University, Suite 4600 -C University Center, Tallahassee, FL 32306 -2540 (PHONE): (850) 644 -2570 (FAX): (850) 644 -4952 URL: WWW. LSI. FSU. EDU

DEEP – An Assessment Tool http: //deep. lsi. fsu. edu/DMVS/jsp/index. htm • Problem – determine progress of learning in complex domains • Approach – identify and annotate key influence factors • Strategy – compare responses to those of known experts and track development • Tactic – minimize extraneous cognitive load on respondents Learning Systems Institute, Florida State University, Suite 4600 -C University Center, Tallahassee, FL 32306 -2540 (PHONE): (850) 644 -2570 (FAX): (850) 644 -4952 URL: WWW. LSI. FSU. EDU

Capturing Problem Conceptualizations Present a problem situation or scenario, ask respondents to: 1. indicate factors (name and briefly describe) they believe critical to resolving the situation 2. indicate how these factors are interrelated (draw links and describe relations) 3. identify the assumptions involved thus far 4. describe additional information that would be required to resolve the situation or solve the problem Learning Systems Institute, Florida State University, Suite 4600 -C University Center, Tallahassee, FL 32306 -2540 (PHONE): (850) 644 -2570 (FAX): (850) 644 -4952 URL: WWW. LSI. FSU. EDU

Learning Systems Institute, Florida State University, Suite 4600 -C University Center, Tallahassee, FL 32306 -2540 (PHONE): (850) 644 -2570 (FAX): (850) 644 -4952 URL: WWW. LSI. FSU. EDU

Problem Conceptualization & Capture Learning Systems Institute, Florida State University, Suite 4600 -C University Center, Tallahassee, FL 32306 -2540 (PHONE): (850) 644 -2570 (FAX): (850) 644 -4952 URL: WWW. LSI. FSU. EDU

DEEP in USE Learning Systems Institute, Florida State University, Suite 4600 -C University Center, Tallahassee, FL 32306 -2540 (PHONE): (850) 644 -2570 (FAX): (850) 644 -4952 URL: WWW. LSI. FSU. EDU

Medical Summary Data 5 Experts Scenario 1 14 Novices Percent cause/effect 73 68. 9% example 0 0. 0% correlation 8 7. 5% 25 23. 6% 106 100% process TOTAL Links Percent cause/effect 185 58. 2% example 24 7. 5% correlation 84 26. 4% process 25 7. 9% TOTAL Links 5 Experts 318 14 Novices Percent Scenario 2 cause/effect Percent cause/effect 28 41. 2% 0 0. 0% correlation 10 14. 7% correlation process 30 44. 1% process 68 100% example TOTAL Links 137 46. 6% 27 9. 2% 123 41. 8% 7 2. 4% 294 100% Learning Systems Institute, Florida State University, Suite 4600 -C University Center, Tallahassee, FL 32306 -2540 (PHONE): (850) 644 -2570 (FAX): (850) 644 -4952 URL: WWW. LSI. FSU. EDU

Node-Link Clusters Learning Systems Institute, Florida State University, Suite 4600 -C University Center, Tallahassee, FL 32306 -2540 (PHONE): (850) 644 -2570 (FAX): (850) 644 -4952 URL: WWW. LSI. FSU. EDU

Comparing Experts & Novices Links Biology Experts – Scenario 2 (N = 5; Links = 128) From / To Percentage Nodes 1 Analyze the mercury concentrations 16 9/7 12. 5% 2 Possible solutions 14 7/7 10. 93% 3 Food chain 12 7/5 9. 37% 3 Source of mercury contamination 12 8/4 9. 37% 4 Limit consumption 11 7/4 8. 59% Biology Novices – Scenario 2 (N = 16; Links = 147) Links From / To Percentage Nodes 1 Biological effects of mercury on fish 41 21 / 20 27. 89% 1 Source of mercury contamination 41 24 / 17 27. 89% 2 Human interaction 29 12 / 17 19. 72% 3 Social awareness 28 13 / 15 19. 04% 4 Analyze the mercury concentrations 24 17 / 7 16. 32% Learning Systems Institute, Florida State University, Suite 4600 -C University Center, Tallahassee, FL 32306 -2540 (PHONE): (850) 644 -2570 (FAX): (850) 644 -4952 URL: WWW. LSI. FSU. EDU

Additional Issues & Measures • Separating structural and semantic analysis • Structural analysis – Central nodes – Terminal nodes (all links in same direction) – Feedback and systemic measures • Similarity metrics – Graph theory – diameter, density, path analysis – Tversky similarity metric Learning Systems Institute, Florida State University, Suite 4600 -C University Center, Tallahassee, FL 32306 -2540 (PHONE): (850) 644 -2570 (FAX): (850) 644 -4952 URL: WWW. LSI. FSU. EDU

First Pass at a Systemic Metric • Hypothesis: experts will tend to think more systemically than non-experts • Indicators of systemic thinking: – Internal feedback (links back to other parts of the system; two-way links) – One possible measure – ratio of unreachable pairs to all possible ordered pairs of nodes in the problem conceptualization Learning Systems Institute, Florida State University, Suite 4600 -C University Center, Tallahassee, FL 32306 -2540 (PHONE): (850) 644 -2570 (FAX): (850) 644 -4952 URL: WWW. LSI. FSU. EDU

7 nodes – possible ordered pairs = 2, 520 lots of internal feedback depicted no unreachable pairs No terminal nodes Learning Systems Institute, Florida State University, Suite 4600 -C University Center, Tallahassee, FL 32306 -2540 (PHONE): (850) 644 -2570 (FAX): (850) 644 -4952 URL: WWW. LSI. FSU. EDU

7 nodes – possible pairs much internal feedback 6 unreachable pairs 1 terminal node Learning Systems Institute, Florida State University, Suite 4600 -C University Center, Tallahassee, FL 32306 -2540 (PHONE): (850) 644 -2570 (FAX): (850) 644 -4952 URL: WWW. LSI. FSU. EDU

7 nodes little internal feedback 6 terminal nodes 38 unreachable pairs Learning Systems Institute, Florida State University, Suite 4600 -C University Center, Tallahassee, FL 32306 -2540 (PHONE): (850) 644 -2570 (FAX): (850) 644 -4952 URL: WWW. LSI. FSU. EDU

An Ending “What we cannot speak about, we must pass over in silence” (Wittgenstein, #7, Tractatus Logico-Philosophicus) “The Moral to this story, the moral to this song, is simply that one should never be where one does not belong “ (Bob Dylan, The Ballad of Frankie Lee and Judas Priest) May the Force be with you … Learning Systems Institute, Florida State University, Suite 4600 -C University Center, Tallahassee, FL 32306 -2540 (PHONE): (850) 644 -2570 (FAX): (850) 644 -4952 URL: WWW. LSI. FSU. EDU

References (1/4) • • Anderson, J. R. , Bothell, D. , Byrne, M. M. , Douglass, S. , Leviere, C. , & Qin, Y. (2004). An integrated theory of the mind. Psychological Review, 111(4), 1036 -1060. Andre, T. (1986). Problem-solving in education. In G. D. Phye & T. Andre (Eds. ). Cognitive classroom learning (pp. 169– 204). New York: Academic Press. Andrews, G. & Halford, G. S. (2002). A cognitive complexity metric applied to cognitive development. Cognitive Psychology, 45(2), 153 -219. Bransford, J. D. , Brown, A. L. , Cocking, R. R. (Eds. ) (2000). How people learn: Brain mind experience and school. Washington, DC: National Academy Press. Chi, M. T. H. , Feltovich, P. J. , & Glaser, R. (1981). Categorization and representation of physics problems by experts and novices. Cognitive Science, 5, 121 -152. Christensen, D. L. , Spector, J. M. , Sioutine, A. , & Mc. Cormack, D. (2000, August). Evaluating the impact of system dynamics based learning environments: A preliminary study. Paper presented at the International System Dynamics Society Conference, Bergen, Norway, August, 2000. Collins, A. , Brown, J. S. , & Newman, S. E. (1989). Cognitive apprenticeship: Teaching the crafts of reading, writing, and mathematics. In L. B. Resnick (Ed. ), Knowing, learning, and instruction: Essays in honor of Robert Glaser (pp. 453 -494). Hillsdale, NJ: Lawrence Erlbaum. Dijkstra, E. (1972). The humble programmer. Communications of the ACM, 15(10), 859 -866. • Dörner, D. (1996). The logic of failure: Why things go wrong and what we can do to make them right (R. Kimber & R. Kimber, Trans. ). New York: Metropolitan Books. (Original work published in 1989). • Dummer, P. , & Ifenthaler, D. (2005, March). Planning and assessing navigation in model-centered learning environments: Why learners often do not follow the path laid out for them. Presentation at the International Conference for Methods and Technologies for Teaching, Palermo, Italy, March 10, 2005. Ericsson, K. A, (1993). The role of deliberate practice in the acquisition of expert performance. Psychological Review, 100(3), 363 -406. Ericsson, K. A. (2001). Attaining excellence through deliberate practice: Insights from the study of expert performance. In M. Ferrari (Ed. ), The pursuit of excellence in education (pp. 21 -55). Mahwah, NJ: Erlbaum. • • Learning Systems Institute, Florida State University, Suite 4600 -C University Center, Tallahassee, FL 32306 -2540 (PHONE): (850) 644 -2570 (FAX): (850) 644 -4952 URL: WWW. LSI. FSU. EDU

References (2/4) • • • • Gardner, H. (1999). Multiple approaches to understanding. In C. M. Reigeluth (Ed. ), Instructional design theories and models: A new paradigm of instructional theory (Vol. II) (pp. 69– 89). Mahwah, NJ: Lawrence Erlbaum Associates. Herl, H. E. , O’Neil, H. F. , Jr. , Chung, G. L. W. K. , Bianchi, C. , Wang, S. , Mayer, R. , Lee, C. Y. , Choi, A. , Suen, T. , & Tu, A. (1999). Final report for validation of problem solving measures. CSE Technical Report 501. Los Angeles: CRESST. Johnson-Laird, P. (1983). Mental models. Cambridge: Cambridge University Press. Jonassen, D. (1999) Designing constructivist learning environments. In C. M. Reigeluth (Ed. ), Instructional design theories and models: A new paradigm of instructional theory (Vol. II) (pp. 215– 239). Mahwah, NJ: Lawrence Erlbaum Associates. Jonassen, D. H. & Hernandez-Serrano, J. (2002). Case-based reasoning and instructional design: using stories to support problem solving. Educational Technology Research and Development, 50 (2), 65 -77. Jonassen, D. H. (2006). Modeling with technology: Mindtools for conceptual change (3 rd ed. ). Columbus, OH: Pearson/Prentice Hall. Kintsch, E. (1993). Principles of instruction from research on human cognition. In J. M. Spector, M. C. Polson, & D. J. Muraida, Automating instructional design: Concepts and issues (pp. 22 -42). Englewood Cliffs, NJ: Educational Technology Publications. Lowyck, J. , & Elen, J. (2004). Linking ICT, knowledge domains, and learning support for the design of learning environments. In N. M. Seel & S. Dijkstra (Eds. ), Curriculum, plans and process in instructional design: International perspectives (pp. 239 -256). Mahwah, NJ: Erlbaum. Lowyck, J. , Pöysä, J. , & van Merriënboer, J. (2003). Conditions of ICT-based design for learning communities. Technology, Instruction, Cognition, and Learning, 1 (2), 153 -182. Mc. Carthy, B. (1996). About learning. Barrington, IL: Excell Inc. Merrill, M. D. (2002). First principles of instruction. Educational Technology Research and Development, 50(3), 43 -59. Miller, G. A. (1956). The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological Review, 63, 81 -97. Milrad, M. , Spector, J. M. , & Davidsen, P. I. (2002). Model facilitated learning. In S. Naidu (Ed. ), Learning and teaching with technology: Principles and practices (pp. 13 -27). London: Kogan Page. Learning Systems Institute, Florida State University, Suite 4600 -C University Center, Tallahassee, FL 32306 -2540 (PHONE): (850) 644 -2570 (FAX): (850) 644 -4952 URL: WWW. LSI. FSU. EDU

References (3/4) • • • Moreno, R. Mayer, R. , Spires, H. , Lester, J. (2001) The Case for social agency in computer-based teaching: Do students learn more deeply when they interact with animated pedagogical agents? Cognition and Instruction, 19(2), 177 -213. Nelson, L. M. (1999). Collaborative problem solving. In C. M. Reigeluth (Ed. ), Instructional design theories and models: A new paradigm of instructional theory (Vol. II) (pp. 241– 267). Mahwah, NJ: Erlbaum. Novak, J. D. (1998). Learning, creating and using knowledge: Concept maps™ as facilitative tools in schools and corporations. Mahwah, NJ: Erlbaum. Paas, F. , Renkl, A. , & Sweller, J. (Eds. ). (2003). Cognitive load theory. Educational Psychologist, 63(1), Whole Issue. Paivio, A. (1986). Mental representations: A dual coding approach. Oxford, England: Oxford University Press. Polson, M. C. (1993). Cognitive theory as a basis for instructional design. In J. M. Spector, M. C. Polson, & D. J. Muraida, Automating instructional design: Concepts and issues (pp. 5 -22). Englewood Cliffs, NJ: Educational Technology Publications. Salas, E. , & Fiore, S. (Eds. ) (2004). Team cognition: Understanding the factors that drive process and performance. Washington, DC: American Psychology Association. Schank, R. C. , Berman, T. R. & Macperson, K. A. (1999). Learning by doing. In C. M. Reigeluth (Ed. ), Instructional design theories and models: A new paradigm of instructional theory (Vol. II) (pp. 161– 181). Mahwah, NJ: Lawrence Erlbaum Associates. Schvaneveldt, R. W. (Ed. ) (1990). Pathfinder associative networks: Studies in knowledge organization. Norwood, NJ: Ablex. Schwartz, D. , Lin, X. , Brophy, S. , & Bransford, J. D. (1999). Toward the development of flexibly adaptive instructional designs. In C. M. Reigeluth (Ed. ), Instructional design theories and models: A new paradigm of instructional theory (Vol. II) (pp. 183– 213). Mahwah, NJ: Lawrence Erlbaum Associates. Seel, N. M. (2001). Epistemology, situated cognition, and mental models: Like a bridge over troubled water. Instructional Science, 29(4 -5), 403 -427. Learning Systems Institute, Florida State University, Suite 4600 -C University Center, Tallahassee, FL 32306 -2540 (PHONE): (850) 644 -2570 (FAX): (850) 644 -4952 URL: WWW. LSI. FSU. EDU

References (4/4) • • • • Seel, N. M. (2003). Model-centered learning and instruction. Technology, Instruction, Cognition, and Learning, 1(1), 59 -86. Seel, N. , Al-Diban, S. , & Blumschein, P. (2000). Mental models and instructional planning. In J. M. Spector & T. M. Anderson (Eds. ), Integrated & Holistic Perspectives on Learning, Instruction & Technolog (pp. 129 -158). Dordrecht, The Netherlands: Kluwer. Spector, J. M. (2000, Fall). Trends and issues in educational technology: How far we have not come. Update Semiannual Bulletin 21(2). Spector, J. M. (2001). A philosophy of instructional design for the 21 st century? Journal of Structural learning and Intelligent Systems 14(4), 307 -318. Spector, J. M. , & Anderson, T. M. (2000). Integrated and holistic perspectives on learning, instruction and technology: Understanding complexity. Dordrecht: Kluwer. Spector, J. M. , & Koszalka, T. A. (2004). The DEEP methodology for assessing learning in complex domains (Final report the National Science Foundation Evaluative Research and Evaluation Capacity Building). Syracuse, NY: Syracuse University. Spector, J. M. , Dennen, V. P. , & Koszalka, T. A. (2005, July). Causal maps, mental models and assessing acquisition of expertise. Proceedings of the 11 th Annual Human Computer Interaction International Conference, Las Vegas, Nevada, July, 2005. Spiro, R. J. , Feltovich, P. J. , Jacobson, M. J. , & Coulson, R. L. (1991, May). Cognitive flexibility, constructivism, and hypertext: Random access instruction for advanced knowledge acquisition in illstructured domains. Educational Technology, 31 (5), 24 -33. Stoyanova, N. , & Kommers, P. (2002). Concept mapping as a medium of shared cognition in computersupported collaborative problem solving. Journal of Interactive Learning Research, 13(1/2), 111 -133. Suppes, P. (1978). Impact of research on education: Some case studies. Washington, DC: National Academy of Education. Sweller, J. (2003). Evolution of human cognitive architecture. The psychology of learning and motivation, 43 , 215– 266. Taricani, E. M. , & Clariana, R. B. (2006). A technique for automatically scoring open-ended concept maps. Educational Technology Research &Development, 54 (1), 65 -82. van Merriënboer, J. J. G. (1997). Training complex cognitive skills: A four-component instructional design model for technical training. Englewood Cliffs, NJ: Educational Technology Publications. Learning Systems Institute, Florida State University, Suite 4600 -C University Center, Tallahassee, FL 32306 -2540 (PHONE): (850) 644 -2570 (FAX): (850) 644 -4952 URL: WWW. LSI. FSU. EDU