t assessmen al swer inform ur Please an

t assessmen al swer inform ur Please an gray on yo in ns shaded. questio e we begin for handout be Demystifying Assessment: Engaging Faculty with Fundamental Assessment Practices for Quality Enhancement SACS Annual Meeting December 4, 2011 Patricia R. Payette, Ph. D. Executive Director, Ideas to Action Associate Director, Delphi Center for Teaching and learning patty. payette@louisville. edu 1

Goals of presentation ü Examine the University of Louisville as a case study regarding QEP assessment practices & principles (focus: critical thinking) ü Share challenges related to assessment ü Share effective practices and best practices ü Apply relevant concepts from the presentation to participants’ own contexts What else you need to know about this session. . . ! 2

“Logic of” Assessment § § One word… Purpose Key Question or Problem Point of View 3

Logic of Assessment at Uof. L § One word: meaningless § Purpose: to foster accountability & improvement § Key Question or Problem: how do we contend with our uneven, decentralized assessment practices to report systematically? § Points of View: students, faculty, staff, SACS, disciplinary norms, best practices in assessment 4

Words of Wisdom from Uof. L Check for, and reinforce, a shared conceptual understanding of • Goals • Rationale • Practices • Protocols related to assessment. 5

Example of demystifying assessment: NSSE http: //louisville. edu/nsseatuofl http: //louisville. edu/institutionalresearch/files/assessments/Every. Card. Counts. Marketing. pdf/at_download/file 6

University of Louisville Mission: Kentucky’s premier, nationally recognized metropolitan research university • • • Established: 1798 in Louisville, KY Total Student Headcount: 22, 031 (Fall 2009) Faculty: 2, 125 and Staff: 3, 961 Operating Budget (2007 -2008): $946 million • Academic Programs (Degrees offered): – Undergraduate degrees, 78 programs (includes certificates, associate degrees, baccalaureate degrees, and post-baccalaureate certificates) – Graduate degrees, 106 programs (includes master’s degree, post-master’s certificates, doctoral degrees) – Professional degrees, 3 programs Source: University of Louisville’s “Just the Facts, ” 2009 -2010 (http: //www. louisville. edu)

Checking my assumptions Are you engaged in your developing, implementing or assessing your QEP ? Fist to Five: Zero: working on our topic One: have our topic, working on proposal Two: SACS visit is coming up, polishing proposal Three: Topic approved, early launching stage Four: Implementing right now Five: Preparing QEP report or just completed it 8

Conceptualizing the QEP … is designed to enhance student learning and/or the learning environment; … is driven by institution’s mission and what is achievable and affordable; … is determined through broad-based involvement of stakeholders; … is to be assessed by SACS through a five-year report that includes achievements, adjustments, and impact data (expected and unexpected) What Makes a Success QEP in relation to Student Learning Outcomes by Dr. Rudolf Jackson: www. sacscoc. org/staff/rjackson/Jackson%20 Excerpt. Governance%20 Seminar. June%202010. pdf 9

SACS acknowledges this reality: Your agenda will change the institution, and your institution will change your agenda. 10

Our QEP: Ideas to Action (i 2 a): Using Critical Thinking to Foster Student Learning and Community Engagement “Our extensive consultation with all university constituencies yielded a surprisingly strong and clear call for education focused on the skills and knowledge needed to deal with real-world issues and problems. ” (i 2 a SACS proposal, p. 22) Proposal: louisville. edu/ideastoaction/resources/archives 11

Focus of this presentation: critical thinking Sharpen our existing focus on building critical thinking skills in the general education program… …. . continuing through undergraduate major courses with an emphasis on applying and refining those skills… …resulting in a culminating undergraduate experience (CUE), such as a senior thesis, research, service learning project, internship, or capstone project that reflects authentic engagement 12

How are we measuring critical thinking? Eva luat ion Qualitative Impact Data aka “testimonials” Assessments CAAP General Education Critical thinking rubric FSSE and NSSE Note about standardized tests 13

Barbara Walvoord Purpose: Informed decision making (continuous improvement) Key Questions: • Are students learning what we want them to? • How can we better help them learn? Key Concept: Assessment includes 3 steps: goals, information, and action 14

Who engages in this cycle? How to pull people together? Pull together a powerful assessment team that reflects assessment interests and expertise of key stakeholders: § At the administrative/executive level § At the QEP/SACS liaison leadership level § At the faculty governance level § At the department/unit level § At the individual faculty/staff/student level We created an 2 a assessment subcommittee with those who can speak across academic 15 cultures

i 2 a Assessment Vision: Systematic, ongoing process to assess the evidence of undergraduate students’ ability to think critically and connect student learning to community for the purpose of enhancing the quality of the undergraduate educational experience and documenting accountability to accreditation agencies. https: //louisville. edu/ideastoaction/what/assessment 16

Goals drive ongoing decisions Our QEP Assessment Goals: ü Triangulation of direct & indirect assessments ü Consistent with one critical thinking framework ü Assessment of outcomes and changes/revisions ü Assessments that are “valued-added” to existing assessment processes ü Faculty input and participation 17

Words of Wisdom • Get comfortable with evolutionary change agenda regarding assessment • Expect QEP will grow beyond original boundaries and conceptual frame • Keep SACS expectations and Five-Year Interim Report frame in mind • Don’t wait for all change conditions to be “perfect” to proceed with assessment 18

Reflect/Apply Considering your assessment focus right now, try to describe your assessment vision (what, why and how) in one sentence: 19

First step: operationalizing our QEP concepts What do we mean when we talk about critical thinking? How will teach for these skills? How do we measure it across our campus? 20

Richard Paul-Linda Elder Framework http: //www. criticalthinking. org 21 Miniature Guide, 2008, p. 3 -6

University-level assessment buy-in ü We mapped Paul-Elder framework to existing assessment of critical thinking through our General Education Curriculum Committee (see handout) ü Considered the fit of standardized tests to our campus learning priorities ü MAPP, CLA or CAAP critical thinking test? CAAP ü NSSE items related to critical thinking and our QEP Resource on mapping you can Google: Stassen, Martha L. A. , Anne Herrington, and Laura Henderson. “Defining Critical Thinking in Higher Education: 22 Determining Assessment Fit. ” To Improve the Academy. Vol 30. 2011

Getting on the same page: shared concept of assessment What are our goals? What did we learn related to our goals? What inferences, conclusions can we draw in order to make informed decisions for our next steps? What specific measures/instruments are complementary to our goals? Gather our data using our instruments. Analyze my data 23

University-level Assessment at Uof. L Our room for growth Its not assessment itself that leads to improvement, it’s the action taken as a result of the assessment. - B. Walvoord 24

University-level Assessment: what we did • Tie QEP reporting to existing annual reporting processes for units. • Work with Office of Academic Planning and Accountability to decide on software needs. • Worked with GECC to align our goals • Transparent/clear in our assessment goals and resources. • Created together tutorials on student learning outcomes (SLOs) to help with uneven skill level: louisville. edu/institutionalresearch/institutionaleffectiveness/student-learning-outcome. html 25

Student Learning Outcomes Online Tutorials 26

Words of Wisdom ü Define terms and concepts ü Create outcomes collaboratively ü Create protocol or plan to explain it all https: //louisville. edu/ideastoaction/what/assessment ü Tie into existing reporting methods ü Don’t try to hide your campus reality ü Don’t wait for conditions to be perfect: Well, what can we realistically find out now from where we sit? What can we do with that information? 27

Reflect/Apply • Where do you see the strengths of your institution’s or colleagues’ engagement with the assessment cycle? • Where are the opportunities for improvement? How might you leverage those? 28

Assessment at Unit Level • Use disciplinary norms and accreditation expectations to align with QEP goals and outcomes • Customized workshops and training • Grant funds to bridge the gap 29

Critical thinking… 30

J. B. Speed School at Uof. L and critical thinking ENGR 100: Intro to Engineering • Critical thinking is using logic to decide what to believe based on accurate and objective evidence. • Critical thinking is the ability to think clearly and rationally. • Critical thinking is the process of conceptualizing, applying, analyzing, synthesizing, and/or evaluating information asguide to belief and action. a a guide to belief and action. Intellectual Standards = blue Pat Elements of Thought = red

Workshops/Training Example “The goal of the workshops, in addition to exposing these faculty members to the Paul Elder critical thinking model, is to develop a grid showing where each of the Elements of Thought and Intellectual Standards will be emphasized within the College of Business core curriculum. ” (see curriculum mapping handout). 32

Supporting Undergraduate i. Nnovation (SUN) Grants This internal funding program is designed to give faculty, staff, or teams of faculty and staff grants of up to $12, 500 to develop, implement, and assess projects that will directly and significantly support the sustained incorporation of selected i 2 a outcomes into undergraduate academic, co-curricular and student support projects, programs, courses and curricula. http: //louisville. edu/ideastoaction/grants 33

Sample SUN Grant assessment projects • Individual course project: “Incorporating Critical Thinking into the Required Business Writing Course (ENGL 306), ” Joanna Wolfe and Steve Smith, Department of English • Program-wide project: “Bachelor of Social Work Program Curriculum Development, ” Noell Rowan and Lynetta Mathis, Kent School of Social Work • Academic Affairs project: “Academic Improvement Model (AIM)”: Janet Spence and Nora Scobie; Undergraduate Advising 34

Words of Wisdom • Respect disciplinary norms and readiness • Balance consistency of curriculum with autonomy for the units • Sometimes you must focus on quality vs. quantity: who is “on the bus” is more important than how many (Jim Collins, “Good to Great”) Bottom line: one size does not fit all 35

Reflect/Apply • Who are/could be your supporters and early adopters? (or: who is your next important group? ) • What resources do you need to get them on board and moving forward? 36

Assessment and individual faculty: meet them where they are. “I’m already doing critical thinking!” 37

Critical Thinking & Faculty • Paul, 1996 • 140 randomly sampled California college faculty • 89% indicate critical thinking is a primary objective of their instruction – 19% could give a clear explanation of critical thinking – 77% had difficulty describing how to balance content coverage with fostering critical thinking – 8 -9% could articulate how to assess critical thinking

Focus on: your students, not SACS Think of a specific course that you teach, or a specific learning context in which you teach and/or mentor students to think critically. Describe in a short list the changes in students’ mindset (or “mental models”) you want to see in them at the end of your time with them in the classroom, lab, etc. (e. g. ask relevant questions). 39

Example from Uof. L faculty Public health faculty want students to be able to…. § Monitor the quality of their own thinking (e. g. “am I fooling § § § myself? ”) Discern dominant discourse from alternative discourse Translate ideas/thoughts into a testable hypotheses Believe there are no stupid questions, just stupid answers Critically evaluate a situation from multiple points of view Postulate an argument and rationally support it Develop an independent voice (despite grades) 40

Making critical thinking visible: A Well-Cultivated Critical Thinker § Raises vital questions and problems, formulating them clearly and precisely § Gathers and assesses relevant information, using abstract ideas to interpret it effectively § Comes to well-reasoned conclusions and solutions, testing them against relevant criteria and standards § Thinks open mindedly within alternative systems of thought, recognizing and assessing, as needs be, their assumptions, implications, and practical consequences § Communicates effectively with others in figuring out solutions to complex problems The Miniature Guide to Critical Thinking, 2008, page 2 41

Formative Assessment: Making critical thinking process visible The SEE-I prompt S: State it E: Elaborate E: Exemplify I: Illustrate 42

Gerald Nosich on the SEE-I “If you can accurately S, E, E, then I a concept or principle in a course, it means you almost certainly have a good grasp of it, that you understand it to a much greater degree than if you are merely able to state it. ” (ASSESSING CLARITY AND ACCURACY) Nosich, G. “Learning to Think Things Through: A Guide to Critical Thinking Across the Curriculum. ” (2009). p. 35. 43

SEE-I for “civil disobedience” • State – A definition is the “active refusal to obey certain laws & a primary method of nonviolent resistance. ” • Elaborate – In other words, a means by which citizens can actively rebel against unfair laws and the demands of the state • Example –An example would be like Rosa Parks refused to give up her seat on the bus • Illustrate – It’s like being a cliff at the edge of the ocean, waves crash against it, but the cliff remains there. 44

When to use a SEE-I • Use it as a starting point for discussion • “I’ll make a statement, and I want to someone to elaborate” • Use it between students or groups during a discussion. • During silence to help understanding • SEEI can also be used as a catalyst. • Provide an elaboration of a concept and ask students to provide examples. Connect it to other concepts and have them start the process over again. • “Who has an example of…? ” More info: http: //en. wikipedia. org/wiki/SEE-I 45

Words of Wisdom • Give theory/research and practical tools • Help faculty reflect on their learning goals, don’t lead with a “deficit • Enlist early adopters who model” can provide examples, testimony across the curriculum • Guide faculty to make an immediate connection to their teaching 46 http: //louisville. edu/ideastoaction/resources/faculty-exemplars

Faculty perspective “I think that for decades I have given my students many opportunities to engage in critical thinking, and I have modeled critical thinking in class discussions. But I don’t think I can claim ever to have taught critical thinking in a systematic way. The framework gives me a way to share a critical thinking vocabulary with students and to chart their progress. I know and can tell my students exactly what I am looking for. ” Faculty Learning Community participant, Department of English 47

Reflect/Apply What are the insights or ideas generated by this session you are taking back to your campus? What are your next steps? 48

Unanticipated outcomes of QEP • Partnering to produce i 2 a research/scholarship • Supporting projects launched by staff and students • Gathering powerful stories of transformation • Learning from challenges of campus culture • Serving as regional/national leaders • Developing unique community and campus partnerships 49

Focus for QEP Committee: Ø Organizing and interpreting qualitative and quantitative data Ø Synthesizing lessons learned Ø Providing robust implementation evidence in 10 pages Ø Deciding on pointed examples Ø Balancing expected “achievements” with unanticipated “adjustments” 50

Resources on i 2 a Ideas to Action: http: //louisville. edu/ideastoaction Annual Reports: http: //louisville. edu/ideastoaction/grants QEP Milestones: VI and beyond: Phase Assessing, Engaging and http: //louisville. edu/ideastoaction/what/implementation Reporting Staff Faculty, Students and to SACS Speak! Videos: http: //louisville. edu/ideastoaction/resources/media Phase VI and beyond: Assessing, Engaging and Evaluation/Assessment Protocol: Reporting to SACS https: //louisville. edu/ideastoaction/what/assessment Learning Communities: http: //louisville. edu/ideastoaction/aboutlc Phase VI and beyond: Info: Dr. Cathy Bays, i 2 a specialist of assessment: cathy. bays@ louisville. edu 51