Maximizing Your Campus Living Room Leveraging Assessment to

Maximizing Your Campus Living Room: Leveraging Assessment to Tell Your Union Story Presented by: Jason F. Simon, Ph. D. Director – Research, Assessment and Planning for Student Affairs University of North Texas Division

About Me • 17 years in higher education, 3 years in Assessment • 3 years in Union Administration • 3 years in Student Affairs (student activities, leadership programming, orientation, international student affairs, and campus safety) • 8 years in Advancement at UC Berkeley (student/alumni mentoring, alumni clubs, student homecoming, major events and alumni career services) • • Ph. D. in Higher Education from UNT M. Ed. In HESA from University of Vermont Originally from NJ – hold the jokes Metaphors and Puns Oh My!

About You • • • Public vs. Private Time in our field? First generation college student? Degrees? Prior assessment experience?

Our Session Today • Individual participation activity – One minute reflection & Mona Lisa Moment • The Comprehensive Union Assessment Toolbox – Evolution of Assessment in Student Affairs – Assessment 101: A primer/review – Process of Assessment Activity: Problem to Report • Creating a Culture of Evidence in the Union • Case Studies in Union Assessment • Closing Individual participation activity – One minute reflection & Mona Lisa Moment

Take A Moment • Take just one minute to respond: – When I think of Assessment in a Union environment I believe… • Please draw a Union staff member conducting assessment activities: – What is this person doing? – What items are on this person’s desk? – If there were posters on the walls describing the work being done what would they say?

Why Do This Work? • Dreams and data are interconnected – Every student has a dream and through our programming and services in our Unions we either help them achieve it or put a roadblock in their path – Data are just numbers unless Union staff can use it to close loops, make new meaning and improve our programming and services – SO…. I believe effort in assessment is a vital component to helping more of our students succeed

Jason’s Big 4 1. Union assessment doesn’t have to be hard – it just has to be intentional 2. The best Union assessment work is pragmatic - be realistic and take small steps on the path 3. Union assessment is a powerful tool to help build, change, grow, enhance, advocate for, improve, understand partner with others to promote student success 4. Everyone in an Union environment can play a part in nurturing assessment – those parts may be different but when combined become very powerful for your facility

Would You: • Drive to Florida without a map, GPS or stopping to ask for directions? • Have a medical procedure done without asking to hear the provider’s plan of action or success rates? • Host a dinner party for 20 people without first asking if your guests have any food allergies? • Yet we in our Unions sometimes… • Offer the same programs and services, the same time of year in the same way regardless of how our students may be evolving over time!

Why this presentation? Why now? How have we arrived at this point as a field?

Assessment 10 Years Ago and Now Satisfaction-Oriented Indirect Evidence-Based Technologically Limited Justification-Focused Methodological Simplicity Bottom-Up Culture Some Master’s level expertise needed • One person shops (usually part of shared duties) • Isolated professional opportunities • • • • Learning Outcome Oriented Indirect and Direct Evidence-Based Technologically Expanding Complex Foci Depending on Goals/Needs Increasing Study Complexity and Analysis Bottom-Up and Top-Down Culture Master’s level expertise the norm, increasingly Doctoral level expertise needed Increasing staff sizes ACUI, NILOA, SAAL, ASSESS, NASPA & ACPA growth

7 Key Areas of Assessment 1. Cultural - Is your Union ready to assess? 2. National Standards – How do you compare to your peers? What does CAS state? What does your EBI scores say? 3. Mission – Is your Union mission relevant? Current? 4. Users – Who actually uses your services? How do they match the larger population? 5. Needs – What do your users need? What are you offering? Is the Union still relevant? 6. Cost Effectiveness – Do users perceive value in the Union? Are you using resources wisely? 7. Learning Outcomes – What are users learning as a result of the Union? How do you know? Do you integrate learning outcomes into programming?

The Iterative Assessment Cycle Cited from an adaptation of work from Peggy Maki, Ph. D. by M. J. Bresciani, Ph. D. METHOD ZONE Gather Evidence Interpret Evidence ASSESS: Mission/Purposes Objectives/Goals Outcomes Implementation How well do we achieve our outcomes? Make decisions to improve; enhance student development; Enhance learning; Inform decision-making, planning, & budgeting

What are some common assessment methods you have heard of or used?

Our Assessment Toolbox • • • • Online surveys Paper-based surveys Focus groups National studies Historical document review Secondary data analysis Reflective journals One minute essays Photo analysis Process cartoons Expense categorization Pre and post tests Meeting Agenda Analysis Marketing/brand encounters Institutional data mining Social media analysis Environmental scans Student art opportunities Campus poster review Direct evidence (ID card scanning and data capture) • Staff initiated rubrics • Student self-scored rubrics • •

Quantitative Assessment Structures • Pre-determined, pre-set, pre-decided • Union staff in background to respondent – limited exposure • Focused primarily on numbers and uses deductive reasoning/logic • Frequencies (% who agree and by how much) • Descriptive Statistics (mean, mode, median) • T-tests (x vs. y – are they really different? ) • ANOVA (x vs. y vs. z…. ) • Regression (how much does x help predict y? ) • Higher Order Processes (HLM, Factor Analysis, etc. ).

Qualitative Assessment Structures • • Emergent Design – Organic in nature Union staff front and center with respondents Concerned with non-numeric data Best for understanding complex social challenges Theme counts and identification of patterns Inter-rater reliability Culture studies Pareto Graphs

What is a Pareto Graph?

Assessment Process Schuh and Upcraft outline a definitive process in their text Assessment in Student Affairs. Let’s Take a Stab at it!

Assessment from Problem to Report Step 1: Identify the problem Step 8: Determine who should collect the data Step 9: Determine how the data will be analyzed Step 2: Determine the purpose of the study Step 7: Determine what instruments will be used Step 10: Determine the implications of the study for policies and practice Step 3: Determine where to get the information needed Step 6: Determine how data will be collected Step 11: Report the results effectively Step 4: Determine the best assessment methods Step 5: Determine whom to study Get some rest for a day, then implement changes tomorrow! Source: Schuh, J. H. , Upcraft, M. L. & Associates. (2001). Assessment practice in student affairs: An applications manual. Jossey-Bass. San Francisco, CA.

Step 1: Identify the problem • What specific circumstances or situations are driving the need for assessment? • What external pressures are driving the need for assessment? • What internal circumstances are driving the need for assessment?

Step 2: Determine the purpose of the study • What information do we need to help solve the problems identified in Step 1? • Rely on this information to then develop the purpose of the study – then stick with it! • Resist the temptation to earn a Nobel Peace Price or find the cure to all of the woes at the Union – keep the activity focused.

Step 3: Determine where to get the information needed • Is the population students? What types of students? Be specific and focused. • What does the literature say? • Do you have previous Union reports, findings or datasets? • Do you have access to benchmarking studies of our peer institutions. • Would faculty, staff or alumni be helpful?

Step 4: Determine the best assessment methods • What is the best way to get the information I need? • Quantitative, Qualitative or Both Oh My! • How to decide? – “What” questions are best answered by Quantitative Research Methods – “Why” questions are best answered by Qualitative Research Methods

Step 5: Determine whom to study • Do we survey the entire student population? • Do we want to sample a segment of the student population? – Gender, age, race and ethnicity – Class standing, GPA, Full-Time, Part-Time – Commuter, resident, distance learner – Service user, student leader, student employee

Step 6: Determine how data will be collected • The method(s) selected MUST be consistent with the purpose of the study – Paper questionnaires – Telephone survey with trained interviewer – Web-based surveys – Individual interviews – Focus group interviews

SOME SPECIFIC UNION IDEAS • 1 minute essays at beginning of semester, middle of year, and end of year • Turn EBI results into a visual format via online video tools or in-house production • Conduct Union Cultural Assessment (resource provided) • Critical Impact Conversations with student employees who are graduating and have been at Union over multiple years • Student employee job description analysis • Door counters: Volume and peak time analysis • Recycling analysis by month and type • Work orders / facility tickets – Analysis of critical needs/patterns • Room registrations/booking – Patterns of use • Ticket distribution: Tracking by ID # and importing into Excel • Survey of Union Programming Board leaders, users, and contacted artists • Programming Board poster/marketing analysis with un-involved students

SOME MORE SPECIFIC UNION IDEAS • Lifespan analysis on all major building equipment components • User survey of all departments who rented space for events/gatherings • Direct observations of student patterns by employees (enough electrical plugs? , enough couches, etc. ) • Student Learning Outcome Analysis (programming, employment, etc. ) • Pre-Test/Post-Test of student employee competencies • GPA analysis of Union involved/employed students vs. uninvolved/unemployed • Retention analysis of Union involved/employed students vs. uninvolved/unemployed • Financial analysis by revenue generating area • Focus groups with building tenants • Focus groups with involved students • Art Gallery visitor counts by genre • Staff profile analysis vs. student profile • Student Collages – Why The Union Matters To Me

Collage Possibilities

SPECIFIC UNION IDEAS - TECHNOLOGY • • Photo contests – show us life in your Union Track Twitter followers and Union hash-tagged tweets Track Facebook Group members and “likes” Partner with academic computing to place important survey links on front page of student learning/registration portals Solicit student videos on how the Union has made a difference for them at your school Write an editorial in student paper and invite students to respond to staff Conduct on the spot / 1 minute polls in Union on laptops or i. Pads/Tablets Use staff to initiate blogs and allow for students to comment and provide feedback

Step 7: Determine what instruments will be used • The instrument must be able to yield results that can be statistically analyzed! • Do we want to use a test from a national source where validity and reliability is assured? • Do we want to develop our own instrument despite the fact validity and reliability will be a concern? • Qualitative – design open ended questions which are standardized, as well as any follow up questions or items of clarification

Step 8: Determine who should collect the data • Qualified and trained individuals are a good start for all assessment (i. e. not students per se) • Need to be cautious about bias – Can we really trust a study that was done exclusively by a person with a personal stake in the outcomes? – Can we really trust a study that was done exclusively be outside experts with no context or awareness of the nuances of the institution? – Bias can be reduced when outside experts in assessment review the process and methods used

Step 9: Determine how the data will be analyzed • Analysis of quantitative data depends on the purpose of the study: Are the respondents representative of the population? If yes, descriptive and differential statistics can then be applied in the quantitative arena (consult experts on ANOVA, MANOVA, Regression, ANCOVA, HLM or SEM if needed) • Themes, trends and variations can be explored in the qualitative study arena and a plan developed to record analyze this data (consult experts with backgrounds in Naturalistic Inquiry Methodology if needed)

Step 10: Determine the implications of the study for policy and practice • We suggest you rely on your findings and present how your findings impact the campus or program area. – What are the implications of this study? – What approaches to solving the problem should be considered in light of this study? – What policies and procedures need to be created or overhauled? – What is the call for action for the Union?

Step 11: Report the results effectively • Understand we must package the findings carefully so that it motivates change • How a study is distributed and formatted may be more important than the results found • Segment your audience and tailor specific reports to specific audiences • Make sure the results get into the right hands of people who can make a difference

Quick Primer on Executive Summaries Capstone process Enables critical reflection Not that long - standard length – 2 pages Not technical - should be written in jargonfree language • Has a TON of advantages (more on that in a minute) • Who authors? Who sees? • •

Suggested Executive Summary • First Section: Introduction and Problem Analyzed • Second Section: Methodology • Third Section: Results • Fourth Section: Discussion, Conclusions and Next Steps • Fifth Section: For More Information • Sixth Section: Appendices and Top 3

Why The Extra Effort? • Pull findings into annual report document for Union or Division • Pull document into Reaccreditation databases as a form of evidence to demonstrate effectiveness • Pull out findings for Union marketing (in all forms) • Pull out findings for budget hearings and grant applications • Use document to share with new hires • Use document to inform changes to strategic plan • If IRB approved, use document to create Conference Presentations (and as a handout) • If IRB approved, with adaptation, document may be suitable for publication • Historical document for future leadership and efforts • Document may be listed on staff resume as an institutional assessment document • Promotes Change….

Make A Change…No Really! revision of intended learning outcomes revision of measurement approaches changes in data collection methods changes in targets/standards changes in sampling techniques changes in facilitation techniques revision of delivery methods revision of program content/service components addition of programs/services deletion of program/services revision of marketing methods Changes to the revision of staffing patterns reallocation of fiscal resources for program/service Decision Making improvements to technology/technology tools changes in scheduling and timing of program/service suggested policy recommendations Changes to the Assessment Plan Changes to the Program/Service Process Adapted from University of Central Florida UCF Academic Program Assessment Handbook, February 2005, Information, Analysis, and Assessment

Creating a culture of evidence and how do know if you have it? 1. Clear (educational) goals in the Union 2. Common use of assessment-related terms in Union staff meetings/conversations 3. Ownership of assessment programs in Union 4. Ongoing professional development around Assessment for staff 5. Administrative encouragement of assessment both internally and externally 6. Assessment of overall institutional effectiveness and role Union plays in this 7. Informational forums about assessment as part of staff training 8. Inclusion of assessment in Union plans and budgets 9. Celebration of successes in Union Assessment 10. Responsiveness to proposals for new endeavors related to assessment 11. Practical assessment plans for Union areas 12. Systematic assessment across Union areas 13. Setting of student learning outcomes within an Union context 14. Comprehensive program review for Union 15. Assessment of co-curricular activities sponsored by Union Adapted from http: //www. aaup. org/AAUP/pubsres/academe/2009/JA/Feat/wein. htm

CASE STUDY TIME Get into groups of 8 -10 Assign a reader, recorder and reporter Read Case Study Add details, flush out ideas, get creative, inventing circumstances is OK within reason • Tackle Your Questions in 15 minutes • Prepare Your Comments in 5 minutes • Report Out When Called! • •

Your Questions 1. 2. 3. 4. 5. 6. 7. 8. What is the reason/need for assessment? What is the information you need? Who will you approach (be specific)? Go around an each group member provide one possible assessment strategy/idea to secure your data. List these and be creative. Who will collect the data? How does your group envision analyzing the data? Who needs to see your findings? Is there a particular order? If you list students, how will you share it with them? Final reaction to Case Study

CASE STUDY 1 Your campus President just confirmed the news you were waiting to hear. Your VP just shared that the Board of Trustees is open to pushing ahead on the construction of a new Union. Beyond the cost of the project, the Board is most interested in how students will respond to the news. Additionally, since students will be asked to pay a percentage of the costs the Board is concerned about the potential blowback of this announcement. You have been asked to deal with these concerns and have 10 weeks until the next Trustee meeting where your VP will make a presentation.

CASE STUDY 2 Your campus is less than 2 years out from re-accreditation. You know that if you don’t get re-accredited your campus won’t be able to get financial aid and award diplomas…a bit of a challenge. Your VP has just shared with you that your accrediting body is very interested in High Impact learning opportunities for students. On-campus student employment has been identified as a high impact practice. Your Union currently employs over 160 students in various capacities. If you were to partner with Dining, your student employee count would be over 300 in the facility. Your VP wants you to work on a way to show that student employees in the Union are indeed learning substantive skills that are consistent with a framework of student learning outcomes.

CASE STUDY 3 As an IPDS alum, you are now considered somewhat of a guru on assessment practice in your Union. Your area manages building reservations and marketing for rentals for the Union. After a tense budget meeting with your campus, you have been given the task of increasing your room rentals and community usage of the facility. You dig down into your historic usage reports and come to learn that your Union has seen a steady 3 -4% decrease in room rentals from the community in the past 5 years. You share your findings with your Union director. You are asked to turn this around. You know you need to first get to the bottom of the factors which might explain this trend and prepare a data-driven action plan.

CASE STUDY 4 You are entering your second year directing the Union Program Board. You have carefully observed the past year and generally held back from making massive changes. Over the summer, you meet with all of the new incoming officers. They share with you that the previous staff member who did your role in the Union for 8+ years created a culture where most acts were repeated every year and new artists were often discouraged. Doing due diligence, you go back and look through all of the previous contracts and event marketing and indeed find out that the students description was accurate. The students were hesitant to even bring it up in the first place. Recognizing that tradition can be a powerful force in higher education you know you need to figure out a way to use student opinions to change this trajectory.

CASE STUDY 5 You are called into a meeting with your Union Director. She shares confidentially with you that the campus advancement office has just successfully stewarded a $3 million dollar gift from an alumna who used to work in the Union. The alumna was a student leader and a student employee. She credits her Union experience as providing her with the foundational “soft-skills” she has relied upon so heavily during her professional career. She wants to direct her gift to the Union as an endowment but also wants to challenge the Union to use the funds to meet currently unfunded (non-scholarship) needs of students. Advancement has notified your supervisor that under the current spending policy (6% of assets) this will result in an annual fund of $180 K to be spent. Clearly this is a substantial pool of new funding, yet donor intent must be maintained. You also know as soon as the announcement is made by your supervisor everyone in the facility will be aggressively pitching ideas. You and your supervisor decided it is best to assess your student users first. Advancement wants to share back details with the donor and you get approval for 2 months to develop a proposal.

Concluding Thoughts

One final drawing: • Take just one minute to respond: – When I think of Assessment in a Union environment I believe… • Please draw a Union staff member conducting assessment activities: – What is this person doing? – What items are on this person’s desk? – If there were posters on the walls describing the work being done what would they say?

Making assessment personal: • Think about your role in your Union – how can you make a small step towards a growing assessment culture in your facility? • When you are thinking about establishing a new program, a change to an existing program or phasing out a program, what data will you draw upon to help you make your case? • In our busy lives, when is the last time you and your fellow staff had a conversation on your users/non-users in your program? How do these two groups of students reflect your larger student population? Is your data anecdotal or based in other evidence?

Additional References Classroom Assessment. Collaboration between Florida Center for Information Technology and University of South Florida. Retrieved September 25. 2008: http: //fcit. usf. edu/assessment/basicc. html.

Q&A