Quality Matters Overview October 14 2008 Deborah Adair

Quality Matters Overview October 14, 2008 Deborah Adair, Ph. D. Director, Quality Matters

“Quality Matters: Inter-Institutional Quality Assurance in Online Learning” § Quality Matters is a not-for-profit subscription service providing tools and training for quality assurance of online courses § Colleges and universities, both public and private, throughout the country are joining our community of practitioners www. qualitymatters. org

QM Process Provides: § Institutional toolset and process to meet quality expectations: § § § Online course design Student learning Improved instruction Assessment and feedback loops Professional development

QM as a National Standard § 270 + current subscribers (http: //www. qualitymatters. org/Documents/Subscriber%20 List%20 for%20 Publication. pdf) § § § 42 states represented QM has trained 4000+ faculty and instructional design staff Recognized by - Sloan C Excellence in Online Teaching and Learning Award -2008 USDLA Outstanding Leadership in the field of Distance Learning

QM Subscribers WASHINGTON MONTANA NORTH MAINE MINNESOTA DAKOTA OREGON VT NH WISCONSIN SOUTH DAKOTA IDAHO MICHIGAN WYOMING NEBRASKA NEVADA RI CONN PENN IOWA OHIO UTAH ILLINOIS MARYLAND KANSAS VIRGINIA MISSOURI KENTUCKY CALIFORNIA NORTH CAROLINA TENNESSEE ARIZONA OKLAHOMA NEW JERSEY DELAWARE INDIANA WV COLORADO MASS NEW YORK ARKANSAS SOUTH CAROLINA NEW MEXICO MISS ALABAMA GEORGIA TEXAS LOUISIANA FLORIDA PUERTO RICO Current Subscribers Statewide Subscribers ALASKA HAWAII VIRGIN ISLANDS

Principles of QM The QM toolset and process § A faculty-driven, peer review process that is… Collaborative Collegial Continuous Centered - in academic foundation - around student learning § Courses do not have to be “perfect” but QM aims at better than just “good enough”

What Quality Matters is NOT § Not about an individual instructor (it’s about the course design) § Not about faculty evaluation (it’s about course quality) § Not a win/lose or pass/fail test (it’s a diagnostic tool to facilitate continuous improvement of online/hybrid courses)

Goal: Make online instruction as good as it can be § § Better than average; more than “good enough” An attempt to capture what’s expected in an effective online or hybrid course at about an 85% level 85 %

History of Quality Matters Winter 2008

Quality Matters Was Launched by Maryland. Online § Maryland consortium for distance learning received 3 -year FIPSE grant in 2003 § Motivation: Need to provide inter-institutional quality assurance for courses in consortium seat bank § Approach: Research-based, collaborative, peercentered § Outcome: Sustainable quality assurance process embraced by institutions beyond MOL and Maryland

Success of the QM Grant § Early presentations generated widespread interest § Maryland. Online began to receive recognition for QM § § § WCET Outstanding Work (WOW) Award, 2005 USDLA 21 st Century Best Practice Award, 2005 Maryland Distance Learning Association (MDLA) Best Program Award, 2005. § The Sloan Consortium online workshops introduced hundreds of faculty members and staff to QM. § Peer reviewer training spread far beyond Maryland: § § 700+ faculty trained to review online courses using the rubric individuals from 158 different institutions in 28 states

QM Today § Self-supporting non-profit program § § Wide-spread adoption § § Independent subscribers (75), consortium subscribers (67), and statewide system affiliates (136) Program development through collaboration § § subscriptions, trainings, and course reviews User community directs program improvement Continuous improvement of tools and services

QM Basics: More than the Sum of its (3) Parts Winter 2008

The Rubric

The Rubric § Eight standards: § § § § Course Overview and Introduction Learning Objectives Assessment and Measurement Resources and Materials Learner Interaction Course Technology Learner Support Accessibility Key components must align.

Rubric Scoring § Points are awarded for 40 specific standards based on: § § the team majority, AND the pre-assigned weighting of each standard § Specific standards have a point value of 1, 2, or 3; the total points possible in a review is 85 § If 2 -3 Reviewers believe that a standard is: § § met, then the full pre-assigned points are awarded not met, then zero points are awarded

To Meet Expectations… A course must achieve: § “Yes” on all 17 of the 3 -point “essential” standards § A minimum of 72 out of 85 points 72/85 = 85%

The Peer Review Process

Quality Matters: Peer Course Review Process Institutions Faculty Course Developers Course continues to be offered Course Meets Quality Expectations Course Revision National Standards & Research Literature Rubric Faculty Reviewers Training Peer Course Review Feedback Instructional Designers

The Peer Reviewer § Characteristics: § Experienced online/hybrid instructor § Trained in applying the Quality Matters Standards and certified by QM § Training updated with each new edition of the rubric § Receives a small stipend for each review

The Peer Review Team § Consists of: § 3 QM Certified peer reviewers § The chair is a Master Reviewer (having received additional training) § One reviewer must be a subject matter expert § At least one reviewer must external to the institution sponsoring the course

What to Expect from Course Reviews § Consistency and Rigor § Professionalism and Commitment § Useful and Constructive Feedback § Positive Outcomes for Faculty

Survey of faculty whose course was reviewed indicates that … § 91% of respondents made changes in the course as a result of the review § 89% of respondents felt that the quality of course design improved as a result of the review Sample comments: “I was too close to see what could be improved. ” “Provides a great way to get an objective view of your course. ” “It made all of my online courses better. ” “It provides a view from a more student oriented perspective. ” “Many elements that might contribute to a student withdrawing can be eliminated. ”

Quality Matters Training

QM Training § QM trainings support the other program components and are integral to a quality assurance effort § QM principles and the rubric standards are at the heart of all the trainings § There are two general categories of training: Faculty Development and Implementing Quality Matters

Types of Trainings Faculty Development Implementing Quality matters § § Applying the QM Rubric Building Your Online Course § Using the QM Rubric § § Improving Your Online Course § Using the QM Rubric § § § Design that Welcomes Your Students Measurable Learning Objectives at the Course and Module Level Choosing and Using Media Effectively § Peer Reviewer Certification Master Reviewer Train-the-Trainer Institution Representative/ Course Review Manager

Why Quality Matters? Winter 2008

Multiple Uses of QM Reported Uses of QM Tools: § Guidelines for initial online course development § Quality assurance of existing courses § Ongoing faculty professional development § Institutional re-accreditation packages § Focus attention on distance learning policies & steering committees

Alignment with Accrediting Best Practices Principles* Quality Matters Principles That education is best experienced within a community of learning where competent professionals are actively and cooperatively involved with creating, providing, and improving the instructional program; QM is a peer review process involving faculty, instructional designers and other support staff in a cooperative effort to continuously improve online instruction. That learning is dynamic and interactive, regardless of the setting in which it occurs; QM treats interactivity and active learning a critical component of every online course. That instructional programs leading to degrees having integrity are organized around substantive and coherent curricula which define expected learning outcomes; QM treats the alignment of expected learning outcomes with the contents, activities and assessments as a critical element in every online course. That institutions accept the obligation to address student needs related to, and to provide the resources necessary for, their academic success; QM expects every online course to address student access to the academic, technical, and student support services essential to student success. That institutions are responsible for the education provided in their name; Adoption of QM standards reflects institutional commitment to online instructional quality, wherever an institution has endorsed the rubric standards. That institutions undertake the assessment and improvement of their quality, giving particular emphasis to student learning; The QM standards are based on research and best practices to enhance student learning in online environments. Adoption of the QM review process is a clear demonstration of institutional or programmatic commitment to assessment and continuous improvement. That institutions voluntarily subject themselves to peer review. QM is essentially a peer review process involving both internal and external peers in the evaluation of courses. *“Best Practices for Electronically Offered Degree and Certificate Programs” adopted in 2001 by CHEA and 8 regional accreditation bodies.

Collaboration and Community in Distance Education § QM provides platforms to help users collaborate. . . § § § With QM in building tools that work for Distance Education With their own institutional users to adapt QM tools to their specific institutional needs Across the community of QM subscribers to share best practices, find new resources, and support collegial efforts to improve student learning outcomes www. qminstitute. org

What’s In It For Faculty? § Improvement of online courses § External quality assurance § Review other courses and gain new ideas for own course; expand professional community § Participation useful for professional development plan and portfolio § Faculty development trainings

Quality Matters Implementation Winter 2008

QM’s Role in Quality Assurance § QM looks at course design The harnessing of technology to deliver instruction and promote student learning § QM provides a process for peer-to-peer feedback for faculty in the continuous improvement of their course § Quality Matters is not the complete answer to quality assurance for online education, but it can be a critical component

Campus Decisions for Quality § Governance § § Engagement § § § For course assessment, development, maintenance? Course Reviews § § How will you gain faculty commitment? Communication, training opportunities, incentives…? Rubric Use § § Who will lead a QM project and where will it report? Mandatory or optional, official or informal? Selection Criteria § Which courses in what order?

Types of Course Reviews § QM-Managed Course Reviews § § § Official, Subscriber-Managed Course Reviews § § QM staff assigns review team and manages process Available with a Supplemental Package, otherwise on an as-available basis Trained institution representative manages according to QM criteria Independent Course Reviews § Schools set up own informal process

Scalability in Implementing QM Managing Your Reviews Delivering Your Training (APPQMR) Customized (not recognized by QM) • Internal QA • No QM requirem ents • No QM fees Customized (not recognized by QM) • Adapt for internal needs • For internal use only • No QM fees QMManaged Subscriber. Managed • External QA • Courses must meet submission guidelines • Fee for Service • External QA • Review follows official QM process • Fee for certification QMManaged Subscriber. Managed • Required for peer reviewer & other QM roles • Fee for Service • required for peer reviewer & other QM roles; Trainer must be certified • Fee for certification

Implementing QM – Roles and Criteria Peer Reviewers Independent Trainers Master Reviewers Course Review Manager Institution Representative Applying the QM Rubric Training Certified QM Peer Reviewer Applying the QM Rubric Training Designated by institution Peer Reviewer Certification (online) Train-the Trainer Training Experience on 2+ course reviews Institution Representative Training Current online instructor Institution on Full Subscription Master Reviewer Training Institution on Full Subscription

Preparing for Course Reviews § Course is mature § Learning outcomes (course & unit) are specified § Review team has access to all elements students do § Use of QM rubric in course development § Faculty familiar with QM rubric

Helpful Links § Current Subscriber List § § Courses Recognized in 2008 § § http: //www. qualitymatters. org/Training. htm Fee Schedule § § http: //www. qualitymatters. org/Courses 2008. htm Training Calendar § § http: //www. qualitymatters. org/Documents/Subscriber List for Publication. pdf http: //qminstitute. org/home/Public%20 Library/Fee%20 Schedule/Quality%20 Matters%20 Fe e%20 Schedule%20 effective%208_1_2008. pdf QM Staff § http: //www. qualitymatters. org/Contact. htm

Thanks to YOU… Quality Matters! Winter 2008

2008 Post Review Interviews: The peer reviewers fulfilled their responsibilities on the review team.

2008 Post Review Interviews: The review followed the official QM course review process:

2008 Post Review Interviews: The feedback provided by the peer reviewers in the final report was constructive and useful: