Making the Grade: Evaluating Online Program Quality

Ever since the very first distance education course was offered, the industry has deliberated how to measure the quality of online programs. Two years ago, the Sloan Consortium (Sloan-C) set out to create an industry standard for quality measurement with the Quality Scorecard for the Administration of Online Education Programs.

The concept of the Quality Scorecard was originally conceived through a six-round Delphi study, led by Kaye Shelton, Associate Professor, Educational Leadership at the Center for Doctoral Studies at Lamar University.

“At the time, there wasn’t much out there to measure quality of online programs from an administrator standpoint,” said Dr. Shelton.

 Since being introduced, institutions of all types have embraced the Quality Scorecard for the purposes of quality assurance. More than 100 institutions have utilized the interactive tool on the Sloan-C website to create a self-study of their program that includes justifications for each score, with the option to upload supporting documentation. Other institutions are using the Scorecard for evaluation, but choose to display their Scorecard and supporting documentation on their own websites.  

The Quality Scorecard utilizes 70 weighted indicators to measure quality across nine categories: institutional support; course development and instructional design; course structure; teaching and learning; social and student engagement; faculty support; student support; and evaluation and assessment.

Institutions utilize the results of the Quality Scorecard to develop strategic objectives for program improvement, for benchmarking, program planning and self-study. In addition, some institutions use the results to demonstrate to accrediting bodies the breadth and quality of their programs.

Becoming Sloan-C Certified

As more institutions adopted the Quality Scorecard, they began asking if Sloan-C could certify their score. Sloan-C now offers a Quality Scorecard Review process and certification for member institutions.

Southern Nazarene University is among the universities that have already undertaken the Quality Scorecard Review process and certification. The University opted into the Quality Scorecard Review process to ensure it was meeting national standards when developing its first online programs.

Gwen Rodgers, Director, Online Resource Center at Southern Nazarene University, said the review process “enabled us to look at our own programs through a different lens. The reviewers gave us detailed feedback that highlighted areas where we could improve and validated what we were doing well.”

Since going through the review process and receiving exemplary status and certification, Gwen Rodgers said, “It has increased our credibility both internally and externally and we’ve received great feedback from accreditors. I highly recommend the Quality Scorecard Review process to any institution that offers online programs.”

How it Works

After an institution self-scores with the scorecard, a program administrator develops justifications and submits substantial artifacts that are placed inside the Sloan-C online repository. Trained scorecard reviewers then evaluate each scorecard and artifact individually, and then come together as a group and discuss their assessment and determine a score for the institution.

Programs that score 189 points or higher after the official review are awarded a Sloan-C Exemplary Program logo that may be displayed on their website. Programs that score less than 189 points receive a report on areas for improvement.

As part of an introductory three-month promotion, Sloan-C will provide an official Scorecard Review by three trained evaluators for member institutions for $5,000. Nonmember institutions have the opportunity to become Sloan-C members and take advantage of the introductory offer.

Dr. Shelton will be discussing how institutions can implement the Scorecard in a pre-conference workshop on November 20 at the 19th Annual Sloan Consortium International Conference on Online Learning, in Orlando, Florida.

Assessing International Online Education Quality

Sloan-C has partnered with the Inter-American Distance Education Consortium (CREAD), the premier Latin American organization dedicated to improving educational quality in distance and online education, to develop a Spanish version of the Quality Scorecard for Latin America. Together, Sloan-C and CREAD reviewed their individual scorecards and then combined the two products into a co-branded scorecard with 91 indicators.

Sloan-C and CREAD will begin introducing the Quality Scorecard through a workshop at the VIII CREAD Caribbean Conference, in early October, in the Dominican Republic.

Dr. Armando Villarroel, Executive Director of CREAD, explained that in Latin America students often must overcome significant obstacles in order to access online education programs. Many have to travel to use a computer. Others must deal with a limited supply of electricity and may live in an area with little to no access to broadband. Despite these conditions, now is an opportune time for Latin America to embrace online learning.

“In a knowledge-based society, knowledge is power,” said Dr. Villarroel. “If the countries that are lagging behind are not able to catch up, the gap is going to become greater than it is already.” 

In addition, Dr. Villarroel notes that Global Universities and MOOCs are already reaching students in Latin American countries. Therefore, it is imperative for Latin American countries to embrace online learning and adopt industry standards.

Sloan-C and CREAD are working with Latin American governments to help develop online program standards and hope the Quality Scorecard will become the standard for measuring online quality in Latin America.

In addition to Latin America, Kathleen Ives, Acting Chief Executive Office and Executive Director of the Sloan Consortium, says that Sloan-C eventually plans to develop Quality Scorecards for Europe, Saudi Arabia and Australia.