Design-based research blends empirical research with the theory-based design of learning environments. It centers on the systematic investigation of innovations designed to improve educational practice through an iterative process of design, development, implementation and analysis in real-world settings (Wang & Hannafin, 2005). Design-based research helps us understand "how, when, and why educational innovations work in practice" (Design-based Research Collective, 2003, p. 5) because the innovations it explores are grounded in educational theory. The research reported in this paper did not start out to be design-based. It was originally conceived as a study exploring the effects of revising one online course based on the Quality Matters framework. Quality Matters is a faculty-oriented, peer review process designed to assure quality in online and blended courses (see: http://www.qmprogram.org/). The QM review process is centered on a rubric, originally developed through a FIPSE grant to MarylandOnline, but which is continually updated. The rubric is based on instructional design principles (Quality Matters, 2005). In the original study, the effects of a QM redesign on students' perception of learning processes were explored using the Community of Inquiry (CoI) survey (Garrison, Anderson, & Archer, 2000; Swan et al., 2008; see: http://communitiesofinquiry.com/). The researchers assumed that improved course design would lead to improved learning processes, and that improved learning processes would, in turn, lead to improved learning outcomes. Learning outcomes were measured by grades on two major assignments as well as overall course grades. Results revealed that learning outcomes indeed increased (although not significantly), but CoI scores actually decreased, confounding the original assumptions of the study, it thus became an ongoing design experiment centered on iterative course redesign to enhance student learning. Iterative course redesign involved an initial QM review/redesign followed by iterative design and implementations based on the results on the CoI survey and took place across four semesters. Figures 1 and 2 show the results of the preliminary design experiment. (Figure 1. Comparison of CoI scores over four semesters) (Figure 2. Comparison of learning outcomes across four semesters) Figures 1 & 2 show that changes made with respect to deficiencies identified by the CoI survey in the spring 2010 and summer 2010 semesters resulted in both higher CoI scores and increased learning outcomes. The data show that the combination of both the QM and CoI revisions across four semesters brought average scores on the research proposal from a 91 to a 97 and on the final exam from an 82 to a 90, while overall course grades went from a 90 to a 99. Moreover, analysis of variance reveals significant differences among final exam scores at the p=.05 level and for overall course grades at the p=.001 level. Post hoc analyses show that significant differences in learning outcomes did not appear until the fall 2010 semester, suggesting that the iterative approach to course redesign made a meaningful difference. Using Cohen's (1992) analysis of eta squared results, effect sizes of the cumulative QM/CoI revisions were small for the research proposal (.11) and the final exam (.16), but moderate in terms of the overall course grades (.29). The linking of online course design and implementation to learning outcomes is long overdue in online education. This ongoing study is not only a first step in that direction but it employs what are probably the two most commonly used theoretical frameworks in online education in the process. Findings suggest that, taken together, QM and CoI revisions can be linked to improved outcomes, but unfortunately not to each other. However, they do suggest a trajectory - QM review and revision of courses and incremental "tweaking" of course implementation relative to deficiencies revealed by the CoI survey - for incremental improvement of online courses. We are beginning to explore the efficacy of such an approach at the program level and will report on ongoing findings at the November conference. Demonstrating a link between the QM framework and student outcomes, and the CoI framework and student outcomes has great theoretical merit, especially as regards the latter and the cognitive presence construct in particular. Moreover, the efficacy of our efforts demonstrates the usefulness of design-based approaches to research on online learning. Participants at this session will learn about the Quality Matters and Community of Inquiry frameworks and how to use them to iteratively improve online courses. Participants will also explore how they might use a general design-based research approach to improve their own courses and programs. The presentation will be structured as follows, with plenty of time given to questions during and after it: Initial Study MTL Program & EDL 541 Quality Matters Community of Inquiry Initial Findings Implications Design Experiment Results Ongoing Study Implications for Practice Discussion References Design-Based Research Collective. (2003). Design-based research: An emerging paradigm for educational inquiry. Educational Researcher, 32(1), 5-8. Garrison, D. R., Anderson, T., & Archer, W. (2000). Critical inquiry in a text-based environment: Computer conferencing in higher education. The Internet and Higher Education 2, 87-105. Quality Matters. (2005). Research literature and standards sets support for Quality Matters review standards. Retrieved July 14, 2010 from http://www.qualitymatters.org/Documents/Matrix%20of%20Research%20Standards%20FY0506.pdf. Swan, K. P., Richardson, J. C., Ice, P., Garrison, D. R., Cleveland-Innes, M. & Arbaugh, J. B. (2008). Validating a measurement tool of presence in online communities of inquiry. e-mentor, 2 (24) at http://www.e-mentor.edu.pl/artykul_v2.php?numer=24&id=543 Wang, F., & Hannafin, M. J. (2005). Design-based research and technology enhanced learning environments. Educational Technology Research and Development, 53(4), 5-23.
