Effective Feedback to the Instructor from Online Homework

Award Winner: 
2004 Sloan-C Effective Practice Award
Author Information
Author(s): 
Gerd Kortemeyer, LON-CAPA
Institution(s) or Organization(s) Where EP Occurred: 
Michigan State University
Effective Practice Abstract/Summary
Abstract/Summary of Effective Practice: 

Michigan State University has been using LON-CAPA for both formative and summative assessment. Feedback from numerous sources has considerably improved instructor awareness of learner difficulties on the short, and the educational materials on the long run. Analysis mechanisms like the ones provided by LON-CAPA can facilitate research in physics education.

Supporting Information for this Effective Practice
Evidence of Effectiveness: 

The usefulness of these different data views is illustrated in following examples (all figures are available from http://www.lon-capa.org/sloancep/). * Course/Problem Overview: Fig. 1 (http://www.lon-capa.org/sloancep/fig1.gif) shows just a small excerpt of the homework performance in an introductory physics course, students in the rows, problems in the columns, each character representing one online homework problem for one student. A number shown is the number of attempts it took that particular student to get that particular problem correct ("*" means more than nine attempts), "." denotes an unsolved problem, blank an un-attempted problem. This view is particularly useful ahead of the problem deadline, where columns with a large number of dots or blank spaces indicate problems that the students have difficulties with. * Individual Student Responses: The symbols in Fig. 1 are clickable, and Fig. 2 (http://www.lon-capa.org/sloancep/fig2.gif) shows one particular problem for one particular student. It indicates that in the presence of a medium between the charges, the student was convinced that the force would increase, but also that this statement was the one he was most unsure about. His first answer was that the force would double; no additional feedback except "incorrect" was provided by the system. In his next attempt, he would change his answer on only this one statement (indicating that he was convinced of his other answers) to "four times the force" - however, only ten seconds passed between the attempts, showing that he was merely guessing by which factor the force increased. * Item Analysis: Fig. 3 (http://www.lon-capa.org/sloancep/fig3.gif) is a per-problem view, and on the left shows which statements were answered correctly course-wide on the first and on the second attempt, respectively, while the graphs on the right indicate which other options the students chose if the statement was answered incorrectly. Clearly, students have the most difficulty with the concept of how a medium acts between charges, with the absolute majority believing the force would increase, and about 20% of the students believe that the medium has no influence - this should be dealt with again in class. * Difficulty: Fig. 4 (http://www.lon-capa.org/sloancep/fig4.gif) shows a plot of the ratio of number of submissions to number of correct responses for 17 problems, from a weekly assignment five days before it was due (about 15% of the 400 students had submitted part or most of their assignment). It immediately becomes apparent that five of the questions are rather challenging, each requiring more than four submissions per success on average. Since this data is available before the deadline, it can be used to adapt lectures and recitations to meet learner needs. * Conceptual Analysis: LON-CAPA allows items to be grouped into "concept groups," and analyses to be performed on the base of such groups, rather than the individual items presented to the learners. Problem 14 from Fig. 4 (http://www.lon-capa.org/sloancep/fig4.gif) is shown in Fig. 5 (a target ="blank"a href="http://www.lon-capa.org/sloancep/fig5.gif">http://www.lon-capa.org/sloancep/fig5.gif). At the time of the analysis, it had five days before it was due.

 

The problem is highly randomizing: the choice of statements and the order in which they are presented varies among students, and so do the labels on the figure. Yet, internally, the statements are grouped into six so-called "concept groups," each focusing on a particular physics aspect of the problem. Every student gets one statement (with the correct labels filled in) from each one of these concept groups.

The item analysis on the result is done by concept group, not by statement, and can thus be carried out independently of the randomization. Success rate on each concept group for the initial submission is shown in Fig. 6 (http://www.lon-capa.org/sloancep/fig6.gif), and for later submissions in Fig. 7 (a target ="blank"a href="http://www.lon-capa.org/sloancep/fig7.gif">http://www.lon-capa.org/sloancep/fig7.gif).

In the version of the problem shown in Fig. 5 (http://www.lon-capa.org/sloancep/fig5.gif), the first question is to compare tension Tz to Ty. It is the most commonly missed statement, corresponding to concept '3' of Figs. 6 and 7. The incorrect answer given by over 90% of the students is that the two tensions are equal, which would be the answer for a pulley with negligible mass. As a result of the analysis, this error was subsequently addressed by discussion in lecture and by a demonstration showing the motion for a massive pulley with unequal masses, which quickly impacted the later response pattern.

How does this practice relate to pillars?: 

learning effectiveness:At Michigan State University instructors are provided with immediate detailed feedback on learner performance on online homework problems, which can be used to quickly adjust lectures, recitation sessions, and individual help to address learner needs. In particular, this feedback is an enabling tool for "Just-In-Time" (JiTT) teaching practices [1,2]. The amount of data gathered from large enrollment courses (in our case, 200-400 students with over 200 randomizing homework problems, each of them allowing multiple attempts), can be overwhelming, and is too diluted to be used unprocessed for effective feedback on learner understanding and their particular misconceptions. The LearningOnline Network with Computer-Assisted Personalized Approach (LON-CAPA [3,4]) allows instructors to quickly and efficiently analyze this data under a number of different aspects.

Estimate the probable costs associated with this practice: 

The LON-CAPA system is open-source freeware and can be downloaded from http://www.lon-capa.org/download.html The system needs at least one dedicated server machine, which, depending on anticipated userload, will cost between $3,000 and $18,000; see http://www.lon-capa.org/hardware.html for details.

References, supporting documents: 

[4] E. Kashy, S. J. Gaff, N. Pawley, W. L. Stretch, S. Wolfe, D. J. Morrissey, and Y. Tsai, "Conceptual Questions in Computer-Assisted Assignments," Am. J. Phys, 63(11), 1000 (1995)

Other Comments: 

We thank the National Science Foundation (ITR-0085921; CCLI-ASA-0243126) for the support of this project. Any opinions, findings, and conclusions or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. Support in earlier years was also received from the Alfred P Sloan and from the Andrew W Mellon Foundations. We are grateful to our own institution, Michigan State University, and to its administrators for over a decade of encouragement and support.

Contact(s) for this Effective Practice
Effective Practice Contact: 
Gerd Kortemeyer
Email this contact: 
korte@lon-capa.org