Online courses are taught increasingly by instructors who did NOT create the content, discussions, or assignments of the course. This one course-many instructors model has several advantages (such as curricular alignment and consistency), but also introduces new challenges for instructors to deliver the courses. In particular, instructors may not hold the same performance expectations as the faculty that created the course. These differences in performance expectations can undermine the effectiveness of the program’s curriculum as learners receive inconsistent feedback on their performance. Perceived differences in performance expectations may also undermine faculty sharing of effective teaching practices or discussion of instructional challenges. Unfortunately, few universities address the role of performance expectations explicitly, either through role responsibilities, committee charters, policy, or procedures.
Moderation sessions are a method for improving faculty-based learning outcome assessments by promoting shared performance expectations among faculty. Operationally, a moderation session involves faculty individually assessing a student’s work using a scoring guide and then as a group discussing points of consensus and disagreement. A facilitator asks questions to identify the basis for assessment judgments and to articulate shared performance expectations amongst the faculty. Facilitators may also prompt analysis of the scoring guide language and its concordance with faculty’s performance expectations. Based on findings from a moderation session, faculty may decide to revise course assignments or scoring guides to improve alignment amongst the program’s curriculum, instruction, and assessment practices. Inter-rater reliability statistics can also be calculated from these sessions as part of an evaluation of a program’s learning outcome assessment system.
Procedure
Setting up a moderation session: Clear and early communication is key to a successful moderation session. Initial communications about the moderation session should describe your goals, timeline, participation expectations, and the value of high-quality assessments for students' success. Make sure faculty know why they were selected to participate in the session and how their participation is key to the success of the session. Give faculty at least one week to complete the assessment before the scheduled moderation session. Also, ensure that the contact details for the conference call and any web conferencing environments are contained in the meeting invite.
Select an assignment and scoring guide for the moderation session that is relevant and appropriate for many program faculty to assess, such as a course early in the program or a final capstone course. Embedded assignments that directly measure program outcomes are excellent choices for moderation sessions.
You may intentionally select a student’s work to focus the moderation session on particular performance expectations or randomly select a student’s work. In either case, avoid revealing your selection criteria to minimize unintentionally biasing the faculty’s judgments. Make sure to remove all identifying information from the student’s work, including the student’s name, the document’s properties, comments, and any other identifying information.
If using a web conferencing environment, set up a poll question for each scoring guide criterion before the moderation session. The response options should be the performance levels. Make sure each poll is set to hide participant’s responses. The facilitator will reveal the poll results once the polling is complete.
Facilitating a moderation session: Begin the session by welcoming the faculty and reinforcing the goals for the session. Facilitators have found it helpful to describe at a high-level the agenda for the session. An effective practice is to have faculty answer each of the poll questions at the beginning of the meeting so the facilitator can review the poll results and decide which criterion to discuss first. To ensure the discussion is focused effectively on building shared performance expectations, it is recommended to discuss the most discordant criteria first as opposed to moving sequentially through the criteria as listed on the scoring guide. Once poll results are revealed, the facilitator uses observations and questions to identify the basis for faculty judgments, such as “What evidence is there in the student’s work suggesting a ‘distinguished’ level of performance on this criterion?” After working through a few criteria, the facilitator can often simply verbally state the results of the poll question to stimulate appropriate discussion. You may also consider collecting evaluation data at the end of the moderation session using poll questions.
After a moderation session: Thank faculty for participating in the moderation session and seek opportunities to recognize their service through inter-department and university communications. If the web conferencing environment saves poll results, inter-rater reliability statistics can be calculated and communicated to participants for monitoring the quality of outcome assessments over time.
