Goals/Objectives • Describe the most common cognitive interviewing techniques (think-alouds, probing, and observation) and discuss implementation of each within various online education and research contexts. • Identify potential response errors related to an instrument design and explain how cognitive interviewing techniques can help overcome the response errors that may threaten validity and reliability of the items. • Assess the quality of questions developed for online assessments and survey questionnaires using the Question Appraisal System (QAS-99). • Perform analysis of problematic questions in reference to wording, syntax, and semantics using a computer tool called QUAID. • Debate the advantages and disadvantages of cognitive interviewing for different research and instructional situations in the online education arena. Research on online courses is increasing, but limited attention is given to the appropriateness and quality of instruments used in research studies. Developing focused and effective data collection tools helps online educators and researchers make better-informed decisions about the data collected during survey or assessment administration. Of the research methods used in assessment of survey and test or discussion questions, cognitive interviewing holds the greatest potential for capturing the thought processes of respondents as they answer questions. Cognitive interviewing is an approach for evaluating sources of response error in survey and assessment instruments during the question/item design process. The goal of this approach is to maximize content validity, a critical step in instrument development. Using the cognitive interview procedures of thinking aloud, verbal probing, concurrent probing, and retrospective probing, the researcher can unveil important information about how participants interpret questions. The quality of data collected via a survey or assessment instrument rests on the notion that the meaning of each question, as written by the question designer, is consistent with respondents' interpretations. Since the intended meaning of a question or term is not always the meaning that respondents recognize, cognitive interviews can be used to identify those potential sources of miscommunication before the instrument is fielded. Online educators as researchers and test developers must strive to reduce survey and test question errors so that they can conduct high-quality research as well as develop assessments that measure what they intend to measure. This workshop is designed for anyone seeking to increase awareness of methods currently available for testing questions, including surveys, online discussion questions, and online test items. Participants will learn to evaluate questions using the Question Appraisal System (QAS-99), an 8-step procedure developed to check for poorly communicated or problematic questions prior to field testing. They will also learn to use cognitive interviewing techniques to further strengthen the quality of instruments and assessments for e-learning research. Participants will also explore ways to apply cognitive interviewing techniques to a variety of e-learning settings as well as discuss the advantages and disadvantages of conducting cognitive interviews with interviewees located at a distance. In addition, a computer tool called QUAID will be introduced to improve the wording, syntax, and semantics of questions on surveys and other question items. Workshop Agenda, Topics/Activities (3 hours) Step 1 (5 minutes) Welcome and Introduction Step 2 (10 minutes) Participant Profiles Step 3 (15 minutes) Small Group Activity 1, Part I: Warm-Up Discussion. - Tell us about your day-to-day activity and what brings you to the workshop today. Please share your responses to the following questions: - Why do we use surveys in e-learning research? - What do we attempt to measure with surveys or other assessment items in online learning? - What makes a "quality" survey question? - What do we know about question evaluation methods for online teaching and learning? - How might you define "cognitive interviewing" at this early point in the workshop? Step 4 (20 minutes) Didactic Slide Presentation on Cognitive Interview Techniques Step 5 (30 minutes) Small Group Activity 1 Part IIa: Sources of Errors in Questions/Instruments. - Review the sample surveys provided (Faculty and Student Survey Questionnaires). - Select two questions from those surveys that have a clarity problem such as wording, technical terms, vagueness, and reference periods (question evaluation purpose). - Write those two questions on the form provided and evaluate them using the QAS-99 process. Small Group Activity 1 Part II b. Using the interview forms provided, practice cognitive interviews with selected members from your group. During this process: - Identify questions that are difficult or confusing to your respondents. - Identify terminology that is unclear to your respondents, and - Assess whether your respondents interpreted questions as intended. Remember to take notes during the process to share your insights and reflections with the larger group. Step 6 (10 minutes) Large Group Sharing and Discussion. Each group will share their work with the larger group followed by a general discussion of the key elements of cognitive interview techniques including possible challenges and problems. Step 7 (10 minutes) Break Step 8 (25 minutes). Didactic Slide Presentation on five principles of memory retrieval, exploring the types of validity associated with surveys, and discussing ways to assess different types of validity. Introduce the QUAID system with demonstration to show participants potential problems with the question, including: (1) unfamiliar technical term, (2) vague or imprecise relative term, (3) vague or ambiguous noun-phrase, (4) complex syntax, and (5) working memory overload. Step 9 (45 minutes) Small Group Activity 2 Part 1: The Case Scenario (15 minutes): Read the case scenario and develop up to 3 questions you would ask recent graduates about their satisfaction with the online program and instructors. Part 2: Practice Cognitive Interview and Reflections (15 minutes): Please practice the cognitive interview process using the questions you developed during Step 1 of this activity. Record your experiences to share with the larger group. Part 3: Large Group Presentation and Discussion (15 minutes): Each group will report their cognitive interview findings to the larger group and share their reflections. Facilitators will provide clarification based on group reflections. Discussion will include advantages and disadvantages of changing an existing measure and the possibility of alternative study designs. Step 10 (10 minutes) End of session. Wrap-Up Workshop Evaluation
