Video

Video by Michael Tate

Enhancing Online PME Assessment - Design-Based Research for Rubric Development and Evaluation in the Global College of PME

Air University Public Affairs

Dec. 8, 2023 | 01:00:01

Dr. Rob Nyland & Dr. Carl Watts

Rubrics play a crucial role in higher education as they ensure the use of transparent, consistent, and fair performance standards (Ragupathi & Lee, 2000). Particularly in the Global College of PME (GCPME) at Air University, an online learning institution that serves over 33,000 active duty, guard, research and civilian students, rubrics hold significant importance. With over 200 full-time and adjunct faculty involved in grading, the use of rubrics ensures consistency. In this presentation, we will discuss the ongoing evaluation studies conducted at the Global College of PME to test newly developed essay and discussion rubrics. This study serves as an exemplary case of iterative designbased research for enhancing educational products. The iterative pilots for these rubrics commenced in the Summer of 2022, utilizing a live comparison study between equivalent courses employing the rubrics and those that did not. During these tests, the evaluation was guided by several research questions: RQ1: How do student grades differ between the existing rubric and the new rubric? RQ2: How does the new rubric impact the student writing process? RQ3: How does the new rubric affect instructor feedback to students? RQ4: How does the new rubric influence interrater reliability? To answer these questions, we collected data from three primary sources: grades, instructors, and students. Instructor and student feedback were gathered through Qualtrics surveys administered at significant stages of the trial. We also conducted an inter-rater reliability exercise in which anonymized student papers were evaluated by two instructors using the new rubrics. The presentation will guide audience members through each phase of the pilot study, highlighting the iterative improvements made after each phase. Furthermore, the presentation will share valuable insights learned from this extensive evaluation project.