"Development and Pilot Testing of an Observational Instrument to Evaluate the Integration of Cognitive Load Theory and Cognitive Theory of Multimedia Learning in First-Year Medical Lectures"

Abstract

Lectures are common educational activities used in health profession education; however, they are often criticised for low learner engagement, satisfaction and learning experiences. Cognitive Load Theory (CLT) and Cognitive Theory of Multimedia Learning (CTML) can address these deficiencies. Currently, no tools exist to evaluate the extent of use of CLT and CTML as theoretical frameworks in lecture design. This research developed and validated a rubric to quantify the integration of CLT and CTML in lecture design. Two research questions guided the study: (1) What is the validity evidence according to the Messick framework for the use of a novel rubric as an assessment tool for Cognitive Load Theory and Cognitive Theory of Multimedia Learning Principles in respiratory lectures at the University of Sydney, Australia? (2) To what extent do anatomy and pathophysiology medical school lectures from the respiratory module incorporate principles of Cognitive Load Theory and Cognitive Theory of Multimedia Learning in their design and delivery? The project was conducted in two phases. Phase 1 focused on rubric development through expert input and iterative refinement. Fifteen assessors scored a lecture on histology of upper airways, yielding an intraclass coefficient (ICC) score of 0.59 (moderate inter-rater reliability (IRR)). Semi-constructed interviews gathered qualitative data, refining the rubric into a 12-item version. A second lecture on microbiology of pneumonia, scored by thirteen assessors achieved an ICC of 0.78 (good IRR), confirming no further revisions were needed. Phase 2 tested the rubric on three additional lectures covering anatomy of the upper and lower respiratory airways and pathophysiology of tuberculosis. This phase assessed the extent that CLT and CTML principles were integrated into lecture design and the consistency of IRR, yielding ICC scores of 0.99, 0.99 and 0.99 (excellent IRR). Principles 1-6, 10 and 11 associated with presentation skills, such as use of voice, visuals, gestures, and content organisation were highly used (92%-100%). Principles 7-9 and 12, associated with cognitive engagement and visual integration, were less frequently observed (0%- 62%). These results validate the rubric for assessing CLT and CTML integration in lecture design, helping medical educators optimise teaching strategies and improve educational experiences.