Evaluation in health professions education-Is measuring outcomes enough?

INTRODUCTION: In an effort to increase the rigour of evaluation in health professions education (HPE), a range of evaluation approaches are used. These largely focus on outcome evaluation as opposed to programme evaluation. We aim to review and critique the use of outcome evaluation models, using the Kirkpatrick Model as an example given its wide acceptance and use, and advocate for the use of programme evaluation models that help us understand how and why outcomes are occurring.

METHODS: We systematically searched OVID medline, Scopus, CINAHL and Pubmed, and hand searched six leading HPE journals to provide an overview of the use of the Kirkpatrick Model as well as a range of programme evaluation models in HPE. In addition to this, we synthesised the existing critiques of the Kirkpatrick Model as an example of outcome evaluation, to highlight the limitations of such models.

RESULTS: The use of the Kirkpatrick Model in HPE is widespread and increasing; however, studies focus on categorising outcomes, rather than explaining how and why they occur. The main criticisms of the model are as follows: it is outcomes focused and fails to consider factors that can impact training outcomes; it assumes positive casual linkages between the levels; there is an assumption that the higher-level outcomes are more important; and unintended impacts are not considered. The use of the Kirkpatrick Model by the MERSQI, BEME and WHO contribute to the myth that the Kirkpatrick Model is the gold standard for programme evaluation.

DISCUSSION: Moving forward, evaluations of HPE interventions must shift from focusing largely on measuring outcomes of interventions with little consideration for how and why these outcomes are occurring to programme evaluation that investigates what contributes to these outcomes. Other models that facilitate the evaluation of the complex processes that occur in HPE should be used instead of Kirkpatrick's.