The Performance Improvement (Pi) score: an algorithm to objectively score performance improvement during E-BLUS hands on training (HoT) sessions. An ESUT project.

INTRODUCTION AND OBJECTIVES: Assessing performance improvement (Pi) is one of the most important roles of a tutor, but is usually based on subjective observation, personal judgement and expertise. Our study is aimed to evaluate the variability of subjective tutor Pi assessment and to compare it to a novel measurement algorithm: 'the Pi-score' MATERIALS AND METHODS: The Pi-score algorithm considers time measurement and number of errors from two different repetitions (first and fifth) of the same training task and compares them to the relative task goals, to produce an objective score. We collected data during 8 courses on the four basic laparoscopic urological skills (E-BLUS) tasks. The same tutor instructed on all courses. Collected data were independently analysed by 14 hands-on-training (HoT) experts for Pi assessment. Their subjective Pi assessments were compared for inter-rater reliability. The average per-participant subjective scores from all 14 proctors were then compared to objective Pi-score algorithm results. Cohen's Kappa Statistics was used for comparison analysis.

RESULTS: 50 participants were enrolled. Concordance found between the 14 proctors' scores was the following: Task1=0.42 (moderate); Task2=0.27 (fair); Task3=0.32 (fair); Task4 (Kappa=0.55, moderate). Concordance between Pi-score results and proctor average scores per participant was the following: Task1=0.85 ("almost perfect"); Task2=0.46 ("moderate"); Task3=0.92 ("almost perfect"); Task4=0.65 ("substantial").

CONCLUSION: Our study demonstrated that evaluation of Performance improvement is highly variable, even when formulated by a cohort of experts. Our algorithm successfully provided an objective score that equals the average Pi assessment of a cohort of experts, in relation to a small amount of training attempts. This article is protected by copyright. All rights reserved.