Passing a Technical Skills Examination in the First Year of Surgical Residency Can Predict Future Performance.

BACKGROUND: The ability of an assessment to predict performance would be of major benefit to residency programs, allowing for early identification of residents at risk.

OBJECTIVE: We sought to establish whether passing the Objective Structured Assessment of Technical Skills (OSATS) examination in postgraduate year 1 (PGY-1) predicts future performance.

METHODS: Between 2002 and 2012, 133 PGY-1 surgery residents at the University of Toronto (Toronto, Ontario, Canada) completed an 8-station, simulated OSATS examination as a component of training. With recently set passing scores, residents were assigned a pass/fail status using 3 standards setting methods (contrasting groups, borderline group, and borderline regression). Future in-training performance was compared between residents who had passed and those who failed the OSATS, using in-training evaluation reports from resident files. A Mann-Whitney U test compared performance among groups at PGY-2 and PGY-4 levels.

RESULTS: Residents who passed the OSATS examination outperformed those who failed, when compared during PGY-2 across all 3 standard setting methodologies ( P  < .05). During PGY-4, only the contrasting groups method showed a significant difference ( P  < .05).

CONCLUSIONS: We found that PGY-1 surgical resident pass/fail status on a technical skills examination was associated with future performance on in-training evaluation reports in later years. This provides validity evidence for the current PGY-1 pass/fail score, and suggests that this technical skills examination may be used to predict performance and to identify residents who require remediation.