Using the American Board of Surgery In-Training Examination to predict board certification: a cautionary study.

OBJECTIVE: Although designed as a low-stakes formative examination, the American Board of Surgery In-Training Examination (ABSITE) is often used in high-stakes decisions such as promotion, remediation, and retention owing to its perceived ability to predict the outcome of board certification. Because of the discrepancy between intent and use, the ability of ABSITE scores to predict passing the American Board of Surgery certification examinations was analyzed.

METHODS: All first-time American Board of Surgery qualifying examination (QE) examinees between 2006 and 2012 were reviewed. Examinees' postgraduate year (PGY) 1 and PGY5 ABSITE standard scores were linked to QE scores and pass/fail outcomes (n = 6912 and 6846, respectively) as well as first-time certifying examination (CE) pass/fail results (n = 1329). Linear and logistic regression analyses were performed to evaluate the utility of ABSITE scores to predict board certification scores and pass/fail outcomes.

RESULTS: PGY1 ABSITE scores accounted for 22% of the variance in QE scores (p < 0.001). PGY5 scores were a slightly better predictor, accounting for 30% of QE score variance (p < 0.001). Analyses showed that selecting a PGY5 ABSITE score that maximized overall decision accuracy for predicting QE pass/fail outcomes (86% accuracy) resulted in 98% sensitivity, 13% specificity, a positive predictive value of 87%, and a negative predictive value of 57%. ABSITE scores were not predictive of success on the CE.

CONCLUSIONS: ABSITE scores are a useful predictor of QE scores and outcomes but do not predict passing the CE. Although scoring well on the ABSITE is highly predictive of QE success, using low ABSITE scores to predict QE failure results in frequent decision errors. Program directors and other evaluators should use additional sources of information when making high-stakes decisions about resident performance.