WE-H-BRC-09: Simulated Errors in Mock Radiotherapy Plans to Quantify the Effectiveness of the Physics Plan Review.

PURPOSE: A standard tool for ensuring the quality of radiation therapy treatments is the initial physics plan review. However, little is known about its performance in practice. The goal of this study is to measure the effectiveness of physics plan review by introducing simulated errors into "mock" treatment plans and measuring the performance of plan review by physicists.

METHODS: We generated six mock treatment plans containing multiple errors. These errors were based on incident learning system data both within the department and internationally (SAFRON). These errors were scored for severity and frequency. Those with the highest scores were included in the simulations (13 errors total). Observer bias was minimized using a multiple co-correlated distractor approach. Eight physicists reviewed these plans for errors, with each physicist reviewing, on average, 3/6 plans. The confidence interval for the proportion of errors detected was computed using the Wilson score interval.

RESULTS: Simulated errors were detected in 65% of reviews [51-75%] (95% confidence interval [CI] in brackets). The following error scenarios had the highest detection rates: incorrect isocenter in DRRs/CBCT (91% [73-98%]) and a planned dose different from the prescribed dose (100% [61-100%]). Errors with low detection rates involved incorrect field parameters in record and verify system (38%, [18-61%]) and incorrect isocenter localization in planning system (29% [8-64%]). Though pre-treatment QA failure was reliably identified (100%), less than 20% of participants reported the error that caused the failure.

CONCLUSION: This is one of the first quantitative studies of error detection. Although physics plan review is a key safety measure and can identify some errors with high fidelity, others errors are more challenging to detect. This data will guide future work on standardization and automation. Creating new checks or improving existing ones (i.e., via automation) will help in detecting those errors with low detection rates.