The Needs and Benefits of Continuous Model Updates on the Accuracy of RT-induced Toxicity Prediction Models within a Learning Health System.

BACKGROUND: and Purpose: Clinical data collection and development of outcome prediction models by machine learning can form the foundation for a learning health system offering precision radiotherapy. However, changes in clinical practice over time can affect the measures and outcomes of patients and hence the collected data. We hypothesize that regular prediction model updates along with continuous prospective data collection is important to prevent the degradation of a model's predication accuracy.

MATERIAL AND METHODS: Clinical and dosimetric data from head and neck patients receiving intensity-modulated radiation therapy from 2008 to 2015 were prospectively collected as a routine clinical workflow and anonymized for the analysis. Prediction models for grade≥2 xerostomia at 3 to 6 month follow-up were developed by bivariate logistic regression using DVH of parotid and submandibular glands. A baseline prediction model was developed with training dataset from 2008 to 2009. The selected predictor variables and coefficients were updated by four different model updating methods: (A) The prediction model was updated by using only recent two-year data and applied to patients in the following test year. (B) The model was updated by increasing the training dataset yearly. (C) The model was updated by increasing the training dataset on condition that the area under the curve (AUC) of the recent test year was less than 0.6. (D) The model was not updated. The AUC of the test dataset was compared among the four model updating methods.

RESULTS: Dose to parotid and submandibular glands and grade of xerostomia showed decreasing trends over the years (2008-2015, 297 patients, p<0.001). The AUC of predicting grade≥2 xerostomia for the initial training dataset (2008-2009, 41 patients) was 0.6196. The AUC for the test dataset (2010-2015, 256 patients) decreased to 0.5284 when the initial model was not updated (D). However, the AUC was significantly improved by model updates (A: 0.6164, B: 0.6084, p<0.05). When the model was conditionally updated, the AUC was 0.6072 (C).

CONCLUSIONS: Our preliminary results demonstrate that updating prediction models with prospective data collection is effective for maintaining the performance of xerostomia prediction. This suggests that a machine learning framework can handle the dynamic changes in a radiation oncology clinical practice and may be an important component for the construction of a learning health system.