Manual review of electronic medical records as a reference standard for case definition development: a validation study.

BACKGROUND: The Canadian Primary Care Sentinel Surveillance Network (CPCSSN) previously carried out a validation study of case definitions for 8 chronic diseases (diabetes mellitus, hypertension, osteoarthritis, depression, dementia, chronic obstructive pulmonary disease, parkinsonism and epilepsy) using direct review of "raw" electronic medical record data. Although effective, this method is time-consuming and can present methodological and organizational challenges. We aimed to determine whether the processed and standardized data contained with the CPCSSN database might function as a reference standard for case definition validation.

METHODS: Using a traditional validation study design, we compared the case identification results of the chart reviews for the 8 chronic diseases with the results of a manual review of the CPCSSN processed data for the same conditions in the same patient sample. Patients were randomly sampled from the June 30, 2012 CPCSSN database, with oversampling of patients with rare conditions.

RESULTS: We analyzed data for 1906 patients. Manual review of the CPCSSN records for case ascertainment yielded sensitivity ranging from 77.5% (95% confidence interval [CI] 73.3%-81.6%) for depression to 97.2% (95% CI 95.4%-99.0%) for diabetes. Specificity was high for all definitions (range 93.1% [95% CI 91.4%-94.7%] to 99.4% [95% CI 99.0%-99.8%]). Positive predictive values and negative predictive values also showed high accuracy of the manual CPCSSN record review relative to review of the raw chart data.

INTERPRETATION: The use of CPCSSN records as the reference standard to validate case definitions substantially reduces the burden on sentinel physicians and clinic managers as well as on researchers while offering a reference standard that is a reasonable substitution for chart review.