Examination of the reliability and readability of Chatbot Generative Pretrained Transformer's (ChatGPT) responses to questions about orthodontics and the evolution of these responses in an updated version.

INTRODUCTION: This study aimed to assess the reliability and readability of Chatbot Generative Pretrained Transformer (ChatGPT) responses to questions about orthodontics and the evolution of these responses in an updated version.

METHODS: Frequently asked questions about orthodontics by laypeople on Web sites were determined using the Google Search Tool. These questions were asked to both ChatGPT's March 23 version and May 24 version on April 20, 2023, and July 12, 2023, respectively. Responses were assessed for readability and reliability using the Flesch-Kincaid and DISCERN tests.

RESULTS: The mean DISCERN value for general questions was 2.96 ± 0.05, 3.04 ± 0.06, 2.38 ± 0.27, and 2.82 ± 0.31 for treatment-related questions; the mean Flesch-Kincaid Reading Ease score for general questions was 29.28 ± 8.22, 25.12 ± 7.39, 47.67 ± 10.77, and 41.60 ± 9.54 for treatment-related questions; mean Flesch-Kincaid Grade Level for general questions was 14.52 ± 1.48 and 14.04 ± 1.25 and 11.90 ± 2.08 and 11.41 ± 1.88 for treatment-related questions; in first and second evaluations respectively (P = 0.001).

CONCLUSIONS: In the second evaluation, the reliability of the answers given to general questions and treatment-related questions increased. However, in both evaluations, the reliability of the answers was found to be moderate according to the DISCERN tool. On the second evaluation, Flesch Reading Ease Scores for both general questions and treatment-related questions decreased, meaning that the readability of the new response texts became more difficult. Flesch-Kincaid Grade Level results were found at the college graduate level in the first and second evaluations for general questions and at the high school level in the first and second evaluations for treatment-related questions.