How AI-written clinical text is judged in perioperative care: a scoping review of evaluation methods

Hospitals are beginning to use AI tools, such as ChatGPT, to write text for patients and clinicians: answers to questions before surgery, instructions for recovery at home, and summaries of a patient's history for the anesthesia team. Before any of this can be trusted, someone has to check whether the AI got it right. This review looks at how that checking is being done.

There is currently no agreed way to judge AI-written medical text. One study may have a single doctor read the answers, another may use a panel of specialists with a scoring sheet, and another may ask a second AI to grade the first. These approaches can reach very different verdicts on the same text, so a claim that an AI "performed well" depends heavily on how the test was run. The main reporting guideline for AI research asks authors to describe their method but does not say what a sound method looks like.

We are going through every published study in perioperative care in which an AI wrote this kind of text, and recording who judged it, how many people judged it, what standard they used, whether the judges were told which text came from the AI, whether they agreed with each other, and whether anyone checked that the AI gives the same answer twice. The review covers both anesthesia care and the wider information patients receive around an operation.

The review will show where current evaluation methods are strong and where they fall short, and it gives us the foundation for the next step, in which anesthesiologists and an AI grader score the same set of AI-written outputs so we can measure how closely they agree. Together, these point toward a practical standard for checking AI-written text before it reaches a patient.