Clinicians are the backbone of Vera. They were our first users, our first critics and our first believers, and their judgment sets the standard for every answer we give. We work with them to train an AI that grades Vera’s answers the way they would, every day. Here is how we built it.
The most dangerous answer in medicine is not the one that is obviously wrong; a clinician catches that in seconds. It is the one that sounds right. A general-purpose model produces plausible sentences; medicine requires verifiable ones. An answer can cite a real guideline and get every fact right, and still leave out the one detail that matters.
Take a question typed between two patients: “dose apixaban for 82yoF w AFib”. The standard dose is 5 mg twice daily. But at 82, she already meets one of the criteria for the reduced dose: a weight of 60 kg or less, or a creatinine of 1.5 mg/dL or higher, would bring the labeled dose down to 2.5 mg twice daily. And with a mechanical heart valve or moderate-to-severe rheumatic mitral stenosis, apixaban is not recommended at all.
An answer that stopped at “5 mg twice daily” would sound right, and in most cases it would be. Nothing in it would be false, and every sentence would survive a fact-check; the danger sits in the sentence that is not there. That is the error that worries clinicians most, and the hardest to see. It is why Vera asks for weight, creatinine and valve history before it answers (Figure 1).
Vera answers hundreds of thousands of clinical questions every day, and each answer has to be safe and accurate. At this scale, we cannot keep clinicians in the loop for every answer; no team of clinicians could. So the question behind our evaluation work is simple: how do you hold every one of those answers to the standard a physician would apply, and make sure Vera stays safe at scale?
We are not the first to ask. The NOHARM study, from researchers at Stanford and Harvard, asked physicians to grade AI recommendations on real consult cases, action by action, counting both the harmful advice an AI gives and the essential steps it leaves out. Its finding: most of the potential harm came from omissions, not from reckless advice. We loved that work. It is the closest thing we had seen to how a clinician actually judges an answer, and it shaped how we built our own checks.
Why grading medical AI is hard
Medicine is not a multiple-choice problem; it is full of nuance. An exam question has one right letter, but the right answer to a real clinical question depends on its clinical context: in the apixaban example, age, weight, kidney function and heart valves all change the answer. A good answer gets all of that right, says where it stops applying, and tells you when to escalate. Two good answers can look nothing alike, and a bad one can look excellent.
That is why the usual shortcut, asking a general-purpose AI model to grade another model's answer, does not work well in medicine. These graders have habits a clinician would never accept:
- They reward length. A longer answer looks more thorough, even when the extra paragraphs add nothing a clinician needs.
- They reward confidence. A fluent, assertive answer reads as correct, whether or not it is.
- They favour position. Shown two answers, many graders lean towards whichever one comes first, or second, regardless of content.
- They miss what is absent. It is easy to check what an answer says. It is much harder to notice what it should have said and did not, which is exactly the apixaban problem.
A 2026 study of real point-of-care questions found that off-the-shelf AI graders often disagree with physicians (Feng et al., Expert Evaluation of Clinical AI Tools on Real Point-of-Care Clinical Queries). So to build a good grader, we had to start with what physicians know, and measure the grader against them before trusting it with anything.
Doctors set the standard
We worked with practising clinicians. Each clinician saw two answers to the same question, blinded and in random order. They did not know which system wrote which answer.
For each answer, they filled in the same structured form:
- Clinical accuracy: is the answer right for this patient, and safe to act on?
- Format: can a busy clinician find the answer in seconds?
- References: do the cited sources exist, say what the answer claims, and come from the right level of evidence?
They tagged every problem against a fixed list: a wrong dose, outdated guidance, a missed safety check, a missing special population, a misattributed citation. Then they wrote down their reasoning in a few sentences and said which answer was better, or that the two were equal.
Independent groups of clinicians judging the same pair reach the same verdict 92% of the time, a high level of agreement for questions this nuanced. That is the bar any automated grader has to approach.
Medicine has never had perfect agreement between experts, and we did not expect it. What we needed was to know how high the ceiling is. It is also why medicine rarely relies on a single opinion. Tumor boards, guideline panels, second opinions and the double reading of mammograms share the same idea: the collective judgment of experienced clinicians is stronger than any single read. Research on the wisdom of crowds in medicine bears this out. Across 1,572 cases, combining the independent judgments of several physicians made diagnoses steadily more accurate (Barnett et al., JAMA Network Open, 2019). We built our evaluation on the same principle. Every comparison is blinded and judged independently by several clinicians, and the standard we hold Vera to is what they agree on together.
What the clinicians taught us
The written reasoning taught us more than the scores did. The main risk the clinicians flagged was rarely reckless advice. As the NOHARM authors had found, it was omission. The same gaps came up again and again:
- A missing adjustment: a renal or hepatic dose cut-off, an age threshold, a weight-based limit.
- An unmentioned population: pregnancy, breastfeeding, children, frail older adults.
- No escalation criteria: the red-flag signs that warrant urgent evaluation or referral.
- No clinical bottom line: the evidence is summarized, but its practical implications are left unstated.
Clinicians also checked the sources behind each answer: the guidelines, trials and papers it cites to support its claims. Here the problem was rarely a made-up citation. More often, an answer cited an older guideline that a newer version had already replaced. That kind of error is easy to miss: the citation is real, the guideline exists, and nothing looks wrong unless you know a newer version has been published.
Together, these patterns told us what our AI grader had to catch: what an answer leaves out, and whether its sources are up to date.
Teaching AI to read like a reviewer
We then built our own graders, which we call judges. Our clinical-accuracy judge fills in the exact form the clinicians used, field for field: the rating, the tagged issues, the written reasoning, the final verdict. It has to write its reasoning before it is allowed to give a verdict, the way a clinician reviewing an answer would.
It was prompt-engineered on real clinician reviews, with their reasoning and methods. The clinicians' written rationales matter most here. They show the judge not only what a clinician concluded, but why: which missing sentence made an answer unsafe, which citation did not hold up.
And it reads every pair twice, with the answers swapped. If its verdict changes when the order changes, it was reacting to position, not content, and we count that against it.
Across the 50 held-out blinded comparisons:
- Clinical accuracy: a general-purpose AI grader reached the clinician's verdict 72% of the time. Ours reaches it 82% of the time.
- References: the best general-purpose grader reached it 60% of the time. Ours reaches it 86% of the time.
- Steady when the order changes: our judge gives the same verdict in both orders 88% of the time, against 75% for a general-purpose grader. The verdict depends on the content, not on which answer comes first.
- Zero flipped verdicts: when our judge disagrees with a clinician, it is over a close call, a "slightly better" against an "equal". It never picked the answer the clinician called worse.
Getting there meant trying judges built on several families of AI models, and they were far from equal. Some families simply grade more like a clinician than others. Some cannot be fine-tuned at all, and for some, no amount of careful instruction-writing ever reached the bar. This is where it gets even more interesting. If the collective judgment of several clinicians is stronger than any single read, the same principle should apply to AI. So we do not rely on a single judge either. Each answer is graded by a jury of judges built on different AI models, each reading the answer independently, and their verdicts are combined into one. Different models have different blind spots, so a jury is less likely to miss what any single model misses: the wisdom of crowds, applied to machines.
Safety at scale, on real answers, every day
Our judges are not a one-off study. They work in two places, every day.
- Before any change reaches clinicians. Every change to Vera, a new model, a new instruction, a new tool, is graded by the judges on a fixed set of questions, from different specialties and different countries, and compared with the version clinicians use today. Like a new treatment measured against the standard of care, a change is released only if it performs at least as well as the current version. This way, Vera never takes a step backward.
- On real questions, every day. Every day, our judges grade a sample of real Vera answers. Beyond clinical accuracy, they check that each answer follows the guidelines that apply where the clinician practices, and that it is clearly formatted. Every morning the team receives a quality report: the trend for each judge, and every answer that crossed an alert threshold, each linked to the full record of how Vera produced it. The judges point us to where Vera can do better, usually new use cases that clinicians have started bringing to Vera before we had built for them. Each flag goes to someone on the Vera team. We fix what can be fixed right away, and the case joins a growing set of permanent test cases that every future version of Vera must pass.
Our goal is simple: knowing what to fix next, and having judges that tell us when it is fixed.
What it changes
The judges do more than catch problems; they show us how to make Vera better. What clinicians look for most, special populations, escalation criteria, a clear clinical bottom line and up-to-date sources, now shapes how Vera writes its answers. The same principle is behind Vera asking for missing information before it answers.
Every change to how Vera writes is graded by the same judges before it reaches clinicians, and none is released unless it performs at least as well as the version before it. Clinicians set the standard; the judges hold every version of Vera to it, every day.
Now you may wonder: what happens when Vera faces the latest frontier models and its competitors, graded by these same judges? That is our next post.
You can only improve what you can measure
There is an old engineering principle, often attributed to Lord Kelvin: you can only improve what you can measure. Medicine has lived by it for a long time. Blood pressure, HbA1c and survival curves are how clinicians know whether a treatment works.
The same goes for clinical answers, but what a good answer looks like can be blurrier than a lab value. It has to be correct, complete for its clinical context, honest about uncertainty and backed by current sources, and it takes a clinician to judge all of that. That makes it one of the hardest things to measure in healthcare. It is also the only way to build something better, and it is why everything we have described here starts with clinicians.
Once you can measure it, you can improve it. Every week, Vera gets a little better, and we can measure exactly how much.
Try it at verahealth.ai.