In medicine, a new treatment is not adopted on its own claims. It is compared with what clinicians already use, on outcomes that matter, and the results are published for anyone to check. Clinical AI should be held to the same standard. Clinicians can now choose between general AI models and tools built for medicine, and they deserve to know how those tools compare on the questions they actually ask, not on exam questions.
Most published results for clinical AI come from standardized benchmarks built on exam questions: a complete vignette, four or five options, one right letter. We have published these scores ourselves, and they have their place, because anyone can reproduce them. But medicine is not a multiple-choice problem. A real clinical question rarely has one right answer; the best one depends on the clinical context, the strength of the evidence and what remains uncertain, and two good answers can look quite different.
Above all, picking the right letter out of five does not show that a system can read a half-finished question typed between two patients, find the current guideline for that clinician’s country, weigh a recent trial against an older consensus and say clearly what the evidence does not settle, in an answer a clinician can read in thirty seconds. That is what we wanted to measure.
So we chose a test built from exactly that moment. In June 2026, researchers at UCSF, Harvard, Stanford and the University of Washington published Real-POCQi: 620 real questions that physicians had asked OpenEvidence at the point of care, across 30 specialties, with identifying details removed (Feng et al., 2026). In a blinded study, 149 practicing physicians, each matched to questions in their own specialty, compared answers from OpenEvidence and three general AI models, and rated OpenEvidence highest on every dimension. The authors released the questions, answers and ratings publicly under a CC BY 4.0 license, and were not involved in our analysis. Because everything is public, anyone can run the same test.
We did: we put the same questions to Vera and to three of the newest frontier AI models, and compared their answers with the published ones.
Six measures, in plain words
Every pair of answers was graded on six measures. Five are the dimensions the study’s physicians rated, worded exactly as in the study. For these, we used the AI judges the study itself tested, GPT-5.5 and Gemini 3.1 Pro running the study’s own judge prompt, not physicians. The sixth is clinical accuracy, graded by the clinician-trained judge we described in our previous post.
- Accuracy: Which response is more factually accurate?
- Clinical utility: Which response is more useful for providing high-quality clinical care?
- Source quality: Which response is supported by more authoritative source material?
- Verifiability: Which answer is easier to verify?
- Completeness: Which response more completely addresses the question?
- Clinical accuracy (our judge): Is the answer clinically correct and safe for the situation described? This judge fills in the same structured form our clinicians use.
Who took part
We put the same questions to Vera, twice, to make sure the result was not a lucky draw. We did the same for three of the newest frontier AI models, GPT-6.1 Sol, Claude Opus 5.5 and Gemini 3.8 Flash, each with web search switched on, as in the original study and as a clinician using them would. For OpenEvidence and GPT-5.5, we used the answers published with the study, generated by its authors in June 2026. We did not run either system ourselves, and their current versions may answer differently.
Every pair of answers was graded blind: the judges never knew which system wrote which answer. Each pair was also graded in both orders, so a judge's preference for whichever answer comes first cannot decide the result. For the study's judges, we used both of the AI models the study used, and a question counts as a win only when their combined verdict favours one side.
Head to head with OpenEvidence
The two accuracy measures, graded by different judges, point the same way. Our clinician-trained judge found Vera’s answer clinically more accurate on 71% of questions, OpenEvidence’s on 3%, and the two equal on the remaining 26%. The study’s AI judges found much the same: 80% to 7%.
Source quality and verifiability are the measures OpenEvidence is best known for. In the original study, source quality was where physicians preferred it most clearly. Here, after completeness, they were also the closest results: 58% to 27% and 55% to 30%.
The whole field
Head-to-head results can hide a lot, so we also compared every system with all five others at once (Figure 2). Each cell shows a system’s win difference: how often its answer was preferred minus how often it was not, averaged across every opponent. A positive score means it won more often than it lost, a negative score the opposite, and zero means an even record. This is the same summary measure the original study used as its primary endpoint.
Against the field, Vera had the highest win difference of the six systems on four of the six measures: clinical accuracy, graded by our judge, and accuracy, source quality and verifiability, graded by the study’s judges. Claude Opus 5.5 had the highest clinical utility and completeness, with answers nearly twice as long as Vera’s. The original study found completeness among the measures most sensitive to answer length.
Two patterns stand out. First, OpenEvidence's strength is real: it scores well on sources and verifiability against the general models, as it did with physicians in the original study. Second, the general models' weakness is just as real: without a curated library of medical evidence behind them, even the best of them struggle to cite well, however good their medical reasoning. A good clinical answer needs both: sound reasoning and the right evidence behind it.
Clinical accuracy against the best AI models
On clinical accuracy, Vera was judged more accurate than every system it faced. It is far ahead of OpenEvidence and Gemini 3.8 Flash, and ahead of Claude Opus 5.5, GPT-6.1 Sol and GPT-5.5. Against the GPT models, most questions were judged equal. When the judge did pick a side, it picked Vera more often.
That last result deserves a plain reading. The best frontier models are now very good at medicine. On many everyday questions, a careful answer from GPT-6.1 Sol and a careful answer from Vera will both be right. The difference shows up on the hard tail: the patient who meets the criteria for a reduced dose, the guideline that changed, the exception buried in a trial's exclusion criteria. That is where an evidence engine built for clinicians earns its place.
Two questions from the test show what that looks like.
"What is the appropriate evaluation for central retinal artery occlusion?" Both answers treated it as an eye stroke and sent the patient for urgent stroke workup. GPT-6.1 Sol ordered inflammatory markers for giant cell arteritis only "when GCA is suspected". Vera ordered them for every patient aged 50 or over, symptoms or not. That is the difference that matters: arteritic occlusion can present without headache or jaw pain, and if it is missed, the other eye can go next. The judge marked Sol down for the missed safety check.
"What is the recommended treatment for perianal rash in a patient with HIV?" GPT-6.1 Sol's answer covered herpes, candida and dermatitis, and never mentioned syphilis. Vera's answer cited the CDC's 2021 STI treatment guidelines, which call for syphilis serology for every anal or perianal ulcer, and gave the treatment: benzathine penicillin G, 2.4 million units intramuscularly once for early disease, with a check for ocular and neurological involvement. In a patient with HIV, that is the diagnosis you cannot afford to miss.
Neither competitor's answer was reckless. Each was missing one sentence, and in both cases it was the sentence that protects the patient.
That smaller set is what matters most in practice. Clinicians often turn to a tool when a question is unusual or the evidence is recent, and those are the questions where the evidence is hardest to find and weigh. Being accurate on most questions is the baseline; the hard questions are where clinical tools are really tested.
Shorter, on purpose
Vera's answers are short: a median of about 300 words, against 521 for OpenEvidence, and roughly twice that for Claude Opus 5.5. That is a choice, not an accident.
A clinician between two patients does not have time to read a review article. Vera is built to put the answer first, then the exceptions, then the evidence behind it. On completeness, one of the measures where length helps most, Vera still finished level with OpenEvidence while using about 40% fewer words.
Longer answers do score better on some measures, our analysis found accuracy and completeness ratings the most sensitive to answer length. You could make almost any system look more complete by letting it write more and think longer.
The caveat worth reading
These verdicts come from AI judges, not from the 149 physicians who graded the original study. That study found AI judges can disagree with physicians, and its physician panel preferred OpenEvidence over the general AI models it tested; Vera was not part of it. The same study found that AI judges often rank systems differently from physicians: they tended to agree with physicians on which system was best, but much less on the rest (Feng et al., 2026).
That is why we built our own judge the way we did. It is trained on clinician reviews and checked against their verdicts before we rely on it, as described in our previous post. It is still a judge, not a physician. The five study measures were graded by the study’s off-the-shelf AI judges, which have not been tuned in that way, and should be read with that in mind.
Ask Vera one of your own point-of-care questions at verahealth.ai.