How Do You Know Your AI Works? A Plain Guide to Evals
A scribe that demos beautifully can still be wrong in ways nobody sees until it is in a chart. Evals are how a practice measures whether a tool actually works, before it goes live and every month after. A non-technical guide to doing it honestly.
In this article
With ordinary clinic software, a test either passes or fails, and you move on. AI is different. The same consultation can produce a great note today and a subtly wrong one tomorrow, and “good” is not a yes or no, it is a rate. So the honest question is not “does it work?” but “how often, and how would we know?”
That is what evals are for. They turn AI quality from a vibe into a number you can measure, defend, and improve.
Start with what good looks like
Before anything touches a chart, decide what a correct answer actually is. Take real encounters from your own practice and write down the right output for each, the ground truth. This sounds obvious, and almost every practice skips it and ends up arguing about quality with no shared definition. If you cannot say what good looks like on ten of your own cases, you are not ready to measure, and you are certainly not ready to deploy.
Build a golden set
Now collect those into a small, representative golden set: real cases paired with their known-good answers, covering the routine visits and the awkward ones. It does not need to be huge. A few dozen well-chosen encounters give you a repeatable yardstick you can run any tool against, including the one you are being sold. Build it from de-identified cases and it becomes an asset you keep when the vendor changes.
How to score it
There are three honest ways to score answers, and good evals usually use a mix.
- Exact-match for structured extraction, where there is one right value: a medication and its dose, an allergy, a date, a billing code. Fast, cheap, fully automatable, and the place where a silent error is most dangerous.
- Clinician review for judgment calls, where nuance or safety matter. It is the most trustworthy and it sets the standard, but it is slow and does not scale on its own, which is why the sample is twenty cases rather than two hundred.
- LLM-as-judge to scale, using one model to score another’s answers against your criteria. It runs cheaply on every change, so you keep tight feedback, as long as you keep human spot checks to make sure the judge stays honest.
The point is to match the method to the task, not to chase a single accuracy number that hides what is really happening.
From vibe to number
The eval run
Every version faces the same yardstick
A score you can defend
Watch it after launch
Evals are not a launch gate you pass once. Real-world inputs shift, and the model underneath you can change, so quality can slip without anyone touching the code. This is model drift, and it is invisible until you measure it.
So keep scoring after go-live, and re-run your golden set on every change. When a number drops, you find out from a monthly review, not from a colleague reading back through a chart six months later. That is the difference between managing quality and hoping for it, and it is the sampling loop that makes a named owner possible at all.
Tie evals to the metric that matters
One trap is to fall in love with an abstract accuracy percentage that does not mean anything to the practice. The strongest evals connect to the real number from your proof of value: notes closed same day, minutes per intake, claims accepted first time, corrections a clinician has to make. An eval score is only useful if a higher score reliably moves that number. Measure the thing that actually changes the outcome, not the thing that is easy to chart.
You cannot trust what you cannot see
Evals are how AI grows up from a clever demo into a system you can rely on. They let you say, with evidence, how good it is, prove it is not getting worse, and improve it on purpose. You cannot improve what you cannot measure, and you cannot trust what you cannot see.
Not sure how good the tool your clinic already runs actually is? That is exactly what evals answer. Book a free consult and we will define what good looks like for one workflow, build a golden set from your own cases, and give you a number you could put in front of a regulator. See more on PHIPA and Guardrails.
Frequently asked questions
What are evals in AI, in plain terms?
Evals are tests that measure whether an AI tool is actually good and reliable, expressed as a number rather than a feeling. You collect real cases from your own practice with known-good answers, run the tool on them, and score how often it gets them right. That score is how you decide if it is ready for a chart, and how you catch it slipping later.
How do we measure if our AI is accurate enough?
Start by defining what a good answer looks like on real cases (the ground truth), gather a small golden set of them from your own encounters, then score the tool against them. Use exact-match for structured extraction like a medication or a date, clinician review for judgment, and an LLM-as-judge to scale, with human spot checks. 'Enough' depends entirely on what it touches: a drafted recall message and a drafted clinical note do not share a threshold.
Do evals stop once we launch?
No, and in a clinic this is the part that matters most. Quality is not one-and-done. A new clinician joins with a different consulting style, your templates change, the vendor ships a new model version on a Tuesday, and a tool that scored well at launch quietly gets worse. Keep scoring monthly on your own cases, so a drop shows up as a number before it shows up in a chart.
Can't we just rely on a good demo?
A demo proves the tool can be right once, on an encounter the vendor chose. It says nothing about how often it is right across your messy real cases, or where it fails. Evals replace that one hopeful data point with an honest, repeatable measurement. The gap between a conference demo and a score on your own encounters is the gap between hoping and knowing.
Found this useful?
Share this with your network on LinkedIn, it helps more than you think.
Enjoyed this read? Get the next one in your inbox.
When there is something worth your time, you will be first to know. No spam, unsubscribe anytime.
Keep reading
Prompt Injection: The Security Hole Your New AI Feature Just Opened
The moment a clinic assistant reads text somebody outside wrote, a patient message, a faxed referral, that text can try to give it orders. A five-scenario quiz teaches you to spot prompt injection, and the three design moves that contain it even when detection fails.
Read article
What Should We Refuse to Sign? Five AI Contract Terms Decided Once, at Signature
Data ownership, training rights, exit, price, and liability are settled the day you sign and renegotiated almost never. In a clinic one of them is not just commercial, it is a privacy obligation you already hold. Here is what each clause decides, what the standard wording quietly grants, and the version worth asking for instead.
Read article
What Are We Exposed To, and What Is Our Policy? The One Page Every Practice Owner Should Write
Your staff are already using AI, so the policy question is not whether to allow it. It is whether the rules exist in writing or in each person's private judgement. Exposure comes in three shapes, and in a clinic the one most policies address is not the one most likely to reach a patient.
Read articleHave software that should be smarter?
Let’s map a free AI-transformation roadmap for your product.