When Your AI Should Look, Not Just Read: Multimodal AI in a Clinic

Almost every clinic AI conversation in the last two years has been about text and speech. Consultations in, notes out. Meanwhile a great deal of what a practice knows never became text at all. It is the requisition that arrived by fax, handwritten and slightly skewed. The intake form a patient filled in on a clipboard. The consult letter scanned in 2019. The wound photo a patient sent because describing it was harder than showing it.

That information exists. It is just unqueryable, and the way it becomes usable today is that somebody at your front desk looks at it and types what they see, one field at a time, between phone calls.

The question is not whether AI can see

It can, and that stopped being the interesting part. Computer vision used to mean a specialised model trained for one narrow job, and optical character recognition meant a brittle pipeline that wanted a clean, flat scan. A general model that reads a crooked fax and returns structured fields is genuinely new. It is also not the interesting question anymore.

The useful question is narrower and much more practical: does the truth of this workflow live in an image, and is a staff member currently the only thing converting it into data?

Answer that honestly and workflows sort into three tiers.

Text is enough. The images in this workflow are decoration, or the fact you need already arrived structured. Adding vision buys latency and cost and nothing else. Most clinic workflows land here, and doing the text version is the right answer, exactly as the plain version is usually the honest first move.

Add eyes to one step. The flow is fine; one point in it depends on a person reading an image. A health card number off a scan, a clinical indication off a faxed requisition, a date off a hospital discharge summary. The rest of the process never changes. This is the highest-return tier and the one practices most often skip past, because it is unglamorous.

Vision first. The image is the entire record. There is no text version. A fax tray, a filing cabinet of scanned letters, a stack of handwritten intake forms nobody was ever going to type. Here vision is not an enhancement, it is the only way that record becomes data at all.

What a vision step actually is

The mistake is picturing a model that “looks at the scan and understands it.” What earns its keep in a clinic is narrower and more boring: an image, plus what the chart already holds, going into a step that must return a fixed structure.

What makes a vision step trustworthy

The fax as it actually arrives What the chart already holds A strict output schema A confidence score per field

The vision step

Returns fields, not prose, and says when it is unsure

Structured facts a person can confirm in seconds

The strict schema is the load-bearing part. Prose about a scan is impressive and unusable; a set of typed fields drops into the EMR and can be checked at a glance. And the confidence score is what makes the human gate cheap: a receptionist confirms three flagged fields instead of re-reading the whole requisition.

The failure modes are not the ones you are watching for

Teams arriving from text AI expect the failure they already know: a confident, fluent, wrong answer. Vision has that too, and it has three of its own that catch people out.

Input variance beats model quality. A clean typed fax is nearly solved. The same requisition handwritten, faxed twice, with a stamp across the indication field, is a different problem entirely. Most accuracy complaints about vision are complaints about capture conditions, and in healthcare the cheapest fix is often persuading a referrer to send it another way, not buying a better model.

There is nothing to cite. A grounded text answer can point at the sentence it came from, which is the whole trust mechanism of an answer engine. An image has no passage to quote. The nearest equivalent is showing the crop the model read a field from, so a person can verify in a glance rather than taking it on faith.

Silent misreads look identical to correct reads. A wrong health card number, or a date of birth off by a digit, sits in the field looking exactly as confident as a right one. Nothing in the output announces the failure, which is why per-field confidence and a person on anything that reaches a chart are not optional extras. They are the design. It is the same human-in-the-loop discipline applied to a channel where verification is harder, not easier.

Vision is worth paying for exactly where a person is currently the only thing turning pictures into data. Everywhere else it is an expensive way to read something you already had.

Where this actually shows up

The pattern repeats across clinic types with almost no variation. Faxed requisitions and referrals. Handwritten intake forms that were never going to be typed. Scanned consult letters and discharge summaries sitting in a document folder nobody can search. Insurance cards photographed at the desk. Wound and lesion photos a patient sent in, which narrow what a clinician looks at and never decide anything.

The shape of the work is the same wherever the images come from: a person still decides, and the model only ever narrows what they have to look at.

What to do Monday

Walk one workflow and look for the moment somebody is staring at a fax and typing. That moment is the whole opportunity, and it is usually one step, not a programme. Scope that single extraction, hold it to a strict schema with per-field confidence, and test it against your own stack of real, skewed, second-generation faxes before anyone celebrates. If the workflow has no such moment, you have just saved yourself a project.

---

Sitting on a fax tray, a scan folder, or forms nobody can query? Book a free consult and we will find the step where someone is retyping what they see, and scope the smallest version worth proving. See more on the Working Pilot.