How to Test AI Clinical Note Accuracy Before You Trust It With a Chart

Every AI scribe demo is the same visit: a straightforward hygiene recall, clear audio, one operator talking, no interruptions. It produces a clean SOAP note and everyone nods.

That demo tells you nothing. The question isn’t whether the tool can write a note. It’s whether the note is true, whether it sounds like you wrote it, and whether it actually ends up in the chart attached to the right provider on the right date.

Here is the protocol I use. It takes about two hours of real clinical time spread over a week, and it will tell you more than any sales call.

Rule zero: never evaluate on a live patient chart

Run every test in a sandbox or demo environment, or on a test patient you created for this purpose. If a vendor can’t give you a non-production environment, that is itself a finding. You are about to deliberately feed this system a garbled recording and a visit where the plan changed — you do not want the output filing itself into a real record while you decide whether it’s any good.

Test 1: Run the same visit three times

Record one visit. Then submit that same audio (or dictate the same content, as close to verbatim as you can) three separate times.

You are looking for variance. A note-writing model is probabilistic. Some variance in phrasing is normal and fine. Variance in clinical content is not.

Compare the three outputs side by side and mark every difference:

• Different tooth numbers → disqualifying
• A finding present in run 2 but absent in runs 1 and 3 → disqualifying
• Different anesthetic, different carpule count, different surface list → disqualifying
• “Patient tolerated procedure well” vs “Patient tolerated well” → fine

If clinical facts move between runs, the tool is generating rather than transcribing, and you cannot audit it at volume. You will catch the first ten drifts and miss the eleventh.

Test 2: Dental vocabulary, specifically

General-purpose speech models were not trained on operatory audio. Say each of these out loud at normal chairside speed and check the literal output:

Also test your own numbering habit. If you dictate “upper right six” and your chart is Universal, does the tool convert to #3, or does it write the words? Both are defensible. Silently guessing wrong is not.

Do this test with your actual suction running. Vendors demo in a quiet room.

Test 3: The hallucination test — the one that actually matters

This is the failure that ends careers, not the one that annoys you.

Dictate a visit and deliberately omit things a model might expect to be there. Say nothing about periodontal probing depths. Don’t mention a periapical radiograph. Don’t state a blood pressure.

Then read the output looking for anything you did not say.

Invented perio measurements are the classic. The model has seen ten thousand notes where a hygiene visit includes probing depths, so it supplies plausible ones — 3,2,3 across the arch. Those numbers are now in a legal record, and if that patient later has a perio dispute or you’re defending a treatment sequence, you signed something you never measured.

Second-worst: procedures never performed. Dictate “we discussed extraction of #17 but patient wants to wait,” and check whether the note documents an extraction.

Score this pass/fail. One invented clinical finding across ten test notes is not an acceptable rate.

Test 4: The plan that changed mid-appointment

Real dentistry: you open the tooth for a two-surface composite, find the decay is into the pulp, and the visit becomes a pulpotomy and a crown discussion, with the patient re-consenting chairside.

Dictate that arc — the original plan, the finding, the change, the conversation, the revised plan. Then check the note for:

1. Does it document what was actually done, not what was scheduled?
2. Does it preserve the informed-consent conversation that happened after the change?
3. Does it keep the sequence, or flatten everything into one paragraph that reads like the plan was always the pulpotomy?

Most tools do #1. Fewer do #2. #3 is where you find out whether the tool understands a visit or just summarizes words.

Test 5: Interrupt the recording

Start dictating. Thirty seconds in, close the laptop lid, kill the app, or walk out of Wi-Fi range. Then come back.

Three questions:

• Is the partial dictation recoverable, or did you lose it?
• How long is the recovery window?
• What was retained to make recovery possible — the audio, or only the transcript text?

That last one is the real question and most vendors won’t volunteer the answer. Retained audio is PHI sitting in a bucket with a retention policy you didn’t write. Ask directly: is dictation audio written to disk, S3, or a database at any point? On our platform, dictation audio is held in memory only and never persisted; only transcript text is kept, for 24 hours, which is what makes an interrupted recording resumable. Whatever the answer is at your vendor, get it in writing, and note that a separate consent-gated recording feature is a different question with a different answer.

While you’re there, ask what happens to a signed note when you need to correct it. The correct answer is a dated addendum. If the tool silently rewrites a signed note, your audit trail is fiction.

Test 6: Your voice or a template?

Take five notes you wrote yourself last year. Take five AI notes of comparable visits. Hand all ten to your associate or hygienist with the headers stripped and ask which are which.

If they can sort them 10/10, the tool is producing generic template prose. That matters for two reasons: your notes are the record of your clinical reasoning, and a chart full of identical boilerplate is the first thing a reviewer flags.

Check whether the tool learns from your edits or just re-runs the same prompt forever. Most don’t learn. Ask.

Test 7: Verify it lands in the chart correctly

This is where evaluations usually stop too early. The note looked good on screen — but did it write into the PMS, dated to the visit, attributed to the treating provider?

Go look in the PMS itself, not in the vendor’s dashboard.

Attribution differs by PMS, and honest vendors say so. Dentrix Ascend files a note against a real provider field. An Open Dental commlog has no provider field at all — so the provider’s name has to be written into the note body. That’s a workable answer, but it’s a different answer, and “structured provider attribution” is not something anyone can truthfully claim on an Open Dental commlog. It is a fair question to put to any vendor, including ours.

Then check how the tool writes. Open Dental publishes a third-party vendor list with explicit trust tiers: Authorized (uses the official API), Authorized in Transition, and Dangerous/Unknown — the last covering direct database access and unwhitelisted plugins. Roughly 90 vendors sit in the Authorized tier; Lassie and Lighthouse currently sit in Dangerous/Unknown. Listing is free, so a vendor’s absence is a choice, not a cost. Look your candidate up before you sign.

Henry Schein One runs the equivalent for Dentrix Ascend: an API Exchange with 140+ authorized vendors, where authorization requires SOC 2 Type II and OAuth 2.0 certification. Ask to see the certification, not the badge.

Worth knowing: Open Dental has no first-party AI scribe. Bola AI is a native partner, not an Open Dental product. Open Dental does sell its own eServices bundle at $165/month — Web Forms, Patient Portal, eReminders, eConfirmations, Integrated Texting, Web Sched, Secure Email, ODMobile — which is worth pricing against whatever else you’re stacking on top.

The scoring rubric

Ten notes minimum, across at least three visit types. Score each category once for the whole batch.

Category 1 is a gate, not a score. Any fabricated clinical finding fails the whole evaluation regardless of total. A tool that scores 95 and invents one pocket depth is worse than a tool that scores 70 and never does, because the 95 teaches you to stop reading.

Interpretation: 85+ with category 1 clean — pilot it on one provider for 30 days with full review of every note. 70–84 — pilot only if the gaps are in categories 3, 8, or 9, which improve. Below 70, or any category 1 failure — no.

What you should expect vendors to dodge

Three questions produce the most evasion, so ask them in email where the answer is written down:

• Is dictation audio persisted to disk or object storage at any point, and for how long?
• What is your provider attribution mechanism in my specific PMS?
• Can a signed note be modified, by anyone, without an addendum record?

“We’re HIPAA compliant” is not an answer to any of those. Neither is a BAA — though you should have one signed before a single patient record moves, and if a vendor will exchange test PHI without one, that tells you what their compliance posture actually is.

The note is a legal document. Evaluate it like one.