Same 10.0% error rate, opposite decision: how to measure meeting transcription accuracy

Here are two machine transcripts of the same twenty-word sentence from a product meeting.

A — "Priya said Helios migration slips to Q3 and we should not sign the vendor contract before security review"

B — "Priya said the Helios migration slips to Q2 and we should sign the vendor contract before the security review"

Each one contains exactly two errors against what was actually said. Each one scores a word error rate of 10.0%. Transcript A dropped the word "the" twice. Transcript B moved a deadline forward by one quarter and deleted the word "not" from a commitment.

If your evaluation process ranks those two systems as equal, the process is the problem — not the models.

This post is the measurement procedure we would hand to somebody told to pick a transcription tool by Friday. It covers what word error rate counts, exactly; why the number misleads on meeting audio specifically; and a five-step protocol you can run in an afternoon with two open-source packages and a 12-minute recording. We have argued separately that transcription accuracy alone was never the deliverable. This is the companion piece — not the argument, the method.

TL;DR:

• WER is (substitutions + deletions + insertions) ÷ reference words, and every word carries identical weight. The negation "not" and the article "the" cost the same.
• The published leaderboards have compressed. On Artificial Analysis's independent board, retrieved August 31, 2026, the top five models sit between 1.7% and 2.6% AA-WER — a spread too narrow to pick a winner from.
• Score five dimensions instead, weighted: load-bearing terms (30), speaker attribution (25), numerals and amounts (20), segmentation (15), raw WER (10). The tooling for the hard parts already exists — cpWER and DER ship in MeetEval, the toolkit built for the CHiME challenges.

Both columns contain two errors against a 20-word reference. Only one of them still records the decision the room actually made.

What word error rate counts, written out in full

WER comes from a single alignment. Take the reference transcript — what humans agree was said. Take the machine's output. Find the cheapest sequence of edits that turns one into the other, using the classic Levenshtein alignment, and label each edit:

• a substitution when the model wrote a different word ("deploy" heard as "delay"),
• a deletion when a reference word has no counterpart in the output,
• an insertion when the model produced a word nobody said.

Then:

WER = (substitutions + deletions + insertions) ÷ reference words

The denominator is the reference length, not the hypothesis length, which is why WER can legitimately exceed 100% — a model that hallucinates freely can rack up more insertions than there were words to begin with.

A worked example, taken from MeetEval's own documentation so the arithmetic is checkable against a published output. Reference: "The quick brown fox jumps over the lazy dog" — nine words. Hypothesis: "The kwick brown fox jump over lazy" — seven words. The alignment finds two substitutions (quick→kwick, jumps→jump) and two deletions (the, dog), no insertions. That is (2 + 2 + 0) ÷ 9 = 0.4444, a WER of 44.4%, and MeetEval prints exactly that: ErrorRate(error_rate=0.4444…, errors=4, length=9, insertions=0, deletions=2, substitutions=2).

The alignment is mechanical and reproducible. What it is not is weighted. Source: MeetEval documentation, retrieved August 31, 2026.

Two properties of that formula matter more than anything else on a vendor's landing page. First, it is flat: the metric has no notion that some words carry the meeting and others carry the grammar. Second, it is speaker-blind: standard WER concatenates everything into one stream, so a system that transcribes every word perfectly and attributes half of them to the wrong person can post an immaculate score.

Why the published numbers cannot separate your candidates any more

Pull the leaderboards and the problem becomes arithmetic. Artificial Analysis, which runs its own evaluations rather than reprinting vendor claims, listed its top five models by AA-WER on August 31, 2026 as Fun-Realtime-ASR-preview at 1.7%, ElevenLabs Scribe v2 at 2.2%, Microsoft's MAI-Transcribe-1.5 at 2.4%, Smallest AI Pulse Pro at 2.4%, and Gemini 3.5 Transcribe at 2.6%. The best open-weights model, Mistral's Voxtral Small, sits at 2.8%.

The entire top of the market fits inside a 1.1-point band. On a 45-minute meeting with roughly 6,300 spoken words, the gap between rank one and rank five is about 57 words — under one word per screen of transcript.

There is a second problem, which is that the benchmark audio is not your audio. AA-WER v2 is an audio-duration-weighted average across roughly eight hours drawn from three sources: AA-AgentTalk at 50%, VoxPopuli-Cleaned-AA at 25%, and Earnings22-Cleaned-AA at 25%. Parliamentary speeches and earnings calls are recorded well, spoken in turns, and largely free of the thing that defines an internal meeting — four people interrupting each other about a project only they have heard of.

Research on genuinely conversational audio has treated this as a separate discipline for years. The CHiME-6 challenge, published April 20, 2020 by Watanabe, Mandel, Barker and eighteen co-authors, built its evaluation on dinner-party recordings precisely because natural conversation with overlap and distant microphones breaks systems that score well on read speech. The AMI Meeting Corpus, 100 hours of recordings captured with both close-talking headsets and far-field room microphones, exists for the same reason.

Here's the point, and we will state it without hedging: a published WER tells you a vendor's model is not broken. It cannot tell you which vendor to buy.

What WER catches and what it structurally cannot

The bottom half of that table is the half that decides whether a transcript becomes a usable note, and standard WER sees none of it. That is not a flaw in the metric — WER answers the question it was designed for in 1990s dictation research. It is a flaw in using it alone in 2026.

Step 1: build a 12-minute clip that behaves like a real meeting

Do not evaluate on a vendor demo file and do not evaluate on eight hours of your own archive. You need one clip short enough to transcribe by hand and dense enough to break things.

Target 12 minutes. At a normal conversational pace of about 140 words per minute and roughly 75% talk density, that yields around 1,260 reference words — enough that a 2% difference in error rate is 25 words rather than statistical noise, and few enough that two people can transcribe it carefully in an afternoon.

Build the clip to a spec, not to a vibe. Ours:

1. Four or more speakers, including at least one joining remotely over a compressed audio channel, and at least one non-native accent in the working language.
2. Ninety seconds or more of genuine overlap — two people talking at once, someone finishing someone else's sentence, a cough over a decision. Crosstalk is where diarization quietly falls apart, and a clip without it measures nothing.
3. Twenty-five or more domain terms: product codenames, internal acronyms, customer names, attendee names. These are the words with the highest meaning-per-character in the room and the lowest frequency in any training corpus.
4. Fifteen or more numerals: dates, quarters, currency amounts, version numbers, percentages. Say some of them ambiguously on purpose ("fifteen" and "fifty" in the same minute).
5. Three or more negated decisions, stated naturally: "we're not shipping that in this release", "let's hold off on the contract".
6. Two simultaneous recordings where you can manage it — the laptop's own microphone and a per-speaker headset track. That is the AMI corpus design, and it lets you separate "the model is weak" from "the room is loud".

Record it once. Store the WAV uncompressed at 16 kHz or better. Every tool you evaluate from here on gets this identical file, forever.

Step 2: write the reference — then freeze the normalisation rules before you score anything

The reference transcript is a measurement instrument, and it has error bars. Artificial Analysis maintains "cleaned" versions of its public evaluation sets specifically because the original reference transcripts contained transcription errors of their own; their note is blunt about it — the cleaned versions "remove transcription errors from the reference text, providing a more accurate ground truth for model evaluation." If a benchmark operator has to correct published ground truth, your first-pass transcript is not gospel either.

So: two people transcribe independently, then reconcile every difference. Budget five to eight times real time for the pair, which puts a 12-minute clip at roughly one working day of combined effort. That is the single largest cost in this entire procedure and there is no way around it.

Then write down your normalisation policy before you compute a single score, because these choices move the final number by several points on their own:

• Numbers: is "Q3" the same token as "Q three"? Is "forty thousand" equal to "40,000"?
• Contractions: does "we're" match "we are"?
• Filler: are "um", "uh" and false starts in the reference at all, and are they scored?
• Casing and punctuation: normally stripped before scoring — which means a tool that punctuates beautifully gets no credit here, and you must score that separately in step 3.

Store the result in SegLST, the JSON format the CHiME challenges use and MeetEval's default: one object per segment with session_id, words, speaker, start_time and end_time. The speaker and timing fields are what make the speaker-aware metrics possible, and retrofitting them later is miserable.

Step 3: score five dimensions, and give raw WER ten points out of a hundred

Install the tooling — pip install jiwer meeteval — and score each dimension on the same clip. JiWER computes WER along with match error rate, word information lost, word information preserved and character error rate, using a RapidFuzz minimum-edit-distance implementation. MeetEval computes the meeting-specific metrics.

Our recommended weights. The dimension every vendor advertises is the one worth the fewest points.

Load-bearing terms — 30 points. Mark every proper noun, codename and piece of domain jargon in the reference. Then compute the same WER formula over only those tokens. We call the result the load-bearing error rate, and it is the single most predictive number in the exercise. In the pair of transcripts at the top of this post, the four load-bearing tokens are Priya, Helios, Q3 and not. Transcript A damages none of them: 0.0%. Transcript B damages two: 50.0%. Same overall WER of 10.0%, a fifty-point spread on the metric that matters.

Speaker attribution — 25 points. Run meeteval-wer cpwer -r ref.stm -h hyp.stm. Concatenated minimum-permutation WER finds the best mapping between reference and hypothesis speakers, concatenates each speaker's words, and scores the result — so words attributed to the wrong person become errors. Then subtract: cpWER minus plain WER is the attribution gap, the portion of the damage that comes purely from not knowing who spoke. Add diarization error rate for the timing view; DER is the sum of speaker error, false alarm speech and missed speech as percentages of scored time, defined in section 6.1 of the NIST RT-09 evaluation plan and implemented in dscore, which MeetEval wraps and which reads RTTM files. If your candidate supports timestamps, use tcpwer --collar 5 instead of cpwer so a correct word attached to the wrong moment is not silently forgiven.

Numerals, dates and amounts — 20 points. Extract every numeric token from the reference and score exact-match accuracy against the hypothesis, after applying your written normalisation policy. Report it as a plain fraction — "13 of 15 correct" — not a percentage, because the counts are small and a percentage implies precision you do not have.

Segmentation and readability — 15 points. This one is human. Have two readers who did not attend the meeting read the raw output and mark, per minute: sentence boundaries in the wrong place, speaker turns merged into one block, and paragraphs that require rereading. Punctuation was normalised out of your WER calculation, so this is the only place it gets scored at all.

Raw word error rate — 10 points. Run jiwer and record the number. It is a genuine floor check: anything above roughly 12% on clean close-talk audio indicates a real acoustic problem worth investigating. Beyond that floor, it discriminates almost nothing, which is exactly why it is worth ten points and not eighty.

Score each dimension 0–100, multiply by its weight, add. The result is comparable across tools and across time, which is the entire point.

Step 4: run every candidate on the identical file, and write down what "identical" meant

This is the step teams skip, and it invalidates everything upstream of it.

Same WAV, same sample rate, no re-encoding between candidates. Same reference, same normalisation policy applied by the same script. Diarization explicitly enabled or explicitly disabled on every tool, recorded either way — a tool evaluated with diarization off is not comparable to one evaluated with it on. Keyword or entity biasing off for the first pass and on for a second pass, both recorded; Microsoft reports up to a 30% WER reduction from supplying a keyword list to MAI-Transcribe-1.5, which is large enough that a comparison mixing biased and unbiased runs is meaningless.

Watch long-audio handling in particular. Artificial Analysis has to chunk its Earnings22 audio to roughly nine minutes for models that cannot reliably handle full-length input, and to roughly 30 seconds for models with tighter limits — their own published note names GPT-4o Mini Transcribe, Amazon Nova 2 Pro, NVIDIA Canary Qwen 2.5B and Alibaba's Qwen3 ASR Flash among the affected systems. Chunking changes results, because errors cluster at boundaries. If a vendor chunks your file, that is a property of the product and belongs in the scorecard, not an artefact to be corrected away.

Finally, record the exact model version string and the date of every run. "Whisper large-v3" is not a version; whisper-large-v3, run 2026-08-31 is.

Step 5: re-run the card when the vendor ships, not when you remember

Hosted speech models change underneath you. The endpoint name stays constant while the weights behind it do not, and nobody sends you an email when your meeting notes get slightly worse at proper nouns.

Three triggers should fire a re-run of the full card, on the same clip you built in step 1:

1. The vendor publishes a changelog entry touching the model, the diarizer, or the punctuation and formatting layer.
2. A calendar interval elapses — quarterly is enough for most teams, monthly if transcription output feeds anything automated downstream.
3. Your load-bearing error rate moves more than 5 points between any two runs, which should trigger an investigation before it triggers a migration.

Keep every run in one table: date, model version string, the five dimension scores, the weighted total. After three or four runs you own something no leaderboard can give you — a time series of how a specific product performs on audio that sounds like your actual meetings.

The bottom line: accuracy is a measurement of your room, not a property of the model

Word error rate is a property of a model on a corpus. What you actually need is a property of a model on your audio, for your vocabulary, with your people talking over each other — and that quantity does not exist until you measure it. Nobody can publish it for you, because nobody else has your codenames in their reference transcript.

The good news is that the measurement is cheap. One 12-minute recording, one working day of careful transcription, two pip installs, and a scorecard that fits on a page. The expensive alternative is the one most teams pick by default: choosing on a published percentage, and discovering the difference six months later, in a note that says the team agreed to something it explicitly refused.

Which number would you rather be wrong about — a decimal place, or a decision?

---

Where Telli.sh fits: we build the layers this scorecard actually weights. Telli.sh runs speaker separation, segmentation, refinement and summarisation on top of speech recognition, with translation into 44 target languages and an interface in 15, because the 90 points of that card that are not raw WER are where a transcript turns into a note somebody can act on. If you want a real data point rather than a benchmark, run your own 12-minute clip through it and score the result with the procedure above.

Start a live AI note

Sources

• MeetEval — a meeting transcription evaluation toolkit — cpWER, tcpWER, ORC-WER, MIMO-WER and DI-cpWER commands, the SegLST format, and the worked ErrorRate(errors=4, length=9, insertions=0, deletions=2, substitutions=2) example, retrieved August 31, 2026
• JiWER — speech recognition evaluation package — WER, MER, WIL, WIP and CER via RapidFuzz minimum-edit distance
• dscore — diarization scoring tools — DER as speaker error plus false alarm speech plus missed speech, per section 6.1 of the NIST RT-09 evaluation plan
• Artificial Analysis speech-to-text leaderboard — AA-WER v2 methodology (~8 hours across AA-AgentTalk 50%, VoxPopuli-Cleaned-AA 25%, Earnings22-Cleaned-AA 25%), the top-five model figures, the cleaned-reference note and the Earnings22 chunking note, all retrieved August 31, 2026
• CHiME-6 Challenge: Tackling Multispeaker Speech Recognition for Unsegmented Recordings (arXiv:2004.09249), April 20, 2020 — the segmented and unsegmented multispeaker tracks, and the dinner-party recording design
• The AMI Meeting Corpus — 100 hours of meeting recordings captured with synchronised close-talking and far-field microphones
• MAI-Transcribe in Azure Speech — Microsoft Learn — keyword biasing and the documented limitations
• Our earlier piece on why transcription accuracy was never the deliverable