2.4% word error rate, and still no idea who said what

In early June, Microsoft's AI Superintelligence team shipped MAI-Transcribe-1.5, and it posted a 2.4% word error rate on Artificial Analysis's independent leaderboard. It covers 43 languages from a single model, holds up on accents and noisy rooms, and can push through an hour of audio in under fifteen seconds.

Then read Microsoft's own Azure documentation for that same model. Under the usage notes, one line: "Diarization isn't supported."

Diarization is the part that tells you who was speaking. So the headline speech model of the year produces a near-flawless wall of words with no idea which of them came out of which mouth. That is not a slip in the release notes — it is a precise snapshot of where this industry actually stands. Hearing is nearly solved. Hearing was never the hard part of a meeting note.

This post is about the distance between an accurate transcript and a note you would actually send to your team: what word error rate measures, what it structurally cannot measure, and which layers sit above raw speech recognition and quietly decide whether a recording becomes useful. If you have ever received a transcript that was technically perfect and practically worthless, this is the explanation for that feeling.

TL;DR:

• Word error rate is bunched near the floor — the top nine models on Artificial Analysis's independent board sit between 1.73% and 3.31%, so picking a "more accurate" engine buys you a few dozen words an hour on a typical meeting.
• WER weights every word identically, so dropping the word "not" from a decision costs the same 0.016% as dropping a stray "um" — the metric cannot see the difference between a typo and a reversed commitment.
• The remaining work is above the transcript: speaker attribution, segmentation, refinement, and summarization. Microsoft's own model card is the proof — best-in-class accuracy, no diarization, no streaming.

Image: Steve Knight, Rockfield Studios, Wikimedia Commons, CC BY 2.0.

What 2.4% actually feels like when you are the one reading it

Word error rate is the oldest and bluntest number in speech recognition, and it is worth understanding exactly because it is quoted so often. Take what was really said — the reference transcript, produced by humans. Take what the model wrote. Count the words it got wrong in three flavors: substitutions (it wrote "delay" where someone said "deploy"), deletions (it dropped a word entirely), and insertions (it invented one that was never spoken). Add those up, divide by the number of words in the reference, and you have WER. Lower is better.

So 2.4% means roughly one wrong word in every forty-two.

Put that on a real meeting. A one-hour meeting where people genuinely talk for about forty-five of those minutes, at a normal conversational clip of around 140 words a minute, comes to roughly 6,300 words. At 2.4% WER, about 150 of them are wrong. Spread across a transcript, that is a handful of errors on every screen.

Here is the question the metric refuses to answer: which 150?

WER weights every word in the language identically. The article "the" and the negation "not" are worth exactly the same. So consider a transcript that renders "we're not going to ship in Q3" as "we're going to ship in Q3." That is one deletion. One word out of 6,300 — 0.016% of the transcript, a rounding error against a 2.4% budget. And the note now records the opposite of the decision the room actually made.

The same asymmetry runs through names, product codes, and internal acronyms — the words that carry the most meaning per character and appear the least often in training data. A transcript can nail 6,150 words of connective tissue and mangle every proper noun in the room, and its WER will look excellent. Anyone who has read back a transcript of their own standup knows this feeling: the words are right, and the meeting is missing.

The leaderboard has bunched up near the floor

This is the part that reframes the whole category. We pulled the Artificial Analysis speech-to-text board on August 4, 2026 — an independent benchmark, not a vendor's own scorecard. Its AA-WER v2 index is an audio-duration-weighted average across roughly eight hours drawn from three datasets: AA-AgentTalk (50%), VoxPopuli-Cleaned-AA (25%), and Earnings22-Cleaned-AA (25%). The top of the board reads like this:

Source: Artificial Analysis speech-to-text leaderboard, retrieved August 4, 2026 (AA-WER v2, non-streaming).

Two things jump out. First, Microsoft's 2.4% is real and independently measured — and it is third, not first. Microsoft's stronger claim, best-in-class accuracy across all 43 languages on the FLEURS benchmark, is the company's own reporting, so read it as a vendor number until someone outside Microsoft measures it.

Second, and far more important: nine models are packed into a 1.6-point band. Run that back through our 6,300-word meeting. The best model on the board misses about 109 words. The ninth-place model misses about 209. Whisper Large v3 — the open baseline a great many products still run, released years ago — misses about 256.

Think about what that means for a buying decision. Moving from a mid-tier engine to the outright leader saves you somewhere around 100 words an hour. That is not nothing, and in medical or legal transcription it genuinely matters. But it is nowhere near the difference between a note you trust and a note you rewrite from scratch. Every serious engine now clears 96% accuracy. The variable that decides whether your meeting note is useful is somewhere else entirely.

Microsoft's release notes are the most honest document in the category

Which brings us back to the line that started this post. Microsoft's Azure Speech documentation for MAI-Transcribe lists the model's limitations plainly: diarization isn't supported, prompt-tuning isn't supported, and the model runs in public preview through the LLM Speech API. Coverage of the launch adds that there is no native streaming API either, which limits real-time use.

Sit with the shape of that. The company put an enormous amount of engineering into the hearing problem — 43 languages, up from 25 in the previous generation, with 18 added without regressing accuracy — and shipped it without speaker labels and without streaming. Both of those are exactly what a live meeting needs.

Our take: this is not a gap in Microsoft's product, it is a statement about the layering of the whole field. Recognition is a component now, not a product. It has become the thing you buy, benchmark, and swap — and the interesting work has moved to everything wrapped around it.

The stenographer's ceiling

Here is the term we keep coming back to internally: the stenographer's ceiling. It is the point past which better hearing stops helping, because the remaining work is not hearing.

A perfect stenographer gives you a perfect record of the words. They will not tell you that the two "yeah, sure" replies came from different people and meant different things. They will not mark where one topic ended. They will not tell you that the twenty minutes about the office move do not belong in the note at all. Four distinct layers live above the transcript, and each one fails in its own way.

Speaker attribution — diarization. This is the machinery that splits a single audio stream into "speaker 1, speaker 2, speaker 3" and keeps those labels stable across an hour. It is a genuinely separate problem from recognition, with its own failure modes: two people talking over each other, someone joining halfway through, a single participant sounding different on speakerphone than on a headset. When it fails, it fails destructively. A transcript that attributes a commitment to the wrong person is worse than one that attributes it to nobody, because a reader will believe it. This is precisely the capability the top-scoring model does not ship.

Segmentation. Raw recognition output is a river. Real speech does not arrive in sentences — it arrives in restarts, trailing clauses, and interruptions, and the model has to decide where sentences break, where a speaker's turn ends, and where the conversation changed subject. Get this wrong and you produce a technically accurate paragraph that no human will read, because there is nowhere for the eye to rest. Get it right and the same words become skimmable. Not a single character of that improvement shows up in WER.

Refinement. Between the raw transcript and the note there is a cleanup pass: stripping filler and false starts, normalizing numbers and dates, fixing terminology the acoustic model had no way to know. "Q three" becomes "Q3." An internal codename that came through as three plausible English words gets restored. This is the layer where domain knowledge enters, and it is the difference between a document that reads like a court record and one that reads like notes.

Summarization. Finally, the judgment call: which parts of an hour were decisions, which were open questions, which were the assignment of work to a named person, and which were the twenty minutes about parking. This is the only layer most readers ever actually consume, and it is entirely downstream of the previous three. A summarizer working from an unattributed, unsegmented transcript is guessing at who committed to what. Give it clean speaker turns and it stops guessing.

The dependency runs in one direction, which is why the accuracy conversation is so misleading. Improving WER from 2.4% to 1.7% improves the input to a chain of four steps that were never measured at all.

Keyword biasing is the industry quietly conceding the point

There is one feature in the MAI-Transcribe-1.5 release that gives the whole game away, and it is the most useful thing in the launch.

The model supports keyword biasing, also called entity biasing: you hand it a list of up to 200 domain-specific terms — attendee names, product names, internal acronyms — and it weights its predictions toward that list, using surrounding context to decide when the bias should apply rather than forcing matches blindly. Microsoft reports up to a 30% WER reduction on FLEURS when biasing is used. Their published example is exact: without a keyword list, three names come out as "Sean," "Oif," and "Societal." With the list supplied, the model recovers "Shaun," "Aoife," and "Xochitl."

Look closely at what happened there. The audio did not change. The acoustics did not improve. The model was not hearing worse before — it was hearing fine and guessing wrong, because it did not know who was in the room. Hand it the attendee list and nearly a third of the errors evaporate.

That is the whole argument of this post, shipped as a product feature by the vendor with the best-scoring model. The most effective accuracy improvement in the year's headline release works by injecting information that was never present in the audio. Which means the remaining frontier in transcription is not acoustic. It is contextual — and context is exactly what the layers above the transcript are made of.

Meanwhile, everyone is racing to capture more audio

The hardware market has drawn the opposite conclusion, and it is worth noticing. INMO's GO3 glasses, at $599, promise real-time translation across 98-plus languages projected onto the lens, plus automatic recording, transcription, and summarization of every conversation and meeting, powered by ChatGPT and Gemini. Those are the vendor's own claims from its product page, and the ambition is clear enough: capture everything, everywhere, hands-free.

Capture was never the bottleneck. A phone on the table has been able to record a meeting well enough for a decade. Making capture more ambient raises the volume of audio flowing into exactly the same unsolved pipeline — and if anything it sharpens the problem, because a device that records every conversation you have all day produces far more material that needs deciding-what-matters applied to it, not less.

What to test instead of reading the WER number

If you are evaluating a transcription or meeting-notes tool, the published accuracy figure will not separate the candidates, because they are all clustered within a couple of points. Three tests will.

Record a real meeting with at least three people and one stretch of genuine crosstalk, then check whether speaker labels stay consistent from minute five to minute fifty — that is where diarization quietly degrades. Next, take a decision that was stated in the negative ("we're not doing the migration this quarter") and search the summary for it; a system that reverses or drops negations will do it here. Finally, compare the summary against your own memory of the meeting and count two things separately: what it included that did not matter, and what it missed that did. That second number is the one nobody publishes, and it is the one that determines whether you will still be using the tool in three months.

The bottom line

Word error rate answered the question "did the machine hear the words?" — and the industry has now answered it. What no leaderboard has answered is "does this tell me what the meeting decided?"

Those are different questions, and only one of them was ever the point. A transcript is a recording of sound. A note is a record of a decision. The gap between them is not measured in percentage points, and closing it will not come from a better microphone or a lower WER. It comes from everything you build on top: knowing who spoke, where the thought ended, which words were noise, and which three sentences of an hour were the reason the meeting happened at all.

The transcript was never the deliverable. It was the raw material.

That is exactly how we build Telli.sh: speech recognition is one component, and the work we treat as the product is the stack above it — speaker separation, segmentation, refinement, and summaries that separate decisions from chatter, in 15 languages. If you want to see the difference between a transcript and a note on one of your own meetings, you can record one and read the result in a few minutes.

Start a live AI note

Sources

• MAI-Transcribe in Azure Speech (preview) — Microsoft Learn — the limitations list, including "Diarization isn't supported"
• Microsoft AI Introduces MAI-Transcribe-1.5 — MarkTechPost, June 8, 2026
• Introducing MAI-Transcribe-1.5 — Microsoft AI
• Artificial Analysis speech-to-text leaderboard — AA-WER v2 figures retrieved August 4, 2026
• INMO GO3 product page — vendor claims for translation and transcription features
• Wikimedia Commons image page