Word Error Rate Is Not the Whole Story: A Guide to Evaluating Transcripts of Accented Speech

What word error rate actually measures

Word error rate, commonly abbreviated as WER, is a standard metric for evaluating automatic speech recognition. It compares a system-generated transcript with a reference transcript prepared according to an agreed convention. The calculation counts three types of word-level differences: substitutions, deletions and insertions. A substitution occurs when one word is replaced by another, a deletion occurs when an expected word is missing, and an insertion occurs when an extra word appears.

The formula is WER = (S + D + I) / N, where S represents substitutions, D deletions, I insertions and N the number of words in the reference transcript. For example, a reference containing 100 words with four substitutions, three deletions and three insertions produces a WER of 10 percent. The metric is straightforward, but its interpretation depends on the quality and consistency of the reference.

Why accented speech requires careful interpretation

Speech recognition systems must interpret variation in pronunciation, rhythm, vowel quality, consonant realization and connected speech. Accents influence these properties in systematic ways. A speaker may use a vowel realization that differs from the pronunciation patterns represented most heavily in a recognition system's development material. A consonant may be weakened, released differently or omitted in a particular phonological environment. These are ordinary features of language variation, not evidence that a speaker is unclear.

When evaluating speech-to-text accuracy across accents, the central question is whether recognition errors are distributed evenly across speakers and linguistic conditions. An overall WER can conceal a substantial difference between groups. For example, a dataset might contain two equally sized groups, with WER values of 8 percent and 20 percent. The combined figure would be 14 percent, which does not show the difference between those groups. Reporting subgroup results makes that pattern visible.

• Record the language variety and relevant dialect information.
• Include speakers with different speaking rates and pitch ranges.
• Distinguish accent variation from recording-quality problems.
• Examine repeated errors on particular sounds or word structures.

Reference transcripts and alignment conventions

A reference transcript is not simply a rough summary of what a speaker intended to say. It is the comparison standard, so its conventions must be explicit. Decide whether contractions are written in their contracted form, how filled pauses are represented, whether repetitions are retained and how partial words are handled. If one reference uses 'cannot' while another uses 'can't', a literal comparison may count a difference that has little practical significance.

Tokenization matters too. Hyphenated expressions, abbreviations, numbers and punctuation can produce different comparison outcomes depending on preprocessing. Most WER implementations normalize text before alignment, but normalization rules vary. A useful transcription tool test documents lowercasing, punctuation removal, number conversion and any other transformations. The same rules must apply to reference and system transcripts. Otherwise, the metric may reflect inconsistent text preparation as much as recognition behavior.

The three main error types in context

Substitutions can be especially consequential when they change a person's name, a technical term or the meaning of an instruction. Consider a reference sentence that says, 'The patient selected fifteen.' If the system writes 'fifty', the difference involves one word, but the contextual consequence may be substantial. This is why a low overall WER cannot automatically establish that a transcript is suitable for every use.

Deletions can remove negation, short function words or repeated material that helps preserve conversational meaning. Insertions can introduce words that the speaker never said, including plausible grammatical completions. Accented speech may also interact with recognition errors involving articles, plural endings and inflectional morphology. These categories should be recorded separately when possible, because they point toward different recognition problems and different downstream risks.

• Substitution: 'fifteen' becomes 'fifty'.
• Deletion: 'I did not approve it' becomes 'I did approve it'.
• Insertion: 'She arrived yesterday' becomes 'She had arrived yesterday'.
• Segmentation difference: one spoken expression is represented as a different number of written tokens.

Beyond the headline percentage

WER measures lexical differences, not every property of a usable transcript. It does not directly measure punctuation quality, sentence readability, capitalization, speaker labels or timestamp accuracy. A captioning workflow may depend heavily on all of these. A transcript with a relatively high WER might still preserve the main topic, while a transcript with a lower WER could contain a critical error in a proper name or negation.

A broader evaluation should add error severity and error distribution to the numerical result. Record whether an error changes meaning, affects a name, alters a number or disrupts speaker attribution. Also inspect the location of errors. Ten substitutions concentrated in a single difficult phrase may have different implications from ten substitutions spread throughout a conversation. These distinctions help teams decide whether a recognition system is appropriate for a particular task.

• Lexical accuracy and error type.
• Proper names, numbers and technical vocabulary.
• Speaker attribution and conversational turn boundaries.
• Punctuation, readability and sentence segmentation.
• Timestamp alignment where synchronized captions are required.

A repeatable protocol for evaluating transcripts

Begin with a collection of recordings that represents the intended environment. Include multiple speakers, different speaking rates and a range of recording conditions. The collection should also reflect the accents and language varieties expected in actual use. A small sample can help identify obvious problems, but it should not be treated as representative of a broad population. Recordings should have consistent reference transcripts and documented transcription conventions before comparisons begin.

Next, calculate WER for each recording and preserve the underlying substitution, deletion and insertion counts. Report individual recording results as well as an aggregate. Where the sample permits, group results by relevant speech characteristics without treating accent labels as complete descriptions of individual speakers. Review a selection of errors in context, then add a separate assessment of names, numbers, speaker attribution and readability. The result is a more informative picture of performance than a single percentage.

• Prepare a consistent reference transcript.
• Document normalization and tokenization rules.
• Calculate WER using the same method for every recording.
• Preserve individual error counts and recording-level results.
• Classify consequential errors and review them in context.

Choosing metrics for the intended task

The right evaluation depends on what the transcript will be used for. A searchable archive may prioritize recoverable content and consistent terminology. Live captions need timely delivery, readable segmentation and reliable speaker attribution. Research transcription may require preservation of disfluencies, repetitions and specialized vocabulary. These tasks share speech recognition technology, but their acceptable error profiles can differ substantially.

Product capabilities also vary by language, region, recording format and deployment configuration. A system may support an accent or language variety in one workflow but offer different behavior under another configuration. Check current documentation for supported languages, speaker diarization, timestamp output and available customization. For the next evaluation, prepare a 10-recording sample with reference transcripts, calculate WER per recording and add an error log for names, numbers and meaning-changing substitutions.

Questions readers ask