Automatic Captioning of Overlapping Speakers: A Repeatable Evaluation Protocol

Why simultaneous speech complicates captions

Many conversations are organized around turn-taking, with one person speaking while another listens. Real conversations also contain interruptions, backchannels, brief acknowledgments and longer stretches of simultaneous speech. These events create a difficult problem for automatic captioning. The system must recognize words from acoustic signals that occupy the same time interval, determine which speaker produced each utterance and represent both contributions in a readable visual format.

The challenge is not limited to recognizing individual words. Two speakers may have similar pitch ranges, overlapping vowel sounds or different speaking rates. A short acknowledgment such as 'right' may occur while another person continues a sentence. If the system suppresses the acknowledgment, the caption may lose conversational information. If it assigns the word to the wrong speaker, the transcript may misrepresent who agreed, objected or interrupted.

Separate recognition, diarization and display

A captioning system performs several related tasks. Automatic speech recognition converts acoustic input into words. Speaker diarization identifies who is speaking and when, usually by assigning speaker labels to segments. Caption rendering then determines how recognized speech is divided and displayed. Some products combine these functions into one workflow, while others expose separate controls or processing stages. Their outputs should still be evaluated as distinct dimensions.

This distinction matters because a system can recognize the words correctly but attribute them incorrectly. It can also identify two speakers accurately while omitting one contribution during an overlap. Finally, it may preserve both contributions in a transcript but display them in a confusing order. A caption quality checklist should therefore avoid collapsing all problems into a single judgment about transcription accuracy.

• Word recognition: are the spoken words represented accurately?
• Speaker diarization: is each contribution attributed correctly?
• Temporal alignment: do the timestamps reflect actual speech?
• Caption layout: can viewers follow the displayed information?
• Completeness: are meaningful simultaneous contributions preserved?

Build an evaluation set around overlap types

A useful evaluation set should contain several forms of overlapping speech. One speaker may interrupt another, two people may begin speaking simultaneously, or a listener may give short acknowledgments while a longer turn continues. Group discussion can create three-way overlap, while remote meetings may add latency and audio-quality problems. These situations place different demands on recognition and speaker attribution.

For a small initial protocol, prepare 10 short segments, each containing a clearly defined overlap event. Include different overlap durations and conversational functions. For example, one segment might contain a two-second interruption, another a brief 'yes' during a continuing explanation, and another two speakers beginning a sentence at nearly the same time. The reference material should preserve both speech streams and identify their timing separately.

• Interruptions during an unfinished sentence.
• Brief acknowledgments during another speaker's turn.
• Two speakers starting at the same time.
• Three-person discussion with partially simultaneous turns.
• Remote conversation with small timing delays.

Create a reference that preserves both voices

A reference transcript for overlapping speech needs more than a single linear sequence of words. Each speaker should have a separate annotated timeline, with word or phrase boundaries appropriate to the intended evaluation. The reference should identify where simultaneous speech begins and ends. If two speakers talk from 12.4 to 14.1 seconds, both contributions should retain that interval even when their words are represented on separate lines.

Use consistent speaker labels such as Speaker A and Speaker B. Avoid assigning names unless the speakers are reliably identified and the use of those names is appropriate. Mark unintelligible portions according to a defined convention rather than guessing missing words. Also record meaningful vocalizations when they affect conversational interpretation. The reference should distinguish a genuine interruption from an ordinary change of turn, since those events have different timing and interactional properties.

A repeatable caption evaluation checklist

Evaluate each segment in the same order. Start with the audio and reference transcript, then examine the generated captions. Identify every meaningful contribution during the overlap and determine whether it appears. Compare speaker labels, word content and timing separately. This prevents a visually tidy caption from receiving an overly favorable interpretation when one participant's words have disappeared.

Timing deserves particular attention. A caption may contain the correct words but appear after the speaker has finished. Conversely, a caption may appear early enough to support reading but contain text that has not yet been spoken. Decide whether the intended application prioritizes live display, post-event captions or a searchable transcript. Those workflows may use different timing tolerances and display conventions.

• Check whether both speakers' meaningful words are represented.
• Identify substitutions, deletions and inserted words.
• Compare speaker labels with the reference timeline.
• Inspect caption onset and offset times.
• Record whether overlapping lines remain readable.
• Note whether one speaker consistently receives less complete captions.

Readability when two captions compete for space

Preserving every spoken word is only one part of caption quality. If two lines appear simultaneously, their visual arrangement should help viewers distinguish the speakers. Labels, line breaks and consistent positioning can support this distinction, but conventions differ between platforms and caption standards. A system that places two utterances in a single sequence may make the conversation appear sequential even when the speakers were talking at the same time.

Caption density also affects comprehension. Long overlapping sentences can produce too much text for a viewer to process comfortably. A practical evaluation should note whether lines are divided at meaningful phrase boundaries and whether speaker changes are easy to follow. For example, two simultaneous captions of 12 words each may be technically complete but visually demanding if they appear together for only two seconds. The reading conditions and target audience matter.

Interpret the results without hiding trade-offs

A useful evaluation summary separates lexical completeness, speaker attribution, timing and display readability. These dimensions may reveal competing strengths. One configuration may preserve more overlapping words but produce crowded captions. Another may display a cleaner sequence while omitting brief acknowledgments. Reporting these differences allows an accessibility team to consider the actual viewing environment and the importance of conversational detail.

Interpretation should also account for the source audio. Microphone placement, room reverberation, background noise and channel mixing can affect how clearly simultaneous speakers are represented. Remote meeting platforms may introduce latency, compression or unequal audio levels. Document these conditions because they influence recognition behavior. Product capabilities may differ across languages, regions, caption modes and software versions, so consult current documentation before applying findings to a deployment decision.

Put the protocol into a practical workflow

A compact protocol can begin with 10 annotated overlap segments and expand as recurring error patterns become clear. Keep the reference transcript, source audio, timestamps and output settings together. Use a separate record for each overlap event, noting whether it involves interruption, acknowledgment, simultaneous starts or multiple speakers. Record both successful and unsuccessful behavior in descriptive terms, without reducing the entire evaluation to one impression.

When a captioning tool is intended for accessibility, involve people familiar with the audience's viewing requirements when deciding how to interpret readability and timing. Relevant accessibility standards and organizational policies may specify additional requirements. The next practical step is to prepare five two-speaker clips and five clips containing interruptions or acknowledgments, then annotate both speaker timelines before generating any comparison summary.

Questions readers ask