11 October 2026
Evidence reviewed: CoSE-E (28 September 2026), EVA-Bench (October 2026) and Falcon-ASR (7 October 2026).
The problem
A voice agent can produce a mostly correct transcript while still misunderstanding the request it needs to fulfil.
That distinction matters in multilingual enterprise systems. An incorrect technical term, action, identifier or other critical word may have a greater operational consequence than several transcription errors elsewhere in the conversation.
Word error rate (WER) remains a useful metric for automatic speech recognition (ASR). But it measures differences between a system's transcription and a reference transcript. It does not directly measure whether the agent understood the user's intent or completed the correct task.
Recent research makes a case for evaluating these outcomes separately.
What does code-switching change?
Code-switching occurs when a speaker alternates between languages during speech, sometimes within a single sentence.
In enterprise interactions, switching may involve technical terminology, service categories, product names or words associated with a particular workflow.
Consider this illustrative Hindi-English request:
"Mera VPN access reset karna hai."
The user wants a VPN access reset.
If an ASR system incorrectly interprets "reset" as "request", the resulting instruction could describe a different action.
This is a hypothetical example, not a measured model failure. It illustrates why word-level transcription errors have different consequences depending on the task.
For a transcription-only application, the principal objective may be accurately recording speech. For an automated service agent, the downstream decision also matters.
What the September 2026 CoSE-E research examines
On 28 September 2026, researchers from ServiceNow AI Research and a collaborator released CoSE-E: A Benchmark for Code-switched Speech Evaluation in Enterprise Settings as an arXiv preprint.
The benchmark focuses on code-switched speech in enterprise workflows, including HR and IT service management scenarios.
It contains 1,212 synthesised utterances across five language pairs:
- Spanish-English
- French-English
- Canadian French-English
- German-English
- Mandarin-English
The researchers evaluated eight ASR models and used multiple measures to examine the effects of transcription errors.
Importantly, the framework considers more than the difference between predicted and reference words.
It includes measures of semantic preservation and downstream answer correctness.
This makes the study relevant to teams developing speech-driven enterprise systems, where transcription is an intermediate step rather than the final product.
Source: CoSE-E, arXiv:2609.35645, submitted 28 September 2026.
Three levels of evaluation
1. Transcription accuracy
WER measures substitutions, deletions and insertions relative to a reference transcript, divided by the reference word count.
It answers a clear question: how different is the recognised text from the reference?
WER is valuable for comparing recognisers under consistent test conditions. But a single aggregate score can conceal differences between language pairs, acoustic environments, speaker groups and high-consequence terms.
For multilingual systems, evaluation should also define how transcripts handle scripts, borrowed vocabulary, mixed-language expressions and normalisation.
2. Meaning preservation
A transcription error does not always change the intended meaning.
Conversely, one incorrect term can materially change what the user is asking for.
A semantic evaluation asks whether the recognised content preserves the information needed by the downstream system.
Relevant cases may include:
- Technical commands and action words
- Product or service names
- Amounts, dates and numbers
- Account or ticket references
- Negation and corrections
- Switches between languages around critical terms
These checks require reference labels and a clear policy for determining semantic equivalence.
A semantic metric should not automatically replace WER. Both measurements provide useful, distinct evidence.
3. Downstream task correctness
For a voice agent, the final question is whether the system performs the intended action.
A support agent may need to classify an issue, retrieve information, create a ticket, trigger an authorised workflow or ask for clarification.
An evaluation should therefore distinguish between:
- The speech that was recognised
- The meaning the system extracted
- The decision or action the system produced
- The verified outcome
This is particularly important when a fluent response can make an incorrect action sound convincing.
The evaluation must be designed around the actual task. A transcription metric alone cannot establish that a downstream workflow succeeded.
Related October 2026 research: EVA-Bench
ServiceNow AI Research's EVA-Bench: A New End-to-end Framework for Evaluating Voice Agents broadens the evaluation perspective.
Its published October 2026 description introduces separate measures for task-oriented accuracy and conversational experience.
These cover areas including task completion, faithfulness, audio fidelity, conversation progression, conciseness and turn-taking timing.
The research describes 213 scenarios across three enterprise domains, with evaluations of 12 systems and controlled accent and noise perturbations.
The central implication is that a voice agent's quality cannot be reduced to the accuracy of any one component.
For example, an agent might correctly recognise the request but interrupt at an inappropriate moment. Alternatively, it may respond smoothly while failing to complete the requested task.
These are different failure modes and require different evaluation evidence.
Source: EVA-Bench, ServiceNow AI Research, October 2026.
Regional speech creates another evaluation problem
Research on one language variety cannot automatically establish performance across every region or deployment environment.
A recent example comes from the Technology Innovation Institute's Falcon-ASR release on 7 October 2026.
Alongside public Arabic benchmark testing, the team carried out additional evaluation on held-out Emirati and Gulf recordings with human-validated transcripts.
The work shows an approach to supplementing public evaluation coverage with regionally relevant testing.
This example concerns Arabic and Emirati speech. It is not evidence about Hindi, Indian English or other Indic language performance.
The broader methodological question remains useful: does the evaluation data reflect the language variety and recording conditions that matter for the proposed deployment?
Source: Introducing Falcon ASR, Technology Innovation Institute, 7 October 2026.
What should an enterprise buyer require from an evaluation dataset?
Before sourcing an existing dataset or commissioning new collection, the buyer should define the model, workflow and failure conditions.
A practical requirement should answer the following questions.
What system is being evaluated?
An ASR model, a speech-to-text-to-agent pipeline, and a fully speech-to-speech agent do not have identical evaluation requirements.
Which language combinations matter?
Specify the actual languages, dialects and directions of code-switching. A general multilingual label does not establish coverage of the required combinations.
Which errors matter operationally?
Identify the terms and decisions most likely to change task outcomes, including commands, numbers, names, negation and domain-specific vocabulary.
What will the audio sound like?
Document the expected device, channel, noise conditions, number of speakers, overlap, interruptions and conversational structure.
What annotations are necessary?
Depending on the evaluation, these may include reference transcripts, language boundaries, speaker labels, timestamps, intent labels, critical entities, expected actions and task outcomes.
How will the labels be checked?
Use a documented annotation policy, appropriate linguistic reviewers, quality sampling and an adjudication process for disagreements.
Is the data independent?
Evaluation records should be appropriately separated from training and tuning material. Potential duplicates, benchmark contamination and reuse should be investigated.
Can the dataset be used for the intended purpose?
The buyer must establish provenance, licensing authority, applicable consent and usage restrictions. Technical suitability does not establish commercial permission.
Together, these questions help turn an abstract request for multilingual audio into a testable evaluation specification.
Synthetic and human speech are not interchangeable evidence
Synthetic speech can be valuable for controlled tests, reproducibility and constructing specific language or task combinations.
It can also help isolate individual failure conditions.
However, synthetic evaluation does.