MTurk is shutting down. Voice AI still needs humans

Sonexis Team··

Published when the company operated as Sonexis. Sonexis is now NaraDatum.

Amazon Mechanical Turk shuts down on 30 September 2026.

For voice AI teams using humans to listen, compare, label or validate speech, there's a more useful question than what replaces MTurk?

Who listens now?

Because the need for human judgement didn't disappear.

Actually, voice AI seems to be moving in the opposite direction.

The models are getting better.

The evaluation problem is getting harder.

A clean voice isn't the same as a good conversation

TTS demos are getting ridiculously good.

Give a model clean text, a quiet environment and a few seconds of audio and it can sound great.

Production isn't like that.

Someone says:

haan actually I wanted the premium plan... wait, what was the other one?

They hesitate.

Switch language.

Interrupt the agent.

Say a product name in English inside a Hindi sentence.

Speak through a terrible phone line.

Correct themselves halfway through.

Now the question isn't just:

did the model generate natural speech?

It's:

did the conversation still work?

That difference matters.

Artificial Analysis launched its Speech Agent Arena on 24 August (see Sources).

Instead of just scoring generated clips, humans actually speak with two hidden voice systems while doing things like booking appointments or ordering food.

Then they choose which conversation they preferred.

For agentic tasks, the benchmark separately checks whether the job actually got done.

That's much closer to the real problem.

The benchmarks are changing for a reason

Artificial Analysis also changed its Speech-to-Speech Index this month.

The current index gives equal weight to:

  • speech reasoning
  • agentic performance
  • human arena preference
  • task success

That's pretty telling.

A technically impressive voice system can still be a bad system to talk to.

And one score can't explain why.

The same thing is happening in TTS research.

A large 2026 evaluation of Indian-language TTS, Preferences of a Voice-First Nation (Anand et al., see Sources), tested seven systems across more than 5,000 native and code-mixed sentences in 10 Indic languages.

More than 1,900 native raters produced over 120,000 pairwise comparisons.

They didn't just ask whether a voice sounded "good".

They separately judged:

  • intelligibility
  • expressiveness
  • voice quality
  • liveliness
  • noise
  • hallucinations

Because those things don't move together.

A system can sound expressive and still get the words wrong.

It can sound clean and still feel dead.

It can perform well in Hindi and fall apart once English appears halfway through the sentence.

That's the bit aggregate scores hide.

India makes this painfully obvious

Take a normal customer call in India.

The caller starts in Hindi.

Uses an English product name.

Has a regional accent.

Switches back.

The connection drops for half a second.

Someone talks in the background.

They interrupt.

The agent interrupts back.

Then the caller says:

nahi nahi, mera matlab woh wala plan tha

None of this is unusual.

But a neat evaluation set can quietly remove almost all of it.

That's dangerous.

Because then you haven't necessarily tested the system your customer will actually use.

You've tested the clean version of the world.

And now MTurk disappears

Amazon has confirmed that MTurk permanently closes on 30 September 2026.

The MTurk worker option also disappears from SageMaker Ground Truth and Amazon Augmented AI.

So teams that still have human review workflows running through that infrastructure need to make a decision.

But replacing MTurk with another giant anonymous crowd isn't automatically the right answer.

For speech, we'd start somewhere else.

Ask what humans are actually needed for.

Maybe it's pronunciation.

Maybe it's preference between two voices.

Maybe it's checking hallucinated speech.

Maybe it's whether Hinglish sounds natural to someone who actually speaks Hinglish.

Maybe it's a regional accent.

Maybe it's whether a voice agent can survive ten minutes of real interruptions and still complete the task.

Those are different jobs.

They need different people.

"Human evaluation" can also be bad

Humans aren't magic.

Give the wrong listener the wrong task and you still get bad data.

If someone doesn't speak the language well, their naturalness rating might not mean much.

If they don't know what they're listening for, two evaluators may answer completely different questions.

If you don't track who evaluated what, you can't properly interpret the result later.

And if consent, usage rights or provenance are unclear, technically useful data can become commercially unusable.

So the replacement for a crowd platform shouldn't just be another crowd.

It should be a better-defined workflow.

Who is evaluating?

Why are they qualified?

What exactly are they judging?

What language, accent and environment does the test represent?

What does the result prove?

And just as importantly:

what doesn't it prove?

Automated judges will take more of this work

They should.

Nobody needs humans manually checking every measurable failure forever.

Automated speech evaluation is improving fast.

There is already research trying to train systems that reproduce human naturalness preferences from large human-feedback datasets.

That's useful.

But notice where those systems get their ground truth.

Humans.

So the likely future isn't:

human evaluation vs automated evaluation

It's both.

Use machines where the signal is reliable and cheap.

Use humans where language, perception, cultural context or actual conversational experience matters.

Then keep checking whether the automated judge still represents what people care about.

So what actually replaces MTurk?

Probably not one thing.

And that's fine.

Some teams will need specialist native-language evaluators.

Some will need managed listening panels.

Some will already have the right internal workforce.

Some will need an existing dataset.

Some will need a small held-out evaluation set built around a specific production failure.

Some won't need new human data at all.

The mistake would be starting with:

we need 10,000 people

before asking:

what are we actually trying to learn?

That's the layer Sonexis works on.

We help AI teams find, verify and evaluate human data for multilingual production systems, and scope new collection when suitable existing data doesn't cover the requirement.

Not every MTurk workflow belongs with us.

Generic microtasks probably don't.

But if your workflow involves people listening to speech, judging voice quality, testing multilingual behaviour or finding failures your automated evals keep missing, it's worth reviewing the workflow before simply moving it somewhere else.

30 September isn't far away.

Better to work out what the humans were actually doing before replacing them.

Sources

← Back to Blog