Multilingual Evaluation Mesh
One evaluation fabric that proves an assistant is as good in Tamil, Swahili and Portuguese as it is in English.
40+
Locales with native golden sets
down 80%
Undetected quality regressions
0.85+ kappa
Judge-human agreement
The problem
Most assistants are measured in English and shipped in forty languages. Quality silently collapses in the long tail: idioms break intent detection, transliteration breaks retrieval, and script-mixed queries break everything. Nobody notices until a market complains, because there is no per-language ground truth to notice with.
This is the single highest-leverage unlock for any product that wants to be global. Every downstream agent, safety filter and ranking model inherits the blind spot, so fixing measurement first compounds across the whole platform.
Agent design
- A generator agent proposes candidate eval items per locale from real query distributions, never from translated English prompts.
- A native-reviewer surface lets contracted speakers accept, rewrite or reject each item, so the golden set is authored locally and only assembled centrally.
- A judge ensemble scores responses with a locale-specific rubric, and disagreements between judges route to a human instead of averaging away.
- A regression runner replays every locale on each model or prompt change and publishes a per-language scorecard, not a single global number.
Guardrails
- No locale ships on machine-translated evaluation data; a language without native-reviewed items is marked unmeasured, not passing.
- Judges are themselves evaluated against human labels per locale, and a judge below agreement threshold is retired for that language.
- Aggregate scores are never published without the worst-performing locale beside them.
Why a single quality number is a lie
A 92% aggregate score usually means 97% in English and 61% in the five languages with the fewest evaluators. The average hides exactly the users you were trying to reach by going global in the first place.
The first product decision here is not a model decision. It is refusing to ship a dashboard that can average a language into invisibility.
Sourcing ground truth without a thousand-person team
Native review is expensive per item, so the agent should spend human attention only where it changes an outcome. Cluster the traffic, sample across intents rather than uniformly, and put reviewers on the items where judge confidence is lowest.
In practice that turns a hundred-thousand-item ambition into a two-thousand-item golden set per locale that actually gets maintained.
How I would sequence it
Quarter one: three locales end to end, including the reviewer tooling and the release gate. Prove that a red locale can block a launch and that the organisation accepts it.
Quarter two: scale the pipeline to the top twenty markets and hand the rubric authoring to regional teams. Central platform owns the mesh, regions own the meaning of quality in their language.