Business Owner Concerns
Stanford research questions whether widely used AI tests measure what their labels promise. For businesses, a high score may be weaker purchasing evidence than it appears.
By Margaret Lund, US Business Correspondent · · 6 min read
AI benchmark scores may not justify the purchasing decisions businesses make from them. Stanford researchers report recurring measurement problems across 56 widely used benchmarks, while a separate study examines why multilingual safety scores can blur model weaknesses with flaws in the tests themselves. The findings challenge the interpretation of scores, not the usefulness of testing.
In an account published by Stanford HAI, assistant professor Sanmi Koyejo and graduate student Sang Truong describe two studies applying measurement science to AI evaluation. The institute partially supported the research. Its account says the papers are due to be presented in October at the Third Annual Conference on Language Modeling in San Francisco; it does not specify a year.
One study examined whether benchmarks measure the properties they claim to assess, and whether tests supposedly assessing the same property agree. Koyejo says the researchers repeatedly found problems across 56 widely used benchmarks. That is evidence of a recurring weakness, not a finding that every benchmark is worthless. The account does not quantify how many failed particular validity checks.
The researchers illustrate the problem with BBQ, a multiple-choice benchmark intended to assess bias. Some questions deliberately supply too little information to support a judgement about people. Selecting an answer based on a gender stereotype rather than acknowledging the missing information is treated as evidence of bias.
But recognising that information is missing also requires comprehension. Koyejo argues that a biased model adept at recognising the question’s structure can receive an apparently favourable result, while an unbiased model that misunderstands it can receive an unfavourable one. The score cannot reliably distinguish the explanations. A test labelled as measuring bias may therefore also be measuring something else.
For a company comparing AI suppliers, that distinction changes what constitutes evidence. A better bias score does not, by itself, establish that a model will make less biased judgements in the intended workflow. It may partly reflect superior handling of the test format. The commercial risk is paying for an improvement in the advertised property when the measured improvement concerns another capability.
Most public AI benchmarks are still created by academics, according to Koyejo, although companies and other organisations also publish them. Developers adopt these tests to compare models and track progress in reasoning, safety and other capabilities. Benchmark creators, meanwhile, have an interest in their tests becoming widely accepted standards. Adoption and scientific validity are different achievements.
The stakes extend beyond technical reputation. Stanford HAI says benchmark results influence billions of dollars in investment and increasingly inform regulation and government procurement. Its account identifies no particular purchasing contract or regulatory provision. The research therefore raises a question about the quality of evidence behind such decisions; it does not establish that a named buyer or regulator made the wrong choice.
The researchers borrow two concepts from psychometrics, the discipline used to assess human abilities. Convergent validity asks whether tests of the same property produce similar results. Discriminant validity asks whether tests of different properties actually distinguish between them. Together, these checks examine whether a benchmark’s label is a defensible description of its output.
That matters because agreement alone is insufficient. Several supposedly different tests might reward the same underlying skill, giving buyers an appearance of broad assurance without genuinely independent evidence about safety, reasoning and bias. Conversely, conflicting results from tests of the same property leave buyers uncertain about which result supports their decision. More scores need not mean more knowledge.
The second study addresses a related problem in multilingual safety testing. According to Truong, nearly all major safety benchmarks are written in English. Translating them can introduce two separate changes: a model’s safeguards may become less effective, and the test itself may change in wording or difficulty.
A lower translated safety score can consequently have more than one explanation. An attack that fails in English may succeed in another language, but a translation may also probe the model differently from the original. Truong identifies limited non-English training material and imprecise translation among the possible contributing factors. A single aggregate score cannot separate them.
For businesses deploying multilingual assistants, this is a distinct extension of the checking problem in AI translation. Here, translation affects the instrument used to certify performance, not merely the output delivered to a customer. An English-language result cannot, on its own, resolve either the model’s safety in another language or the reliability of the translated assessment.
Truong’s position is that benchmark quality has become consequential because rankings affect model valuations, developers’ financial decisions and policymaking. He argues for interpreting scores through measurement science rather than treating the number as self-explanatory. The source provides researchers’ reactions, but no response from model developers, benchmark publishers, business buyers or regulators.
Koyejo’s argument is narrower than a call to abandon benchmarks. He supports their use in procurement and regulation, while questioning whether anyone has established that a particular score supports the decision being made. In his assessment, that check is usually missing. This is a criticism of the connection between evidence and decision, not an argument against quantitative evaluation.
On multilingual safety, Koyejo says measurement models allow researchers to separate several influences: safety robustness across languages, the difficulty of the challenge, general language-processing difficulty and a prompt’s ability to reveal cross-language safety gaps. That decomposition could make a poor result more informative. A supplier facing a language-understanding problem has a different issue to address from one whose safeguards fail despite understanding the request.
The distinction also limits what can reasonably be inferred from neighbouring kinds of tests. A broad safety score does not settle a specific operational question such as whether an automated purchasing system resists hostile instructions. Our coverage of prompt-injection tests for automated purchases concerns that narrower business exposure. The relevant evidence must match the behaviour being relied upon.
Murray Wilkinson, founder of Found Revenue, an AI business services provider with three UK locations, says owners raising this worry should start by listing where AI already touches their business, then set clear checks on cost, accuracy and data before committing further.
The largest missing check, in Koyejo’s account, is against actual deployment. He argues that benchmark scores predict how systems will behave, yet those predictions are rarely recorded in a form that can later be compared with observed outcomes. Comparing tests only with other tests leaves their relationship with business reality unresolved.
For procurement, the practical consequence is specific: an evaluation needs an explicit claim attached to it. If a buyer treats a score as evidence that a customer-facing assistant will behave safely in a particular language, that interpretation needs to be testable against the deployed system. Otherwise, a later failure cannot readily reveal whether the model disappointed, the benchmark misled or the purchasing inference was too broad.
Regulatory interpretation requires the same separation. Questions about unresolved AI safeguards in the UK cyber bill sit alongside, rather than replace, the question of whether technical evidence is sound. The Stanford account announces no UK or US legal requirement and offers no compliance certification. Its contribution is methodological: a score needs validation before it can bear the weight of an important decision.
For UK and US business owners, the exposure lies in buying assurance that a score cannot actually provide. A supplier comparison may influence contract value, the amount of human oversight retained and which customer interactions are automated. If the test mixes bias with comprehension, or language difficulty with safety, those decisions can rest on the wrong interpretation. Multilingual deployments add another uncertainty: translated assessments may not be equivalent to their English originals. The useful distinction is between evidence that ranks a model and evidence that supports a particular use. They may overlap, but this research gives buyers reason not to assume they do.
Original reporting by AI Business Reality Check. Primary source: Stanford HAI.
