Frontier y Center: ¿Quién evalúa las evaluaciones?

The most interesting questions in AI are being asked by information theoreticians, focusing on providing context to emerging AI agents. A few weeks ago, we explored this with a blog about the Open Knowledge Format, a specification that formalizes the LLM-wiki pattern into a portable, interoperable format. This blog generated interest, so we’re continuing this series. Today, we hear from two members of Google Data Cloud’s frontier AI team, discussing how to systematically evaluate whether an agent can answer questions based on its context. A passing grade is the least interesting thing an exam can tell you. It says the student cleared the bar, leaving you in the dark about their failures, effortless passes, or what to teach next. Yet, this is how we evaluate AI agents. We run a fixed benchmark, calculate a score, and declare progress. We hand our agents a pass/fail exam when we actually need a map of their capabilities. For data agents, this map matters for data discovery in search and retrieval. Discovery is a needle in a haystack problem. Real users phrase their questions imperfectly, and inferring what datasets to retrieve presents a challenge. The interesting question is not whether the agent can pass, but how vague can the question get before the agent breaks. An exam cannot easily answer that, but a map can. Today, we share an approach rooted in information theory that we’ve been leveraging to add detail and nuance, i.e., fidelity, to benchmarks. This allows us to better understand agents’ performance. Along the way, the added fidelity exposed deeper issues with the quality of emergent evaluation cases. Difficulty, measured as information content, is often stratified into tiers. This can happen organically, e.g., pervasive and enduring failure scenarios are deemed difficult. Or it can be from labels applied by humans or machines categorizing some questions as easy or hard for an agent to answer correctly, based on the context provided in the query. While this kind of sentiment-based labeling is not the only way to label test cases, it’s frequently used despite its imperfections. Despite being an industry staple, assessing every evaluation case by hand is unrealistic at scale. What we need is a rigorous approach that can modulate the difficulty of evaluation cases. We’re iterating on a meta-benchmark we call Discovery Bench: a framework that modulates an evaluation case by generating “easy” and “hard” variations of every case. This allows us to audit how close or how far an agent is from succeeding in those cases. The lever for modulating the difficulty of an input query comes via a tried-and-trusted concept that’s present across information theory and machine learning: surprisal, or the likelihood of an output given a set of inputs. In our case, a query’s surprisal represents the uncertainty that remains about the correct dataset given the query. The thinking behind our approach is simple: A term or a phrase in an evaluation query has high informative power when it sharply distinguishes the target from everything else in the corpus. Therefore, we can adjust the difficulty of evaluation cases by adding or removing terms with varying levels of informative power. Let’s work through a real example from KramaBench, a publicly available benchmark. One of KramaBench’s datasets has information about orbiting satellites, and the example query from the suite includes the following text: …the total count of satellite major altitude changes for satellite 48445 during 2024 using TLE history. The token TLE is sharply distinguishing; it points almost uniquely at the TLE_____48445 table from the dataset. Strip it, and the query degrades to the count of satellite altitudes for satellite 48445, whose vague phrasing now matches density tables, precise-orbit files, and decay logs alike. Surprisal makes this quantitative: rare, pointed terms carry more bits than common ones. The remaining surprisal of a query is how much uncertainty is left about its answer. As surprisal approaches zero, the query has become specific enough to pinpoint exactly one dataset. The heart of the idea behind Discovery Bench is this refinement loop, which we call iterative surprisal-based query refinement, or iSQR, which generates cases with higher or lower informative power to test where an agent can start successfully answering the query. We have an F1 agent that’s built for recall (on Gemini 3.1 Pro). Running it against KramaBench and across the full sweep of ambiguity levels traces a curve. At high ambiguity, the curve shows an F1 of 0.34, 0.76 at neutral, and 0.81 at medium, and 0.78 at low. Two findings fall out immediately. First, the cliffs. This query scores a perfect F1 = 1.00 at neutral phrasing — and 0.00 at high ambiguity. It is the satellite-48445 case from above: drop the distinguishing token TLE and the agent loses the table entirely. Same query, same agent, same ground truth; one notch vaguer and it falls off a cliff. A static benchmark tests the neutral phrasing, stamps solved, and reports flat ground where there is a precipice. Pass/fail was particularly misleading in that it did not just miss the cliff, but it told us the terrain was level. Second, the sweet spot. For Discovery Agent, medium ambiguity beat neutral, and low ambiguity sometimes underperformed it. More specificity is not monotonically better for the system being evaluated; there is an optimal amount of steering. This is a graded, actionable signal. This is the how close, how hard texture we were missing from a scalar. It tells you where to hill-climb, or improve, the agent: in our case, straight at concrete failure modes like time-sharded tables (precision collapsing to ~8% as the agent over-retrieves 21 near-identical shards for a two-table answer) and context blow-up (F1 dropping from 0.75 to 0.32 once a query triggers long search chains). The map did not just say that the agent failed, but it said where, and why. Note that our hypothesis that less ambiguity and more context (via steering terms) should improve retrieval generally holds true, but for the specific Discovery Agent being exercised, the idiosyncratic “sweet spot” meaningfully highlighted trade-offs in its implementation. We’re not alone The field is converging on meta-benchmarking and exerting greater control of how we challenge and evaluate our agents. A growing body of work uses item response theory, the latent-ability model behind standardized testing, to treat difficulty as a measured quantity rather than a label: tinyBenchmarks and metabench show that a handful of informative items reproduce a model’s full score, and PSN-IRT turns the same lens on benchmark quality itself. Others audit the ground truth directly: MMLU-Redux found that 6.49% of Massive Multitask Language Understanding (MMLU) questions are mislabeled, and Platinum Benchmarks re-cleaned ten datasets to minimize both label errors and ambiguity — the same two axes we sweep for. And ambiguity is increasingly treated as intrinsic rather than noise: AmbigQA showed that a large fraction of real questions admit multiple readings, and later work finds that apparent hallucinations often stem from query ambiguity rather than model failure. What we have not seen elsewhere is the combination: information-theoretic ambiguity sweeping applied as a meta-benchmark over live enterprise data. A benchmark we trusted turned out to be broken We built our first evaluation on kramabench-astronomy, a benchmark established in the field, and one which other teams had already leaned on for their own evals. Teams derived benchmarks from this dataset, and we hypothesized subtle issues may have been introduced over time. When we actually read the benchmarks used by teams, with Gemini’s help, we found it was wrong in meaningful ways: ground-truth tables that did not answer their query, a question whose 124 sharded tables exceeded what some teams’ retrieval APIs could even return, months specified where exact dates were required. Quietly broken ground truth means quietly wrong conclusions not just for us, but for every prior analysis built on it. This is the generalized crux of the matter: an evaluation is itself an artifact that can be defective, and almost nobody evaluates it. We instrument the agent and trust the ruler, but where do we validate that the measuring stick makes sense? When two maps disagree Now the recursive turn: If difficulty is something we generate, then we need to evaluate the generator itself; we should not trust it blindly either. So we built the same ambiguity sweep two ways: steering terms from a pure-LLM guess, versus terms grounded in TF-IDF surprisal. The two disagreed violently. At high ambiguity, the LLM-built sweep scored the agent at F1 ≈ 0.34; the grounded sweep, ≈ 0.85. One of these maps is badly distorted. The grounded one, predictably, is the more robust: surprisal gives it a footing the free-running LLM lacks. This is evaluate your evals, made concrete. The information-theoretic lens does not only grade the agent along a continuous axis; it grades the benchmark’s own construction, and adjudicates between the two. Evaluate your evals We have spent years optimizing agents against rulers we never measured. The bitter irony is that better models make this worse: as agents clear coarse benchmarks, the score saturates near the top and the exam loses its ability to highlight where the agent can be improved. So the call to action is uncomfortable and overdue: evaluate your evals. Read your ground truth. Treat difficulty as a measured quantity, not a label: sweep it, plot it, find the bit-width where your system breaks. Ask not just did it pass? but how close was the miss, how hard was the pass, and would a slightly vaguer question have sent it off a cliff. Build evaluations that produce signals; not just verdicts. There is a genuine tension to sit with here. Difficulty-as-entropy is only as reliable as the model that estimates the entropy. There’s a risk that if we push too hard on a measurable proxy, we optimize the ruler instead of the agent. That is not a reason to retreat to pass/fail; it is a reason to keep the evaluator under the same scrutiny as what it is evaluating. The moment we stop asking who evaluates the evaluators is the moment our maps stop being useful again.