Finding the Rhythm of the Enterprise: A new SOTA on the BEAVER Benchmark
Introduction
Enterprises have a rhythm. A semiconductor company may investigate thousands of yield excursions, compare thousands of lots, or trace problems across products, tools and process steps. The exact questions change, but the patterns of analysis repeat.
That repetition creates something valuable: history. Every successful analysis captures a little of how the enterprise actually works with its data—what tables matter, how they relate, and how business questions are translated into computation. A new question may never have been asked before, yet resemble hundreds of problems the enterprise has already solved.
We tested whether an AI agent could take advantage of this accumulated history. On BEAVER, a text-to-SQL benchmark built from real enterprise data warehouses, the best previously published result is 11.4%. On a stratified sample of 300 questions, an agent augmented with Enterprise Knowledge (EK)—a searchable history of previously answered questions and their successful SQL—achieved 33.3% execution accuracy, roughly 3× the previous state of the art.
We tested eight other interventions, including additional annotations, more deliberation and a model upgrade. None produced a measurable improvement. The biggest gain came not from making the model smarter, but from helping it find the rhythm of the enterprise: recognizing patterns in how the organization had successfully solved related problems before.
The implication goes beyond SQL. Enterprises repeatedly investigate similar anomalies, make similar decisions and execute similar workflows. Their history contains patterns for how work gets done. AI that can discover and reuse those patterns does not have to approach every problem as if the enterprise were starting from scratch.
Why enterprise text-to-SQL is hard
Text-to-SQL promises a natural-language interface to a database: ask a question, get an answer, no query language required. Public benchmarks have made rapid progress toward this promise, and leaderboard numbers on curated academic schemas are now very high. Enterprise data warehouses look nothing like those schemas.
A production warehouse accumulates tables the way an organization accumulates process: by addition, rarely by refactor. Names abbreviate inconsistently, foreign keys are frequently undeclared even when the relationship is real, and the same business concept is often represented in several tables that are not obviously interchangeable. A model that writes syntactically correct SQL against such a schema can still fail in a way that is easy to miss: the query runs, returns a well-formed table, and answers a different question than the one that was asked. Getting the SQL to execute is necessary and, in this setting, far from sufficient. The harder problem is deciding what the question means against a schema that offers no shortcuts for finding out.
That distinction, between generating syntactically valid SQL and correctly interpreting the question and the schema, motivates everything that follows. The benchmark used to study it, and the result obtained on it, are both about interpretation, not generation.
The BEAVER benchmark
Spider and BIRD, the two benchmarks that most shaped text-to-SQL research over the last several years, are built from synthesized or curated schemas designed to be tractable: modest table counts, clean naming, foreign keys that are present when they should be. BEAVER departs from that lineage deliberately. It is constructed from three real production data warehouses, and its difficulty is a property of the schemas themselves rather than of query complexity for its own sake.
| database | tables | columns | declared foreign keys | public questions |
|---|---|---|---|---|
dw |
97 | 1,530 | 0 | 5,787 |
neutron |
175 | 819 | 163 | 1,017 |
nova |
110 | 1,322 | 25 | 1,053 |
The three warehouses contribute 7,857 publicly released question–SQL pairs between them; a further private test set is withheld and scored only through the leaderboard. Our evaluation, the bank, and every re-run baseline draw from the public pool alone.
dw declares no foreign keys at all and contains 17 tables whose names match %subject%, with no structural signal for which one a given question intends. Seventy-eight percent of the 300-question sample used here requires three or more joins, and the median question runs 76 words, long enough to function as a specification written in prose rather than a natural query:
"For each academic year in which the Mathematics department offered subjects in the current term, what is the department name, the variance and standard deviation (using STDDEV only and never STDDEV_POP) of the total units for subjects offered in that year and department…"
Scoring is execution accuracy, and the comparator is stricter than the prose suggests. Rows compare as a set, column order and count must match the gold result exactly, and values compare as raw strings after stringification, so 1 and 1.0 are counted as distinct answers. There is no partial credit: a query that gets every clause right except the grain of aggregation scores zero, identically to a query that fails to execute.
The resulting failure profile is informative on its own. Of 263 misses in the reference configuration (no EK augmentation), only 3.8% represent a failure to produce working SQL at all. The remaining 96% executed cleanly against a real enterprise database and returned a well-formed table answering a subtly different question than the one asked; 54% of misses had the correct columns and an incorrect row count. Of those wrong-row-count cases, the majority return too many rows via an unintended join multiplication, and the remainder return too few via an over-aggressive filter. This is a benchmark about question interpretation and query grain, well before it is a benchmark about SQL syntax.
How we measure
A short glossary, since some of these terms are used precisely in what follows:
- EX (execution accuracy). The fraction of questions for which the predicted SQL, executed against the live database, returns a result set identical to gold's under the comparator described above. All-or-nothing per question.
- SC@k (self-consistency at k). The model is sampled k times independently; the answer returned by the largest cluster of matching predictions is taken as the final answer. SC@1 is a single pass with no voting.
- Pass@k (best-of-k). The oracle-selection upper bound: a question counts as solved if any of k independent samples matches gold. It is not a deployable metric (it assumes a perfect selector), but it bounds how much headroom additional sampling could realize.
- p-value. Throughout, statistical comparisons are paired on the same 300 questions and tested with McNemar's exact two-sided test on the discordant pairs, reported alongside the number of questions fixed and broken by the intervention, since with effects this small the two counts are more informative than the significance verdict alone.
Evaluation setup. BEAVER's public question pool runs to several thousand questions per database, and this study compares many arms (competing methods, model sweeps, bank-size sweeps, and test-time-compute budgets), each of which must be re-run in full. Scoring every arm on the entire pool would be prohibitively slow and expensive, so a fixed, representative subset was drawn once and reused across every configuration. Three hundred questions were drawn by stratified sampling, 100 per database, matched to the benchmark's own category × detailed-category × has-domain-knowledge labels so the subset preserves the difficulty mix of the full pool rather than skewing toward easy or hard questions, and committed as a fixed manifest so that every configuration compared here runs the identical questions and every comparison can be paired question-by-question. The strongest available open competing method, ReFoRCE, was re-run on this same set of 300 rather than compared against its published leaderboard number (the reasoning for that choice is the subject of the next section). The Enterprise Knowledge bank, introduced in full below, holds every publicly released question for the connected database except the 300 under evaluation, which are excluded by identifier.
The headline result
Every method re-run on the same 300 questions, in the same setting, with no hints and no annotations. "As published" figures are shown for reference, not used in the comparison. The 33.3% entry is our system, a single CRAFT agent with EK augmentation on a GPT-5.6 Luna backbone.
The comparison that establishes the headline result is like-for-like. Four competing methods were re-run on the same 300 questions rather than comparing against quoted leaderboard figures, because a leaderboard figure is computed on a different, private set of questions and any gap could be a property of the sample rather than of the method. On our 300, the strongest competing method, ReFoRCE, scores 13.7%, while our own system, a single CRAFT agent with EK augmentation on a GPT-5.6 Luna backbone, reaches 33.3%. Each of the four re-run methods lands within +2.3 to +7.1 points of its published figure, and the relative ordering is preserved: ReFoRCE remains the strongest of the four here, as it is on the public leaderboard. Four independently built systems shifting in the same direction by a similar margin is what a marginally easier draw from the same underlying distribution looks like, and that advantage applies equally to our own result. The gap on identical questions is therefore 19.7 points (two-proportion z = 5.7, p = 1.3e-8), and a submission to the private leaderboard should be expected to score below 33.3%.
Two findings support the analysis that follows. First, the entire improvement is attributable to one intervention, EK augmentation (the retrieval of worked precedent on demand), and not to any of the eight further interventions tested, including two annotation oracles and a model upgrade. Second, the gain is interpretive rather than structural: an oracle that hands the agent the correct tables outright, reported among the null results below, produces no measurable improvement, so what precedent transfers is how to read the question, not which tables to use.
Enterprise Knowledge augmentation
The system under test runs a single agent with read-only access to one database, discovering the schema and locating relevant tables entirely by querying at runtime; there is no separate schema-search facility. EK augmentation adds one further capability: a searchable bank of previously answered question–SQL pairs for the same database, consulted on demand once the agent has read the question and formed a view of what precedent would help. Search is two-stage: a first call returns compact, ranked hits carrying no SQL, filtered on the structure of the question (the tables involved, the query shape, the join count); a second, deliberate call reads one hit in full. This keeps the context from growing in proportion to the size of the bank even as the agent consults it dozens of times in a run.
The bank holds every publicly released question for the connected database except the 300 under evaluation, which are excluded by identifier.
The result of adding this one capability is an improvement from 12.3% to 33.3% (66 fixed, 3 broken, p = 1.9e-16), with gains on every database and the largest gain where the schema is least tractable:
Enterprise Knowledge helps most where the schema offers the least structural signal: dw, with zero declared foreign keys, gains 26 points.
| database | zero-shot | with EK |
|---|---|---|
dw (97 tables, zero foreign keys) |
5.0% | 31.0% |
neutron (175 tables, 163 foreign keys) |
18.0% | 35.0% |
nova (110 tables, 25 foreign keys) |
14.0% | 34.0% |
Adoption was near-total: 299 of 300 runs searched the bank and read at least one example. The gains concentrate exactly where the system previously struggled most. Gold queries requiring three joins improved from 11.9% to 47.6%, CTE-shaped queries from 8.7% to 46.4%, and 80–150 word questions from 6.9% to 37.1%. Queries under 40 words were already handled well and did not change.
A 21-point improvement from a single intervention warrants scrutiny, so the principal contamination risks were examined directly. Exclusion by identifier does not entail exclusion by answer: 7 of the 300 evaluation questions have a byte-identical gold SQL present in the bank under a different identifier. Removing all seven and rescoring the remaining 293 yields 32.4%, essentially unchanged. Separately, of 296 predictions produced under EK augmentation, exactly one is byte-identical to a bank entry. The agent is retrieving and adapting precedent; it is not reproducing it verbatim.
Scaling with corpus size
A full bank of several thousand examples per database represents a genuine adoption cost, so its size was varied to see how much of the effect a smaller corpus could deliver. The capped banks were built by nesting rather than by drawing fresh each time. The bank was capped at 100, 200, 500, and 1,000 examples per database, and each of those banks was taken as a successive prefix of a single seeded shuffle of that database's public pool. Each larger bank therefore strictly contains the smaller one (the 200-bank is the 100-bank plus the next 100 examples, and so on up to the full pool), so the change between two adjacent sizes reflects only the examples added, not a more or less favorable draw. The same held-out 300 are excluded from every bank regardless of size.
The curve is logarithmic and has not saturated at the largest bank size tested; a searchable bank of only 100 examples per database already outperforms 15 examples embedded directly in the prompt. Bank sizes are per database, not pooled across the three.
| bank size (per database) | accuracy |
|---|---|
| 100 | 21.3% |
| 200 | 22.3% |
| 500 | 26.0% |
| 1,000 | 27.7% |
| full pool | 33.3% |
The curve rises logarithmically and has not flattened by the largest capped size tested. One hundred examples per database, a plausible hand-curated set, already delivers 9 of the full 21-point gain, at the same per-query cost as any other size. Every capped size, including 1,000, remains significantly worse than the full pool (p = 0.0095). There is no small sufficient core; the effect keeps growing with corpus size.
A separate pair of arms removed search entirely and instead embedded 15 exemplars per database directly in the system prompt: one set chosen by a coverage-maximization objective, one drawn at random. These static arms transfer only weakly, reaching 17.7% for the selected set and 15.7% for the random set: roughly 5 of the 21 available points survive without search, and the rest do not. A searchable bank of only 100 examples outperforms 15 examples supplied directly in the prompt (4 fixed, 15 broken, p = 0.019), which locates the effect in per-question selection rather than in the mere presence of precedent. On dw (zero foreign keys, a hundred near-synonymous tables), both static arms scored 5.0%, identical to the no-examples baseline, while EK augmentation raised dw to 31.0%. No fixed set of fifteen examples can cover the specific table a given question happens to require; only a query, issued per question, can locate it.
Model capability as a multiplier
The same 300 questions were evaluated across seven models, each once as a zero-shot LLM and once inside the same system with EK augmentation enabled, varying only the model.
In isolation, model capability barely moves the result. With Enterprise Knowledge, the spread tracks generation rather than raw capability, and only recent-generation models convert precedent into accuracy.
In isolation, model capability moves the result very little: every model tested, regardless of vintage or vendor, scores between 9.7% and 16.7% zero-shot. Raw model capability is evidently not what this benchmark rewards.
With EK augmentation enabled, the spread runs from 9.3% to 47.7%, and it tracks model generation rather than model size. Recent-generation models (gemini-3.6-flash, claude-sonnet-5, gpt-5.6-luna) convert retrieved precedent into accuracy; the preceding generation (claude-sonnet-4.5, claude-haiku-4.5, gpt-5.2) benefits only marginally, and in some cases scores below its own zero-shot result.
Neither the model nor the retrieved knowledge is, on its own, the lever. The lever is a model recent enough to use the knowledge it is given.
In isolation every model clusters within seven points; handed the same bank, only recent-generation models convert it into accuracy. The two have to meet.
This also refines the corpus-scaling result above: within a single model generation, scaling model size produces no further gain (gpt-5.6-luna to gpt-5.6-terra, 33.3% to 32.0%, p = 0.63). The gains from more model capability appear across generations, precisely where the capacity to exploit EK augmentation appears, not within a generation as size increases.
Test-time compute
Enterprise Knowledge augmentation and test-time compute compound rather than substitute for one another. CRAFT and ReFoRCE were compared with and without EK, at single pass (SC@1), eight-vote self-consistency (SC@8), and best-of-eight selection (Pass@8), on the same 300 questions, run against two backbone models: GPT-5.6 Luna and Gemini 3.6 Flash.
Enterprise Knowledge lifts every method at every compute budget, and CRAFT with EK leads at every budget tested. The chart shows the GPT-5.6 Luna backbone; the table below gives both backbones.
| system | Luna SC@1 | Luna SC@8 | Luna Pass@8 | Gemini SC@1 | Gemini SC@8 | Gemini Pass@8 |
|---|---|---|---|---|---|---|
| CRAFT + EK | 30.3% | 34.7% | 44.3% | 50.8% | 53.7% | 62.0% |
| ReFoRCE + EK | 19.1% | 22.7% | 33.1% | 27.8% | 33.3% | 53.7% |
| ReFoRCE | 11.7% | 13.0% | 15.7% | 16.8% | 18.0% | 21.0% |
Three observations follow from this table, and they hold on both backbones. EK augmentation improves every method at every budget: applied to ReFoRCE alone, it raises SC@1 from 11.7% to 19.1% on Luna and from 16.8% to 27.8% on Gemini. CRAFT with EK leads at every budget tested, on both backbones, from 30.3% at a single pass to 34.7% at eight votes on Luna, and from 50.8% to 53.7% on Gemini. And Pass@8 for CRAFT with EK reaches 44.3% on Luna and 62.0% on Gemini, well above the corresponding voted SC@8 figure in both cases, meaning the correct answer is frequently already present among eight independently sampled candidates, and that improved selection among existing candidates represents genuine remaining headroom, distinct from the retrieval intervention itself. (The single-pass figures in this table come from the test-time sweep and differ slightly from the 33.3% single-pass result reported elsewhere in this post, which is drawn from the main evaluation run; both are within the sampling noise described under Limitations.)
Cost and latency
EK augmentation is not a cost imposed on the run; it is marginally cheaper than the configuration it improves on, because an agent that has a plausible idea of where to look probes the database less.
| configuration | accuracy | median completions | median wall clock |
|---|---|---|---|
| zero-shot | 12.3% | 58 | 194 s |
| EK augmentation | 33.3% | 52 | 214 s |
Per-question cost averages $0.06, roughly $18 for a full 300-question sweep, and this figure is flat across bank sizes: a larger corpus is searched, not read in full, so the number of completions the agent issues does not grow with the size of the bank.
Interventions without measurable effect
Eight further interventions were tested and produced no significant movement in accuracy.
| intervention | result | test | verdict |
|---|---|---|---|
| oracle domain knowledge | 10.7% → 10.7% | 9 fixed / 9 broken, p = 1.00 | no effect |
| oracle gold tables and join keys | 13.7% vs 12.3% | p = 0.42 | no effect |
| probe cap (20 turns) | 12.0% vs 10.7% | p = 0.45 | no effect |
| competing-interpretation subagents | 12.3% vs 12.0% | p = 1.00 | no effect |
| question formalization | 12.0%, −11 pts coverage | p = 1.00 | no effect, reverted |
| bank tables derived from SQL, not annotation | 32.3% vs 33.3% | p = 0.74 | no effect |
| larger model, same generation | 11.3% vs 12.3% | p = 0.55 | no effect |
| larger model, on top of EK | 32.0% vs 33.3% | p = 0.63 | no effect |
Only Enterprise Knowledge moved the result. The remaining eight interventions, tested individually, sit near the harness baseline regardless of what they supplied.
Most of these interventions operated mechanically yet changed nothing. The domain-knowledge tool, for instance, was invoked in 297 of 300 runs and returned genuine predicates on the 55% of questions that carry annotated domain knowledge; it worked exactly as designed and moved accuracy by zero. Question formalization, which required the agent to write out each question as explicit output, grain, population, and verbatim-constraint fields before writing any SQL, was adopted in every run and cost 11 points of answer coverage for no accuracy gain.
The set is coherent in retrospect. Each null intervention supplied the agent with a fact about its own question, altered how much it deliberated, or changed which model was doing the deliberating. None of them supplied a worked example of how a question of the given kind is answered against the given schema. That is the one thing EK augmentation supplies, and it is the single intervention, among nine tested, that moved the result.
A deployment reading
There is a deployment reading of this result. The Enterprise Knowledge bank is not a static asset that ships with the system; it is the accumulating record of questions the system has already answered. In production, every question a user asks and every query that is confirmed correct becomes a candidate entry, so the bank grows with use and the system's accuracy on a given warehouse improves the longer it operates there. What this benchmark isolates as "EK augmentation" is, in a live setting, simply memory: the system getting better at a specific enterprise's schema by remembering how its own past questions were answered.
Read that way, the null results sharpen into a design requirement. Raw model capability, business-context injection, schema oracles, and deeper deliberation all left the number unmoved; only accumulated, searchable precedent moved it. A deployable enterprise text-to-SQL system therefore needs a mechanism for retaining and retrieving its own worked history as a first-class component, not a better base model or a richer one-shot prompt bolted onto a stateless agent.
Limitations
The 300-question sample is marginally easier than the private test set, uniformly across the four re-run baselines, so a leaderboard submission should be expected to score below 33.3%; concretely, each of the four competing methods scored a few points above its published figure when re-run on our 300, and all four moved in the same direction by a similar margin. The simplest explanation for four independently built systems improving at once is not that each got better but that our particular draw of 300 is slightly easier than the private set the published figures were computed on. Crucially, that shift is small (a few points, not tens), which is itself evidence that the public pool our sample is drawn from and the private test set are close in distribution; the leaderboard split is not a different kind of question, just a marginally harder draw of the same kind. That same easier-draw advantage applies to our own 33.3%, so on the private set it should be expected to dip by a comparable few points rather than collapse. What the re-runs establish with confidence is therefore the 19.7-point gap on identical questions, which is unaffected by how easy the draw is, since every method faced the same questions, not that the absolute 33.3% transfers unchanged to the private set.
Each configuration was swept once. At n = 300 the standard error is approximately 2.7 points around 33%, so differences below roughly 5 points between arms are not statistically resolvable, and none reported here rests on a difference that small. Every conclusion in this post concerns BEAVER specifically; the interpretation thesis, that the binding constraint is understanding the question rather than generating syntax, may not transfer to benchmarks whose questions are phrased more naturally than BEAVER's are.
Benchmark data defect. In the course of this work, BEAVER's tables annotation was found to disagree with the gold SQL it purportedly describes on 20.6% of dw entries and 52.0% of neutron's: the annotation either omits a table the gold SQL actually joins, or lists one the SQL never reads. Correcting the bank to derive tables from the gold SQL directly, rather than from the annotation, changed the headline result by nothing measurable (32.3% versus 33.3%, p = 0.74). That the result is insensitive to a data-quality defect concentrated in exactly the field most related to table identity is itself further evidence that table identity was never the active ingredient; it is reported here as a defect for the benchmark's maintainers, not as a correction that changes any number in this post.
References
- Yu, T., Zhang, R., Yang, K., et al. "Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task." EMNLP, 2018.
- Li, J., Hui, B., Qu, G., et al. "Can LLM Already Serve as a Database Interface? A Big Bench for Large-Scale Database Grounded Text-to-SQLs." NeurIPS, 2023.
- Chen, P. B., et al. "BEAVER: An Enterprise Benchmark for Text-to-SQL." arXiv:2409.02038, 2024.
- Pourreza, M., Rafiei, D. "DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction." NeurIPS, 2023.
- Gao, D., Wang, H., Li, Y., et al. "Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation." (DAIL-SQL) arXiv:2308.15363, 2023.
- Deng, M., et al. "ReFoRCE: A Text-to-SQL Agent with Self-Refinement, Consensus Enforcement, and Column Exploration." arXiv:2502.00675, 2025.
- Wang, X., Wei, J., Schuurmans, D., et al. "Self-Consistency Improves Chain of Thought Reasoning in Language Models." ICLR, 2023.
- Robertson, S., Zaragoza, H. "The Probabilistic Relevance Framework: BM25 and Beyond." Foundations and Trends in Information Retrieval, 2009.
- McNemar, Q. "Note on the Sampling Error of the Difference Between Correlated Proportions or Percentages." Psychometrika, 1947.
- Wilson, E. B. "Probable Inference, the Law of Succession, and Statistical Inference." Journal of the American Statistical Association, 1927.
- sqlglot: A Python SQL Parser, Transpiler, and Optimizer. Open-source project documentation.