This document analyzes ELT-bench, DRAMA-bench, REPRO-bench, and QUIET. The rankings of the benchmarks would be:
ELT-Bnech#
tldr: Existing data-engineering benchmarks largely evaluate isolated capabilities rather than constructing a complete Extract–Load–Transform pipeline. ELT-Bench fills this gap with 100 end-to-end pipelines, 835 source tables, and 203 target data models. Agents must:
- Inspect source schemas and records.
- Configure Airbyte connectors and Terraform infrastructure.
- Extract data from heterogeneous sources.
- Load the data into Snowflake.
- Configure dbt and write SQL transformations.
- Match the required target schema and contents.
- Diagnose failures spanning multiple tools.
ELT-Bench evaluates four agent frameworks: Spider-Agent, SWE-Agent, Augment Agent, and OpenHands CodeActAgent. Its strongest end-to-end baseline is OpenHands CodeActAgent with Claude 3.5 Sonnet, which achieves 73% SRDEL and 11.3% SRDT, at an average cost of $1.41 and 72.2 agent steps per pipeline.
Evaluation metrics#
- SRDEL: Percentage of the 100 pipelines for which every required source table was extracted and loaded with the correct row count.
- SRDT: Percentage of the 203 requested data models whose rows, required columns, and cell values matched the ground truth. Extra columns are permitted.
A major limitation is that SRDEL checks each loaded table using
SELECT COUNT(*). It therefore verifies table presence and row count, but not whether the cell values, types, duplicate identities, or provenance are correct.
| Agent framework | Model | SRDEL ↑ | SRDT ↑ | Average cost/task | Average steps |
|---|---|---|---|---|---|
| Spider-Agent | Claude 3.5 Sonnet | 23% | 0% | $3.51 | 63.3 |
| Spider-Agent | GPT-4o | 15% | 0% | $2.03 | 43.7 |
| SWE-Agent | Claude 3.5 Sonnet | 37% | 1.0% | $5.22 | 60.0 |
| SWE-Agent | GPT-4o | 0% | 0% | $5.22 | 114.3 |
| Augment Agent | Claude 3.5 Sonnet | 45% | 2.5% | $1.11 | 50.9 |
| Augment Agent | GPT-4o | 1% | 0% | $0.79 | 30.4 |
| OpenHands CodeActAgent | Claude 3.5 Sonnet | 73% | 11.3% | $1.41 | 72.2 |
| OpenHands CodeActAgent | GPT-4o | 0% | 0% | $1.00 | 38.9 |
| Spider-Agent | Claude 3.7 Sonnet + extended thinking | 57% | 3.9% | $4.30 | 89.3 |
| the following is the harness for the strongest performing model. |
| Setting | Value |
|---|---|
| Model | Claude 3.5 Sonnet |
| Agent framework | OpenHands CodeActAgent |
| Interaction strategy | Generates executable code actions inside the OpenHands environment |
| Agent configuration | Based on the SWE-bench leaderboard configuration |
| Maximum agent steps | 100 |
| Context management | OpenHands conversation-condensing action |
| Average realized steps | 72.2 |
| Average cost | $1.41 per pipeline |
| Exact model snapshot | Not specified in the paper |
Isolated-transformation experiment#
this experiment introduces isolated SRTD evaluations where the agent only needs to configure the DBT and generate the transformation model
| System | Model | Isolated SRDT |
|---|---|---|
| OpenHands CodeActAgent | Claude 3.5 Sonnet | 15.8% |
| MAC-SQL | Claude 3.5 Sonnet | 15.3% |
| MAC-SQL | GPT-4o | 15.3% |
| Spider-Agent | Claude 3.7 Sonnet + extended thinking | 11.8% |
| SWE-Agent | Claude 3.5 Sonnet | 4.9% |
| Spider-Agent | Claude 3.5 Sonnet | 3.5% |
| Augment Agent | Claude 3.5 Sonnet | 2.0% |
Preloading the source tables raises OpenHands’ SRDT from 11.3% to 15.8%, an improvement of only 4.5 percentage points. This indicates that extraction failures contribute to end-to-end failure, but transformation reasoning and SQL construction remain the dominant bottleneck. The fact that specialized MAC-SQL reaches only 15.3% also suggests that the difficulty is not explained solely by weak agent orchestration.
Headroom#
| Evaluation layer | SRDEL | SRDT | Nominal transformation headroom |
|---|---|---|---|
| ELT-Bench | 73% | 11.3% | 88.7 points |
| ELT-Bench (new models evaluated) | 96–98% | 20.20–22.66% | 77.34–79.80 points |
| ELT-Bench-Verified | 96–98% | 32.51% | 67.49 points |
The later ELT-Bench-Verified study reports:
- SWE-Agent with Claude Sonnet 4.5: 96% SRDEL and 22.66% SRDT under the original grader.
- A ReAct baseline with Claude Sonnet 4.5: 98% SRDEL and 20.20% SRDT under the original grader.
- Both systems: 32.51% SRDT, or 66 of 203 models, after benchmark correction. Even after using a newer model and correcting the benchmark, 137 of 203 models remain unsolved, leaving 67.49 percentage points of nominal transformation headroom.
Extraction and loading headroom
Given the results of extraction and loading on new models, this appears saturated for the source types and configurations covered in ELT-Bench. However because SRDEL verifies only row count, this does not establish that extraction and loading are semantically solved. A stronger evaluator would also compare:
- Cell contents and checksums.
- Column names and data types.
- Null and duplicate behavior.
- Primary-key identity.
- Incremental synchronization behavior.
- Source-to-destination provenance.
ELT-Bench Issues that were surfaced#
ELT-Bench-Verified found that:
- 82.7% of the 81 audited failed transformation tasks contained at least one benchmark-attributable issue.
- 33.0% of column-level mismatches were attributed to benchmark problems rather than genuine agent failures.
- Problems included ambiguous specifications, overly rigid evaluators, and incorrect ground truth.
- 30 unreliable target columns were removed.
- Correcting the benchmark raised SRDT from 22.66% to 32.51% without changing the agent or model. When an agent runs stage 1, it must configure Airbyte to pull data from five different sources and load it into Snowflake. The evaluation script and ELT bench check only three things:
- does the expected schema
AIRBYTE_SCHEMAexists in SnowFlake? - does each expected source table exists?
- does
SELECT COUNT(*) FROM <table>return exactly the row count recorded in the benchmark'stable.json? The pipeline succeeds if every table exists and has the right number of rows. It doesn't verify column rows that the table could be wrong about schema fidelity, such as wrong column types and missing columns. Also no checks that the data actually flowed through air byte. The agent holds Snowflake byte credentials so in principle it creates the table and inserts the right number of rows by passing the extraction tooling.