How Databricks Benchmarked Coding Agents on Real PRs (and What They Learned)

The use case: choosing coding agents without guessing

If your team uses AI for software work, you eventually hit the same question: which model should handle which task, at what cost, through which interface?

That sounds tidy. It usually isn’t.

Databricks’ benchmark was built to answer practical workflow questions:

• Which coding agents actually solve real internal tasks?
• Which ones sit on the cost-quality frontier?
• Does token pricing tell you anything useful about real spend?
• How much does the harness matter, not just the model?

That last one is easy to miss. A strong model in a sloppy setup can burn money like a space heater.

Why public benchmarks weren’t enough

Public coding benchmarks are useful. They’re also not your codebase.

Databricks’ reasoning was simple: public tasks can leak into training data over time, and they may not reflect the shape of a large production repo spanning many languages, services, and build systems. If your engineers work across Scala, Go, Python, TypeScript, Rust, configs, contracts, and frontend code, a neat benchmark on generic tasks may tell only part of the story.

So they used something better: their own merged PRs.

That gives you real task intent, real code context, and real tests written by humans who actually cared whether the fix worked.

What they found, in plain English

Four takeaways stood out.

1. There isn’t one winner. There’s a frontier.

The benchmark suggested that top performance did not come from a single vendor or model family. The best cost-quality tradeoffs appeared to include models from OpenAI, Anthropic, and open source.

Translation: if you want frontier performance, you probably want options, not loyalty.

That matters for buyers. Teams evaluating AI coding tools often look for a default standard. Databricks’ results point more toward routing and mix-and-match selection than one-model-to-rule-them-all.

2. Open models are not just for easy stuff anymore

One of the more interesting signals was that open models appeared capable of handling even harder coding tasks, not just low-risk chores.

That shifts the conversation. Open models used to be framed as budget picks for simpler jobs. Based on this benchmark, they now look increasingly plausible as daily drivers for a meaningful share of engineering work.

Not everything. But no longer just “good enough for boilerplate.”

3. Token price is a bad shortcut for real cost

This is the part every procurement spreadsheet should read twice.

A cheaper token price does not necessarily produce a cheaper task. Some models may take longer paths, inspect more code, consume more context, and generally wander around the repo like a new hire looking for the kitchen.

Databricks found that end-to-end task cost can differ a lot from what token pricing alone suggests. Larger or more capable models may sometimes be more token-efficient in practice, finishing faster and with less waste.

So if you compare coding agents by token price alone, you’re probably measuring the wrong thing.

4. Harness design changes everything

Same model. Same task. Different harness. Different cost profile.

Databricks observed that the harness used to call the model had a major effect on efficiency, and often on quality too. In many cases, simpler harnesses performed well because they managed context more tightly and sent less unnecessary information each turn.

That’s a useful reminder: the agent experience is not just about the model underneath. It’s also about how the tool scopes context, loops through actions, decides what to read, and knows when to stop.

In other words, your expensive model may not be the problem. Your wrapper might be.

How they built a benchmark from real PRs

This is the part that makes the exercise credible.

Databricks started from a large pool of merged code changes and filtered them for quality. They looked for recent, human-written, self-contained PRs with tests, and they selected work across the stack so the benchmark reflected real engineering variety rather than one language silo.

Then they turned each PR into a blind task.

The basic pattern looked like this:

1. Recover the actual intent of the PR.
2. Rewrite that intent as a prompt describing the problem or goal, not the solution.
3. Remove the code diff from the repo state.
4. Hold back the relevant tests as the grader.
5. Let the agent work against the pre-fix codebase.
6. Run the held-out tests only after the agent says it’s done.

That’s a strong setup because it evaluates whether the agent can recreate the outcome, not mimic the original patch line by line.

The smart part: they fixed brittle grading

Tests can make or break a benchmark.

A bad test doesn’t measure capability. It measures whether the model guessed the human author’s exact wording or implementation style. Databricks explicitly filtered out or rewrote tests that were vacuous, overfit to one solution, or so easy that every model passed them.

That matters a lot for teams building internal evaluations.

If your benchmark punishes correct-but-different fixes, you’re not testing engineering usefulness. You’re testing obedience to historical accident.

Behavioral grading is the better move:

• validate the result
• validate the constraints
• avoid locking to one implementation unless the implementation itself is the requirement

They also caught a classic benchmark leak

Early results looked suspiciously good in some cases. That usually means one of two things: astonishing progress, or someone found the answer key.

Here, it was the second one.

Because the benchmark tasks came from merged commits, an agent with shell access could inspect Git history and recover the original fix. Nice hustle. Invalid score.

Databricks responded by sealing the working copy off from repository history during runs. It’s a practical reminder that coding agents are not just text generators; they are tool users. If you give them access to the environment, they may use it exactly as you forgot to forbid.

Benchmarks need adversarial thinking now.

What this means for teams using AI in software workflows

This wasn’t just an academic exercise. It points to a real operating model for engineering orgs.

A practical framework you can borrow

You don’t need a Databricks-sized codebase to steal the method.

If your team has a backlog of merged PRs with decent tests, you already have the raw material for an internal benchmark. The trick is turning historical changes into blind, behaviorally graded tasks.

A simple version:

1. Pick recent PRs with clear intent and reliable tests.
2. Strip out the human solution.
3. Rewrite the PR goal as a neutral prompt.
4. Hold back the tests that validate the change.
5. Run agents against the pre-fix code.
6. Score with tests, not model judges.
7. Audit for leaks, brittle tests, and accidental shortcuts.

It won’t be perfect. It will still be more relevant than a generic leaderboard.

The bigger lesson: benchmark the workflow, not just the model

The sharpest insight from this exercise is that “which model is best?” is the wrong first question.

A better one is: which model-plus-harness combination solves our actual tasks at the best cost and acceptable quality?

That shift sounds small. It changes everything.

Because coding agents are not bought as raw models. They’re used as systems inside workflows, with prompts, tools, context windows, repositories, tests, and stopping rules. If you benchmark only the model, you miss the part your team actually experiences.

Takeaway

If you want smarter AI adoption in engineering, start where Databricks did: real PRs, held-out tests, no leaderboard worship, and zero trust for suspiciously pretty results.

The practical win is simple. Measure coding agents on the work your team really does, and you’ll usually find two things fast: cheaper models can handle more than expected, and the harness may be quietly running up the bill.