VulnBump: Can AI Agents Fix Vulnerable Dependencies Without Breaking the Build?

Can AI agents fix vulnerable dependencies without breaking the build? The fix itself is rarely the mystery. Over six months in 2026, Snyk Open Source recorded 459 million vulnerability findings across ~300,000 organizations. 94% of these vulnerabilities had a known fixed version, yet only about 10% had been resolved.

That happens because the hard part is upgrading safely: most vulnerable packages are transitively imported, about a quarter of fixes require a major version bump, and even minor and patch releases, which are supposed to be backward compatible, often contain breaking changes.

VulnBump, new research we published at Snyk, puts AI agents to the test on 307 real upgrades of vulnerable dependencies. An upgrade counts as a PASS only if the vulnerabilities are gone and the project still installs, builds, and passes its tests.

What's new: to our knowledge, VulnBump is the first benchmark for AI agents built around security-driven upgrades: every task starts from real vulnerabilities that Snyk found in a real project, across six package managers. Earlier upgrade benchmarks check only for breakage, not whether the vulnerability is gone.

So, can AI agents fix vulnerable dependencies without breaking the build? Mostly, yes, but the model alone doesn't decide it. On the same model, Claude Sonnet 5.5, Snyk's Evo Remediation Agent passed 90.2% of upgrades. Anthropic's Claude Code passed 77.5% when given Snyk's recommended upgrade and scan (Claude Code + Snyk scan), and 59.6% with no Snyk data (Claude Code unaided). That's a gap of more than 30 percentage points. For reference, the package manager bump, a baseline with no AI that only applies Snyk's recommended version, already passed 46.9%.

Key results on VulnBump

Keeping the build green wasn't the hard part: all three agents did that about 93% of the time. Removing the vulnerability was. In 70 of its 124 failures, Claude Code, unaided, said it had fixed vulnerabilities that a rescan still found.

Our findings echo VulnBench, our June benchmark of AI security review, which found that Claude and Snyk Code miss different vulnerabilities and concluded that the two work best combined. VulnBump tests a different task, upgrading vulnerable open source dependencies, and points the same way: with Snyk's recommended version and scan, Claude Code's PASS rate rose by 17.9 points. Evo Remediation Agent shows how much further a purpose-built agent can go: 12.7 points beyond Claude Code with the same Snyk scan. VulnBump also follows up on the remediation results we shared in May, now with a newer model and six package managers.

How we built VulnBump

We started from real projects, matched to production. We collected 767 candidate projects from GitHub, including popular open source projects and repositories that pin known-vulnerable packages. Each had to install, build, and pass more than 85% of its tests before any change, and Snyk had to find at least one vulnerability. From the 99 that qualified, we picked 43 that match Snyk production data for ecosystem, vulnerability severity, direct vs transitive dependencies, and size of version jump.

VulnBump dataset comparison with Snyk Open Source production data

We turned each recommended upgrade into a task. Each task asks the agent to apply one upgrade that Snyk recommends: one package, moved to a version that fixes its vulnerabilities. Every task starts from a fresh copy of the project and is scored on its own. The 43 repositories yield 307 tasks covering 2,066 fixable vulnerabilities, across Maven, Gradle, npm, Yarn, pip, and Poetry, the six package managers behind about 92% of production findings.

We added a verifier that didn't trust the agent. When the agent finished, we copied its changes into a clean container and checked them from scratch. There, an attempt counted as a PASS only if it cleared all four gates:

1. The dependencies install with the command CI would run.

2. A new Snyk scan confirms the vulnerabilities are gone.

3. The project builds.

4. Every test that passed before still passes.

The VulnBump pipeline

The harnesses we tested. All three agents ran on Claude Sonnet 5.5, with a limit of 110 turns per upgrade, in the same container for each language:

• Evo Remediation Agent is Snyk’s agentic remediation engine. It uses Snyk intelligence and can run its own scan, build, and tests, and repair errors before finishing. We ran it in auto-approve mode, so it applied every fix, including upgrades Snyk rates as likely to break the build.

• Claude Code Snyk scan is Anthropic's general-purpose coding agent, given Snyk's recommendation for the upgrade: the package to upgrade, the target version to move to, and the vulnerabilities it fixes. It can also run a Snyk Open Source scan itself to check its work.

• Claude Code unaided gets the same task without the target version or any Snyk access. It has to determine which version fixes the vulnerabilities on its own, for example, by searching vulnerability databases on the web.

• The package manager bump is the non-agentic baseline: it applies Snyk's recommended version with the project's own package manager and changes nothing else.

How the repositories were chosen, the exact prompts and tools each agent had, and the leakage controls are all in the research paper: VulnBump: A Benchmark for Fixing Vulnerable Dependencies in Real Projects While Keeping the Build and Tests Green, along with the limitations: we tested one model, ran each upgrade once, and used Snyk Open Source as the judge, which can only count against Claude Code unaided, since the other two agents work from the same Snyk findings. We also didn't penalize new vulnerabilities that an upgrade pulled in.

Result 1: The same model, with three different outcomes

Evo Remediation Agent passed every gate on 90.2% of upgrades (and fixed 95.9% of the 2,066 vulnerability instances). That's 12.7 points more than Claude Code + Snyk scan, roughly one more working fix in every eight upgrades, and 30.6 more than Claude Code unaided.

And the ranking doesn't just hinge on a few large repositories: averaged per repository, PASS rates were 93.1% for Evo Remediation Agent, 80.1% for Claude Code + Snyk scan, 68.0% for Claude Code unaided, and 51.3% for the package manager bump.

Result 2: Agents need to know which version is actually safe

On the same model, giving Claude Code Snyk's scan and recommended upgrade raised its PASS rate by 17.9 points, from 59.6% to 77.5%.

The clearest difference is which version it picks. Unaided, Claude Code settled on a version Snyk still flags 64 times. With Snyk's scan, that dropped to 13. Vulnerability databases don't always agree on which version fixes a vulnerability, so a version that looks safe in one source can still be flagged in another. 

Snyk's guidance matters most for transitive vulnerabilities, in packages the project doesn't declare directly. Counting only upgrades that passed every gate, it more than doubled the share Claude Code fixed, from 28% to 59%.

Result 3: Snyk's advice alone solves about half, and then the agents separate on the rest

The package manager bump splits the 307 upgrades in two: On the 144 it passes, Snyk's recommended version is all it takes, and an agent can only keep or lose an easy fix. The other 163 need more than a version bump, such as adapting code to breaking changes or resolving dependency conflicts.

Agents on the 144 upgrades Snyk's advice solves, and on the other 163

With Snyk's guidance, agents rarely break what the bump solves, while Claude Code unaided regressed on 20 of the 144, in 15 of them by picking a version Snyk still flags. The real difference becomes evident on the hard half: with the other 163 upgrades, Evo Remediation Agent passed 82.2%, Claude Code + Snyk scan passed 61.3%, and Claude Code by itself passed 36.2%.

Result 4: A green build can hide an unfixed vulnerability

The easiest way to keep a build green is to leave the vulnerability in place, and that's mostly where we saw Claude Code failing. The rescan was the first gate to fail in 107 of Claude Code’s unaided 124 failures, 48 of Claude Code + Snyk scan's 69 failures, against 9 of Evo Remediation Agent's 30.

First failed verification gate for every failed attempt

Evo Remediation Agent's failures are visible (and that's not a bad thing). 21 of its 30 failures broke the build or tests, mostly on major upgrades that other agents often avoided. A loud failure is much safer than a silent one: when one upgrade does go wrong, the project's own build and tests show it, where a developer can actually see it, and eventually act upon it.

Result 5: Knowing the target version isn't enough

Snyk's scan gets Claude Code a long way, but getting the upgrade right takes more than that: with identical findings and targets, Evo Remediation Agent passed 39 more upgrades than Claude Code Snyk scan, 23 of them major jumps.

Evo Remediation Agent combines Snyk's intelligence with a loop in which it builds, tests, and repairs the project before it finishes. While our setup can't isolate which component makes the difference, it does show how far the system around a model can impact the result, which is consistent with DepBench, where changing only the harness changed results by up to 16.3 points.

Result 6: Major jumps and transitive dependencies remain the hardest tasks

All agents still struggle with major framework migrations: 23 upgrades, mostly Spring, Spring Boot, and Dropwizard migrations, defeated all three agents. 

PASS rates fell to 81% from 96% for Evo Remediation Agent, to 61% from 88% for Claude Code + Snyk scan, and to 42% from 70% for Claude Code unaided, respectively, on major vs non-major upgrades. Major upgrades are also where code has to change: about half of Evo Remediation Agent's major version PASSes edited the application source, compared to 5% of its non-major PASSes.

Upgrades passing every gate, by the size of the version jump

Transitive vulnerabilities, where the vulnerable package isn't declared in the manifest, make up 70% of fixable instances and are harder for every harness tested. Counting only fixes from attempts that passed every gate, Evo Remediation Agent fixed 73% of transitive instances, Claude Code + Snyk scan 59%, and Claude Code unaided 28%, against 91%, 71%, and 62% respectively of direct ones.

PASS rate by dependency type (direct vs transitive)

How does VulnBump fit with VulnBench?

VulnBench evaluates how AI coding agents find vulnerabilities in the code you write: how consistently they detect the same issues across repeated runs, and how closely their findings match a deterministic SAST baseline. VulnBump benchmarks whether they can fix vulnerabilities in the code you depend on: upgrading vulnerable open-source dependencies without breaking the build or tests.

When you read the two together, they point the same way: models bring reasoning that deterministic tools just don't have. On their own, they report different findings across runs and often stop short of a fix. Across the platform, we put the shared lesson more simply: the generator cannot be the validator.

What does this mean for teams handing security work to agents?

• Judge a fix by the evidence the agent didn't produce: An agent's report of its own fix is a claim, while a rescan, a build, and a test run become evidence. If an agent is fixing your backlog, hold it to the same checks.

• Score the vulnerability and the build together: Either one alone hides a failure. Breakage-only checks miss fixes that never happened, while vulnerability-only checks miss the fix that broke the project.

• Give the agent the security data: Snyk's scan and recommended upgrade were worth 17.9 points to a general-purpose agent, on the same model.

Evo Remediation Agent is Snyk's agentic remediation engine, available through the Snyk CLI and as a cloud service. It's one of the agents in Evo Agentic AppSec, which launches today.

Run your agent through VulnBump

The VulnBump harness and all 43 evaluation cases are open-sourced at github.com/snyk-labs/snyk-vulnbump.

The full methodology, failure analysis, and limits are in the research paper. If you're building a remediation agent yourself, run it through VulnBump and let us know what you find.

VulnBump was built by Gabija Pasiunaite, Ryan McMorrow, Brian Clark, Liran Tal, and Luke Tu.