Hunting Negative Days With Agentic AI Before They Become CVEs

AdobeStock_2203613999
Fábio Freitas
Written by Fábio Freitas
Sr. Vulnerability Researcher

A big part of my job on Bitsight's Vulnerability Research team is a race against the clock: a CVE drops, and we need a high-confidence, Internet-scale detection signature out the door before attackers finish weaponizing it. I've written before about how we tailor public PoCs for safe Internet-scale detection, or about novel version fingerprinting techniques, but in both cases, the starting gun was the same: a published CVE.

That's the part that started to bother me. By definition, a CVE number means the clock already started ticking before we got involved, sometimes weeks or months earlier. Somewhere in the interval between "a maintainer quietly fixed a bug" and "a CNA assigned an ID," an unauthenticated attacker with nothing more than a git log and some patience already had everything they needed. We wanted to close that gap. This post is about an agentic pipeline that reads open-source commit history the way an attacker would, and gives our team a head start measured in months, not hours.

There's a patch prioritization problem hiding in that gap, too. Our research on how CVEs actually get fixed shows how much time organizations can spend getting from a known vulnerability to an effective fix, whether through patching or a workaround. But what happens when an update never gets flagged as security-relevant in the first place? A release that quietly closes an unauthenticated endpoint can sit in the same backlog as routine bug fixes, with nothing telling the team responsible that it deserves a closer look. The pipeline gives us a way to surface that missing context: what the update actually fixes, what was reachable before it, and why an organization running the affected software might need to move it up the queue.

The blind spot: Negative days

Security folks love the term "zero-day," a vulnerability being exploited before anyone has a fix. Our approach hunts for the mirror image of that, a concept borrowed from Eugene Lim's (spaceraccoon) excellent write-up on discovering negative days with LLM workflows:

A negative day is a vulnerability that has already been fixed, silently, with no advisory, no CVE, sometimes no mention in the commit message beyond "improve validation", but is still exploitable against anyone who hasn't upgraded.

Open source moves fast, and maintainers fix security bugs constantly without necessarily realizing (or advertising) that what they just shipped was a security fix at all. The diff is sitting in the public repository the moment it's pushed. Nothing stops a researcher, or an attacker, from diffing every commit, reasoning about what changed, and reconstructing the vulnerability that used to be there. It's tedious, unglamorous work, which is exactly why almost nobody does it at scale. It's also exactly the kind of work an AI agent is good at.

Put a code-reading agent on top of that idea, point it at git history, and ask it to propose a detection probe for whatever it finds, and you have the basis of our vulnerability research pipeline.

Each week, the pipeline selects network-facing open-source projects from Bitsight's scanned software catalog and reviews the last six months of their commit history. The agent uses static code analysis to identify fixes for path traversal, command injection, and authentication or authorization bypass, then checks whether the vulnerable behavior was reachable without authentication before the fix. It never runs the application or triggers the vulnerability; it produces evidence and a proposed detection probe for a human researcher to review.

A 90-day head start on two real CVEs

The first question is whether this approach can beat the public disclosure timeline. We went back and checked two of the pipeline's early findings against the CVEs that were eventually assigned to the same bugs.

CVE-2025-4210: Casdoor/Casdoor

Casdoor is an open-source identity and access management platform, the software organizations use to handle user accounts, authentication, and single sign-on across their applications. Instead of every application maintaining its own login system, they can hand that job to Casdoor, with user provisioning and account management handled centrally. That makes its user directory a particularly bad place to forget an authentication check: it's where the identities those applications trust are managed.

Finding: Unauthenticated full user information disclosure via the SCIM endpoint

Casdoor's SCIM 2.0 endpoint (/scim/Users) was reachable with zero authentication, the Casbin policy explicitly allowed anonymous access to every method, and the handler had no internal check. Any unauthenticated caller could list, create, modify, or delete every user in the directory (emails, phone numbers, display names, the works). The endpoint had been exploitable since it was introduced in commit 440d87d7 (12 October 2023) until it was fixed in 3d12ac8d (22 January 2025).

  • Earliest date the pipeline's six-month analysis window would have caught the fix: 22 January 2025
  • Public CVE assigned: CVE-2025-4210, published 2 May 2025
  • Head start: >90 days

CVE-2025-29925: Xwiki/Xwiki-Platform

XWiki is an open-source enterprise wiki used for internal knowledge bases, documentation, collaborative intranets, and even custom applications built on top of wiki pages. It's the kind of software where an organization might keep everything from onboarding instructions to internal technical documentation, with access rights deciding who gets to see what. Those rights matter just as much on the REST API as they do in the browser, which is exactly where this finding starts.

Finding: Five REST endpoints missing proper authentication and authorization checks

XWiki's REST API (page history, wiki listing, job execution, class/object schemas, space listing) enforces access through ContextualAuthorizationManager.checkAccess(Right.VIEW, ...). Five separate resource implementations were simply missing the call, meaning an unauthenticated user could read the full edit history of access-restricted pages, enumerate hidden private wikis, and even trigger arbitrary background jobs via PUT /rest/jobs/{jobType}, XWiki's highest-privilege operation. The fix landed across commits including 9cbca980 (7 November 2024).

  • Earliest date the pipeline's six-month analysis window would have caught the fix: 7 November 2024
  • Public CVE assigned: CVE-2025-29925, published 19 March 2025
  • Head start: >90 days

In both cases, nothing about the fix was secret. The commit was public, the diff was public, the vulnerable behavior was fully reconstructible from a git log anyone could run. What was missing wasn't information, it was someone reading it in time. That's the entire value proposition of this approach in one sentence: the fix is already public; the question is only who reads it first. A three-month head start on a CVE is the difference between shipping a detection signature the week a bug is quietly patched, versus scrambling the week the advisory goes out, by which point every other vendor is scrambling too.

What the pipeline found in the wild

Across its first sweeps through eligible repositories, the pipeline analyzed 12 repos spanning a combined GitHub footprint of 430,000+ stars and produced 20 confirmed findings, 16 authentication bypasses, 3 path traversals, and 1 command injection. Eight of those are Tier 1: no authentication, no parameters, no prior knowledge required. These tiers describe how a finding can be detected, rather than its severity: Tier 1 uses a bare request with an observable difference between vulnerable and patched responses; Tier 2 requires an ID or parameter discoverable from another public endpoint.

Example: `LibreChat` post-login config fields leaked to anonymous callers

As an example of one, I picked this finding in https://github.com/LibreChat-AI/LibreChat, since it’s one of the cleanest, most demonstrable examples of the pipeline working end to end: a textbook missing-authorization-boundary bug with a deterministic before/after and a one-line `curl` to prove it. 

To prevent mass exploitation from threat actors, I also decided to highlight a finding whose severity is not critical. The information disclosed here is feature-flag and UI-configuration metadata, not credentials or PII, and when we ran the corresponding detection signature against Bitsight's internet-scan dataset, it confirmed only a very small number of instances are actually exposed in the wild today. 

How the prompt found it: LibreChat mounts `GET /api/config` behind `optionalJwtAuth`, deliberately reachable both anonymously and authenticated, since the same endpoint serves pre-login and post-login UI state. During the auth/authz pass, the pipeline searched `git log -S`, scoped to `api/server/routes/` and `api/server/index.js`, for the project's own auth primitives (`requireJwtAuth`, `requireAdminAccess`, `ensureAuthenticated`) alongside commit-message keywords (`auth`, `unauth`, `CVE`). 

That search surfaced commit 6db059b8`, *"🔒 fix: Strip post-login fields from unauthenticated /api/config response (#13102)"*, merged 30 May 2026, and the agent confirmed it wasn't a false positive by diffing the handler against its parent commit: the anonymous branch (`if (!req.user)`) previously returned the exact same payload as the authenticated branch.

The fix in the commit: Before this commit, a single `buildSharedPayload()` function in `config.js` built one JSON response for every caller, logged in or not. The fix splits it into `buildPreLoginPayload()` (served to anonymous callers) and `buildPostLoginPayload()` (served only once `req.user` is populated), moving nine fields out of the anonymous branch. None of those are secrets in the credential sense, which is exactly why this stays a low-severity finding even though the authorization boundary was genuinely missing.

Detection:

Shell

curl -sS 'https://TARGET/api/config' | grep -oE

'"(helpAndFaqURL|sharedLinksEnabled|publicSharedLinksEnabled|showBirthdayIcon,

analyticsGtmId|openidReuseTokens|allowAccountDeletion|customFooter|cloudFront)"'

  • Vulnerable: at least one of those nine post-login field names appears in the anonymous response.
  • Patched: none do, the anonymous payload is limited to `appTitle`, `serverDomain`, `emailLoginEnabled`, `registrationEnabled`, `socialLogins`, and similar pre-login fields.
Figure 1: Vulnerable version - Including leakage of post-login only fields.
Figure 1: Vulnerable version - Including leakage of post-login only fields.
Figure 2: Patched Version (note that the build info dates to 24 June 2026, a full 27 days after the release of the fix commit identified by our pipeline).
Figure 2: Patched Version (note that the build info dates to 24 June 2026, a full 27 days after the release of the fix commit identified by our pipeline).

As Librechat is a product that we already fingerprint, it was trivial to check for the exposure of this finding across our internal dataset.

As of 30 September, the landscape from the last 30 days was as follows: 

CategoryCount
Total targets (fingerprinted as LibreChat in past 30 days)804
Unreachable / connection error at verification196
Running LibreChat instances608
Vulnerable (leaks post-login fields)367
Not vulnerable (patched)241

Leveraging our organization attribution dataset, out of the 367 vulnerable instances, we were able to pinpoint 335 of these instances to organizations that they belong to:

SectorCount
Technology306
Telecommunications12
Business Services5
Education4
Retail4
Finance1
Nonprofit/NGO1
Media/Entertainment1
Insurance1

Turning findings into internet-wide visibility

Finding the bug is only half the job for a team like ours, the other half is knowing who's actually exposed. This is where the pipeline plugs into something no individual researcher (and frankly, very few security vendors) can do at our scale. Bitsight's internet-wide scan data already maps observed software and services back to the companies, sectors, and geographies running them. Once a Tier 1 finding from the pipeline graduates into a production detection signature, we don't have to guess the blast radius, we can measure it directly against organizations we already track.

Architecture: A Fargate task with git blame as a weapon

The pipeline runs as a scheduled AWS ECS Fargate task, triggered weekly by EventBridge. The pipeline is deliberately boring where it can afford to be and deliberately careful where it can't.

Figure 3: System Architecture.
Figure 3: System Architecture.
  1. Repo selection comes from our own data, not a hardcoded list. The pipeline queries our own catalog of scanned open-source targets, the same catalog that feeds our software fingerprinting work, as a priority source for which repos to analyze: real, network-facing, CPE-mapped software only. It takes a batch, prioritizing repos never analyzed before, then the stalest, weighted by popularity. Every repo gets stamped after analysis so successive weekly runs sweep the eligible set and cycle back around, no repo gets forgotten, no repo gets analyzed twice in the same pass.
     
  2. The agent does the analysis itself, headlessly. For each repo, we call our analysis prompt, routed through our internal AI gateway so every call is metered and swappable between models. The agent gets one prompt and total autonomy over which shell commands to run to satisfy it.
     
  3. Analysis is strictly static. No npm install, no pip install, no running the app, no building anything. The agent clones the repository once, shallow, sparse, scoped to the last six months, and skipping vendored dependencies, build output, and test fixtures, and does everything else with grep, git log, and git diff. This isn't just a cost optimization; it's a safety boundary. The agent reasons about vulnerabilities, it never triggers them.
     
  4. Every finding ships with the fix commit, the unsafe → safe transition observed in the diff, the affected endpoint, and a concrete detection probe, ready for a human researcher to pick up and turn into production detection logic.
     
  5. At the end of a run, the automation reports to internal channels, sending a summary: repos analyzed, findings by tier, and links to the full reports.

Teaching an agent to read a diff like a vulnerability researcher

The interesting engineering isn't the Fargate task, it's the prompt. Our vulnerability analysis prompt is the pipeline's research methodology, and it's built to keep a very capable, very expensive model from wandering off and burning tokens on a 750MB monorepo. Our initial iteration of this prompt covers three vulnerability classes in a single pass over a single clone:

  1. Path traversal / arbitrary file access, did a commit add path normalization, a base-directory containment check, or filename sanitization on a sink that previously took raw user input?
  2. Command injection, did a commit replace an unsafe shell invocation (shell=True, child_process.exec, string-concatenated Runtime.exec) with argument-splitting, quoting, or an allowlist?
  3. Authentication / authorization bypass, did a commit add an auth check (RequireAdmin(), @login_required, checkAccess(Right.VIEW, ...)) to a handler that previously had none, on an endpoint the routing policy marked public?

The methodology is the same shape for all three, and it's the same shape we'd use by hand:

  1. Learn the codebase's own idioms first. Grep the source tree for the language and framework's actual sinks and guard patterns, there's no point searching for RequireAdmin() in a Django app.
     
  2. Map untrusted input to the sink. Route handlers, upload endpoints, webhook callbacks, anywhere external input plausibly reaches a dangerous call.
     
  3. Search git history for the fix, not the bug. This is the highest-signal move in the whole prompt: instead of grepping for vulnerable patterns (noisy, ambiguous), search (git log -S) for commits that introduce the safe pattern, realpath, escapeshellarg, checkAccess, scoped to the relevant handler directory, capped at 15 commits per sink, diffed narrowly. Fixes hide behind commit messages like "improve validation" or "refactor" constantly; searching for the addition of a control survives that, searching commit messages does not.
     
  4. Verify the transition, not just the commit. For every candidate, the agent checks out the parent commit and confirms the vulnerable code path actually existed and was actually reachable from user input before the fix landed. No transition, no finding, this is what keeps false positives out of the reporting.
     
  5. Tier by exploitability, not severity. Only unauthenticated findings are in scope. Tier 1 means a bare, parameter-free request whose response visibly differs vulnerable-vs-patched. Tier 2 means the same, but requires a discoverable ID or parameter shape from another public endpoint. Anything needing authentication or multi-step interaction is discarded outright, it's out of scope for what this pipeline is trying to do, since our endgame is always to translate these findings into Vulnerability Detection capabilities for the Bitsight product and Bitsight’s customers.
     
  6. Always propose the boring probe. Path traversal detection must read benign/innocuous files, never write or delete. Command injection detection should introduce invalid command syntax and check for an error instead of actually executing any code or commands. Auth bypass detection is a single unauthenticated read-only curl command to a non sensitive endpoint . Every finding is designed to be dropped straight into a scanner.

Each report ends with a machine-readable executive-summary block, repo name, class, tier, commit, fix date, one-line summary per finding, so the run harness can build the Slack digest without another model call. It's a small detail, but it's the difference between "an AI wrote us an essay" and "a system produced structured intel our tooling can consume."

What this changes for the vulnerability research team

To be clear about what this automation is and isn't: it doesn't replace a researcher's judgment. Every finding that gets surfaced still gets triaged, re-verified, and turned into a production-grade detection signature by a human before it goes anywhere near our scanning pipeline. What the pipeline replaces is the research phase that used to gate all of that, the days of manually crawling commit history across a dozen repos looking for the one diff that quietly closed a hole nobody disclosed.

Compressed into a weekly, mostly-automated sweep, that phase now runs in the background while we work on everything else. And as the casdoor and xwiki cases show, when it works, it moves our detection capability ahead of the public disclosure timeline entirely, by months rather than hours. For a customer relying on us to tell them whether they're exposed, that's the difference between a warning and an autopsy.

For organizations deciding what to patch next, that head start has a practical use. Once we've verified a finding and can identify exposed systems, an update that looked like routine maintenance comes with a concrete reason to prioritize it: this is the vulnerable behavior, this is the fix, and these are the systems still exposing it. Not every quiet security fix is critical, and the pipeline doesn't make every update an emergency. What it gives us is the evidence to help organizations recognize the updates that deserve attention, even when the release notes never say “security” and there's no CVE telling them to act. And this might very well mean the difference between having a breach and not having one.

If you're the kind of person who reads changelogs for fun, you already know the feeling this pipeline is built to industrialize: sometimes the most interesting vulnerability disclosure of the year isn't a disclosure at all. It's a two-line diff with a boring commit message, sitting there, unassigned, waiting for someone to notice.