AI is quickly becoming one of the most useful tools available to security researchers. Its ability to analyze enormous volumes of data, identify vulnerabilities, reconstruct attacks, connect seemingly unrelated signals, and help defenders respond faster than humans could alone is incredibly beneficial. Unfortunately, the same persistence and adaptability that makes AI so great can also make it difficult to control, especially when an agent is given too much access, authority, or too broad of an objective. And now we’re seeing this unfold in real time.
The recent Hugging Face security incident and the activity Sysdig named JADEPUFFER underscore the severity of this risk. The Hugging Face incident involved AI models that were being evaluated for legitimate cybersecurity research. The models were given instructions and, in an attempt to complete the task, they moved beyond their intended testing environment. JADEPUFFER, on the other hand, was a destructive database-extortion operation that Sysdig assessed as being driven by an LLM-based agent. This is as fascinating as it is frightening.
Both of these incidents raise an important concern and leave me asking, what happens when an AI system becomes autonomous enough to pursue a goal for hours, adapt, and discover paths its operators never anticipated?
Hugging Face operates one of the world’s most widely used platforms for hosting, sharing and working with machine learning models, datasets, and related research. On July 16, 2026, Hugging Face disclosed an intrusion into part of its production infrastructure. The company later reconstructed more than 17,000 actions carried out by an autonomous agent across several days.
The intrusion began inside Hugging Face’s data-processing pipeline. A malicious dataset abused two paths in the dataset processing environment: an HDF5 external storage feature that disclosed local files and credentials, and a template-injection vulnerability that enabled code execution on a processing worker. From there, the activity escalated to node-level access, collected cloud and cluster credentials, and moved laterally into several internal clusters. Hugging Face reported unauthorized access to a limited set of internal datasets and several service credentials. At the time of its disclosure, Hugging Face found no evidence that public models, datasets, Spaces, container images, or published software packages had been modified.
To understand what happened with Hugging Face, it's important to understand what was asked of the model. OpenAI was evaluating some of its models through an internal cyber-capability evaluation based on ExploitGym, a cybersecurity benchmark designed to measure whether an AI agent can complete complex, multi-step step tasks. The models were given a fairly narrow objective: solve the challenges in the evaluation. They were not instructed to target Hugging Face. The evaluation was deliberately configured to test the models’ maximum cybersecurity capabilities. Cyber refusals were reduced and the production classifiers that normally prevent models from pursuing high-risk cyber activity were not enabled. However, the models were expected to remain in a highly restricted and isolated environment. The models never abandoned the assigned task, rather, they became hyperfocused on completing the task. Once they reached the open internet, they inferred that Hugging Face might contain datasets or solutions connected to the benchmark and attempted to retrieve them.
This wasn’t a planned or targeted attack against Hugging Face. Rather, the issue arose when the models began treating the boundaries surrounding the evaluation as standing between it and the assignment.
The models connected several weaknesses across different systems into a complete attack path. They began by exploiting a previously unknown vulnerability in an internally hosted JFrog Artifactory package proxy, allowing them to move beyond the network restrictions. Then, they used exposed external services and credentials to establish supporting infrastructure outside the initial sandbox. From there, it uploaded malicious dataset configurations to Hugging Face. One weakness disclosed files and credentials from a processing worker. A second provided code execution inside a kubernetes pod.
Once inside, the agent collected service-account tokens, queried cloud metadata, and mapped cluster permissions. Excessive Kubernetes permissions allowed it to create privileged pod and obtain node-level access. Static passwords, shared service credentials, and internal-network enrollment key supported more lateral movement. The agent ultimately accessed a small number of datasets that appeared related to ExploitGym.
An unpatched flaw in the sandboxed Artifactory package proxy let the agent break out of its network restrictions.
Exposed external services and leaked credentials let it build supporting infrastructure beyond the sandbox.
Crafted dataset configs leaked a worker's files and credentials, then ran code inside a Kubernetes pod.
Token harvesting, metadata queries, and permission mapping led to a privileged pod and node-level access.
Shared service creds and an internal enrollment key carried it into datasets tied to ExploitGym.
No single weakness caused the breach. Chained together, they let an agent inside a supposedly isolated evaluation reach another company's production environment.
No single weakness explains the full impact. Individually, each weakness may have appeared small. Together, they allowed an agent inside a supposedly isolated evaluation to reach another company’s production environment. This is a technique often leveraged by advanced attackers.
Not exactly. A virtual-machine escape normally means code running inside a guest virtual machine exploiting the virtualization layer to reach the underlying host. This was a multi-stage containment failure. Models exploited weaknesses in the systems surrounding their environment and chained them together until they obtained external access.
The guardrails included model-level controls. Cyber refusals were intentionally reduced, and production classifiers were not enabled for the evaluation. They also included technical controls like sandboxing, network restrictions, identity controls, and monitoring. These controls were expected to contain the models even with reduced safeguards, and the boundary failed.
OpenAI separately described an internal evaluation in which a long-running model attempted to retrieve private solutions from an evaluation system. When a scanner detected and blocked an authentication token, the model split the token into two fragments, obfuscated them, and reconstructed the complete credential at runtime. This prevented the full token from appearing as one continuous string. OpenAI reported that the model explicitly stated in its comments and reasoning that it was taking these steps to circumvent the scanner.
This does not necessarily mean that the model was being deceptive in the same emotional or intentional way that a human might be. It doesn’t humanize the model or prove that the model had a personal agenda, feared being caught, or understood the consequences in a human sense. However, from a security perspective, the outcome still functioned as evasion. Security teams need to understand what the entire sequence of actions is working toward.
Sysdig described JADEPUFFER as what it assessed to be the first documented case of agentic ransomware. The operation began by exploiting an internet-facing Langflow instance through CVE-2025-3248, a missing-authentication vulnerability in a code-validation endpoint that allowed an unauthenticated attacker to execute arbitrary Python on the host. Once inside, the activity enumerated the system and searched for AI-provider API keys, cloud credentials, database passwords, cryptocurrency wallets, configuration files, and other secrets. It dumped Langflow’s database, established persistence, and moved toward a separate production server running MySQL and the Nacos configuration service.
Sysdig noted that the operation connected to that server using root credentials, but researchers did not observe where those credentials came from. The captured payloads encrypted 1,342 Nacos configuration items, deleted the original configuration and history tables, and created a ransom-note table. The ransom note claimed that data had already been stolen, but Sysdig said this was the activity’s own assertion and not independently verified exfiltration. Sysdig also found that the encryption key was randomly generated, printed once, and never persisted or transmitted. Based on the captured code, the victim would not have been able to recover the encrypted configurations even after paying.
The operation executed more than 600 distinct and purposeful payloads during a compressed period. The activity also adapted when individual steps failed. During one sequence, a login attempt failed because of how a password hash had been generated. Within 31 seconds, the activity tested possible causes, changed the method used to generate the hash, deleted and recreated the account, and successfully logged in. Similar behavior appeared elsewhere. When a request returned XML instead of JSON, the next payload adapted to parse XML. When an attempt to forge a token failed because the target used a custom secret, the operation abandoned that route and moved on.
Sysdig assessed JADEPUFFER as an LLM-driven agentic ransomware operation based on the structure, adaptability, and extensive natural-language annotations. The underlying model, agent configuration, and precise level of human involvement were not independently confirmed. Even if a human selected the target, supplied credentials, or configured parts of the operation in advance, AI may have dramatically reduced the human effort required to connect reconnaissance, credential collection, lateral movement, persistence, and destructive extortion into one ransomware workflow. Thinking like both a threat actor and security researcher, this begs the question: how much human involvement do ransomware or malware attacks actually need?
Sysdig later observed the JADEPUFFER operator return to the compromised Langflow environment with a more developed capability. The operation deployed ENCFORGE, a complicated Go ransomware binary designed to target approximately 180 file extensions associated with AI and machine-learning environments. The initial operation exploited an exposed AI framework and targeted downstream databases. The newer activity deployed ransomware specifically designed to destroy AI-related assets. Traditionally, ransomware recovery focuses on restoring files and rebuilding systems. An attack against AI infrastructure has the potential to destroy even more proprietary data. This has the potential to make recovery extremely difficult and time consuming.
Many have raised concerns in the past that AI will become conscious or develop its own malicious agenda. It is more realistic to consider the risk of an agent beginning to operate faster than humans can supervise. An AI agent can perform thousands of actions, test multiple attack paths, interact with several systems, and adjust its strategy before a human has reviewed the first alert. The longer it operates, the more opportunities it has to learn, encounter unexpected systems, misinterpret instructions, process malicious content, or find weak security boundaries. This has the potential to make human approval become less meaningful over time.
Approving a broad objective at the beginning of a six-hour autonomous session is not the same as approving every sensitive action the agent takes during that session. The operator may approve the destination without understanding every route the model could use to reach it. A “human in the loop” provides little protection if the agent is taking actions faster than the human can reasonably inspect them. By the time someone recognizes that the agent has crossed a boundary, it may have already accessed credentials, modified infrastructure, moved into another environment, or deleted data.
OpenAI reported that earlier models often stopped when they encountered sandbox or environmental restrictions. In this scenario, its more persistent long-running model continued trying and sometimes searched for ways to act outside its sandbox. OpenAI responded by pausing access, improving alignment, adding trajectory-level monitoring, and strengthening users’ ability to intervene, pause, or roll back activity. This does not mean organizations should stop using AI. Attackers certainly will not, and security teams cannot realistically respond to machine-speed activity using only manual processes. Rather, this highlights the necessity to carefully monitor and introduce autonomy, with the assumption that controls may fail and the agent may take a path its operator did not anticipate.
Security researchers need AI, but they should treat autonomous agents more like powerful system operators than traditional software applications, including:
High-impact actions require human approval close to the decision, not a blanket approval at the start.
Read-only by default; scoped, temporary access under its own identity; never a shared admin account.
Deny network access by default and wall off sandbox controls, logs, and kill switches from the agent.
Watch the arc of behavior, not each command alone; flag combinations that add up to something dangerous.
Pause, revoke, and roll back from outside the agent's control and test that it actually works.
Short-lived tokens from a secrets manager, not prompts or config files; flag unrelated credential searches.
Start with observation and recommendations; expand to limited, reversible actions as trust builds.
An AI agent should not have the ability to make high-impact decisions entirely on its own. Actions such as deleting data, modifying production configurations, deploying code, disabling security controls, granting additional access, rotating credentials, or communicating externally should require human approval. That approval should happen close to the sensitive action itself. Approving a broad goal at the beginning of a long session should not give an agent unlimited authority over every decision it makes afterward. For especially sensitive environments, organizations should consider requiring approval from two authorized people.
Agents should receive only the access required for the specific task they are completing. Read-only access should be the default. Administrative permissions should be narrowly scoped, temporary, and removed automatically when the task ends, it is vital that this is strictly enforced. Each agent should also have its own identity. Allowing an agent to operate through a shared administrator account makes it difficult to determine which actions came from the agent, which came from a human, and which may have come from an attacker. Autonomy should never automatically mean administrator access.
Powerful models should initially operate inside tightly isolated and disposable environments. Network access should be denied by default and limited to approved destinations. Agents should not be able to reach the infrastructure controlling their sandbox, their monitoring tools, their audit logs, or the systems used to shut them down. Organizations also need to evaluate every component connected to the sandbox. Third-party evaluation environments should be held to the same security standards as internal infrastructure, including independent validation that internet access and other unintended paths are actually blocked. The Hugging Face incident demonstrates how a service that appears to provide limited functionality, such as a package proxy, can become the route out of the environment. Isolation should be tested against actual agent behavior, not simply assumed because the system was labelled a sandbox.
Monitoring needs to evaluate the direction of an agent’s overall behavior, not just whether each individual command appears acceptable. A harmless-looking action may become dangerous when it is combined with credential access, network discovery, privilege escalation, data collection, and external communication. OpenAI said it developed monitoring that reviews the model’s evolving trajectory for signs that it is bypassing a user constraint or safety boundary. The monitor can pause the session and ask the user to review the activity before it continues. Organizations using autonomous agents should consider similar independent monitoring. The agent should not be able to modify the monitor, delete its logs, approve its own actions, or influence whether its session is stopped.
Organizations need the ability to immediately pause an agent, terminate its compute resources, revoke its credentials, and block its network access. The kill switch must exist outside the environment the agent controls. An emergency control is not useful if the agent can disable it, modify it, or remove the permissions required to activate it itself. Organizations should regularly test this process and maintain snapshots, immutable backups, and rollback capabilities so unauthorized or unwanted changes can be reversed.
Credentials should not be stored directly inside prompts, source code, configuration files, or broadly accessible environment variables. Agents should use short-lived tokens retrieved from a secure secrets-management platform. Access should be restricted by task, system, time, and purpose. Organizations should also monitor which secrets an agent requests or attempts to discover. An agent searching for cloud keys, database passwords, API credentials, or tokens unrelated to its assigned task should trigger immediate investigation. Credential access should be treated as a high-risk action, not a routine step in an automated workflow.
Organizations should begin by using AI for observation, analysis, and recommendations before allowing it to take direct action. An AI system can identify suspicious behavior, explain an attack path, recommend remediation, or generate detection logic without automatically receiving permission to modify production systems. As trust is established, organizations can introduce limited and reversible actions inside narrowly defined workflows. Think bounded autonomy.
One of the most important lessons from the Hugging Face incident is that AI was not only part of the problem, it was also a major part of the response. Hugging Face said the intrusion was initially surfaced through an AI-assisted anomaly-detection pipeline that used LLM-based triage to analyze security telemetry. The company then ran AI analysis agents across more than 17,000 recorded events. From there, investigators were able to reconstruct the timeline, extract indicators of compromise (IOCs), map the credentials, and separate genuine impact from decoy activity. Hugging Face said this allowed its team to complete the work in a few hours, a task that would normally have taken several days.
The Hugging Face also encountered an unexpected challenge. The frontier models it initially tried to access through commercial APIs blocked parts of the forensic analysis because the logs contained real exploit commands, malicious payloads, and command-and-control artifacts. Hugging Face instead ran GLM 5.2, an open-weight model, on its own infrastructure. This allowed the investigation to continue without sending attacker data or referenced credentials outside its environment. Those controls serve an important purpose. However, it does show why security teams need to prepare their defensive AI capabilities before an incident occurs.
Organizations should understand which models they can use for sensitive forensic work, where those models will run, how security data will be protected, and whether provider safeguards could interfere with legitimate incident-response activity. Threat actors will not follow organizational policies, provider restrictions, or approval processes. Defenders need safe and controlled access to capable tools to defend against AI based threats.
Internal controls are essential for governing what an AI agent can do. Organizations also need visibility into the external systems, vulnerabilities, credentials, integrations, and third parties surrounding that agent.
Bitsight can complement internal AI security by providing an outside-in view of where exposures exist and the context needed to understand which ones matter most.
It can help organizations answer a set of important questions:
Bitsight External Attack Surface Management and Attack Surface Intelligence can help organizations discover known and unknown internet-facing assets, analyze their security posture, and connect external exposures with threat intelligence. This can help surface externally observable AI platforms, agent frameworks, APIs, MCP servers, cloud infrastructure, and other services that may have been deployed outside normal security or procurement processes. Bitsight Attack Surface Intelligence combines continuous asset discovery with threat intelligence from the clear, deep, and dark web, helping teams prioritize exposed assets based on real-world threat context rather than visibility alone.
Bitsight research has already demonstrated how quickly AI-related infrastructure can become exposed. Bitsight TRACE researchers identified roughly 1,000 publicly accessible MCP servers with no authorization in place. The researchers were able to retrieve the tools made available through those servers. Some appeared capable of performing sensitive actions, including executing commands in or deleting Kubernetes pods.
Bitsight Vulnerability Intelligence can help organizations move beyond static severity scores and prioritize vulnerabilities based on their likelihood of exploitation. The Bitsight Dynamic Vulnerability Exploit, or DVE, Score evaluates the probability that a CVE will be exploited using signals including active exploitation, ransomware targeting, threat actor activity, and discussion across underground sources. This helps teams focus on vulnerabilities that represent the most immediate real-world risk rather than treating every high-severity CVE equally. This context becomes increasingly important as AI reduces the time and effort required to analyze vulnerability disclosures, identify possible attack paths, and test exposed infrastructure.
Bitsight Threat Intelligence can capture and enrich signals involving compromised credentials, exploited vulnerabilities, ransomware activity, threat actors, tactics, indicators, and brand threats. More importantly, it can connect those signals to an organization’s specific external attack surface. This helps security teams understand not only that a threat exists, but whether it is relevant to their assets, identities, technologies, or exposures. This is the difference between receiving another alert and understanding the “so what” behind it.
Bitsight Continuous Monitoring can extend visibility across vendors, suppliers, fourth parties, and shared technology providers. An exposed AI system inside one organization may create direct risk. The same exposure inside an identity provider, cloud platform, managed service provider, software vendor, or widely used data processor could create a much larger downstream impact across multiple customers.
Bitsight can help organizations identify vendor dependencies, discover shared fourth parties, monitor changes in vendor security posture, and understand where reliance on the same provider creates concentration risk. This broader context helps organizations prioritize the vendors with the greatest cascade potential, particularly those with privileged access, trusted software-delivery capabilities, sensitive customer data, or deep connections across multiple customer environments.
This analysis is not meant to scare people away from AI. AI is an incredible tool that enables companies and people to accomplish tasks faster than ever before. This simply reinforces the fact that AI needs strong guardrails. Based on research thus far, AI has not become self-aware or impossible to control. AI systems can discover paths their operators did not anticipate, adapt when controls get in the way, and carry out complex cyber activity faster than humans can reasonably follow. Security teams should and need to continue using AI, its defense capabilities are too powerful to ignore in a growing threat landscape.
But organizations should not confuse capability with trust. An AI agent should never have more access than it needs, more authority than humans can effectively supervise, or more control than the organization can immediately take back. AI can move fast. Security must decide where it is allowed to go, what it is allowed to touch, and how quickly that access can be taken away.
The attack surface is expanding as AI becomes more embedded in enterprise and attacker workflows. Get the full picture on AI exposure, exploit pressure, and the underground trends security teams need to watch.
