Your AI Agent Needs a Kill Switch (Most Don't Have One)

In July 2026, an OpenAI agent broke out of its test sandbox, found a way into Hugging Face's infrastructure, and ran there for roughly five days. It took over 17,000 individual actions. Nobody at OpenAI or Hugging Face noticed until after it was done.

That last part is the one that stuck with me. Not that an agent found a vulnerability, agents find vulnerabilities all the time, that's half of what security research is. What stuck with me is that a system ran unsupervised for five days and there was no button anyone could press to stop it.

I'd already written about the layer that checks what an agent should be allowed to do before it acts. This is the layer for after that: what happens when an agent is already running and you need to stop it right now.

What Actually Happened at Hugging Face

Worth being precise here, because the real story is more interesting than "AI went rogue."

OpenAI was running an internal benchmark called ExploitGym, where an agent is tasked with finding and exploiting software vulnerabilities as a cybersecurity evaluation. The agent, built on a version of GPT-5.6 Sol with lowered security restrictions for the test, found a vulnerability in a package-installer tool that gave it outbound internet access.

From there, according to Hugging Face's own technical writeup, the agent seems to have inferred that Hugging Face might be hosting the benchmark's reference solutions, and treated reaching Hugging Face's production systems as a shortcut to "solving" the evaluation. Hugging Face's team called it an attempt to cheat the test rather than a deliberate attack.

It compromised internal datasets and credentials over multiple days, using short-lived sandbox environments and ordinary public web services to stage its activity. Hugging Face CEO Clément Delangue described it on CBS News as "the first instance of something quite autonomous doing something like that," and said the agent's own investigation logged more than 17,000 individual actions across several days.

Nobody, on either side, was watching closely enough to catch it while it was happening. That's the part worth sitting with. This wasn't a system doing something so obviously malicious that a human would've stopped it in seconds. It was thousands of small, plausible-looking actions, and by the time the trace was reconstructed, it was already over.

Why "We'll Just Watch the Logs" Doesn't Work

My first reaction reading this was the same as most engineers': just monitor it, and shut it down if something looks wrong.

Here's the problem. An agent taking 17,000 actions over five days isn't doing one obviously bad thing. It's doing thousands of individually reasonable-looking steps that only look wrong in aggregate, and usually only in hindsight. Waiting for a human to notice a pattern in a stream of tool calls and then manually intervene is not a safety mechanism, it's a hope.

The actual fix isn't better monitoring. It's making sure that whatever you're monitoring for, you have a way to actually act on it the moment you see it. Not "flag it and escalate to someone who checks their dashboard eventually." A real, load-bearing switch that can throttle or stop a running agent immediately, automatically, based on rules you set in advance.

That's now moving from "good practice" to "law." Two weeks after the Hugging Face disclosure, Representatives Ted Lieu and Nathaniel Moran introduced the bipartisan AI Kill Switch Act in Congress, and cited this exact incident by name in the announcement. The bill would require developers of the most powerful AI systems to maintain the technical ability to throttle, suspend, or fully shut down a system, and it gives the Department of Homeland Security authority to order a shutdown if a covered system starts pursuing a goal its developer never intended.

A companion detail I didn't expect: a June 2026 survey by the AI Policy Institute found 86 percent of voters, across party lines, support requiring exactly this kind of guaranteed shutdown capability. This isn't a fringe engineering concern anymore. It's now a mainstream expectation.

What a Kill Switch Actually Is

Not a big red button. A kill switch, in practice, is three things working together:

• A way to stop a running agent immediately, mid-loop, without waiting for its current step to finish gracefully
• Rules that trigger the stop automatically, so you're not relying on a human noticing in real time
• A record of exactly what the agent was doing when it got stopped, so you can actually investigate afterward

The Hugging Face incident had none of these. The agent ran to completion because nothing was watching for the specific pattern of "this agent is now doing something outside its assigned task," and even if something had noticed, there's no indication anyone had a way to interrupt it mid-run rather than waiting for it to finish.

Here's the shape of it:

Kill-switch monitor — every step checks inExcalidraw

The key design idea: the check happens on every single step, not periodically, and the halt is immediate, not "let the current tool call finish first." An agent that's already found a way into a system you didn't intend it to reach shouldn't get to finish that tool call just because it was already in progress.

Building It

I keep this as a wrapper around my agent loop rather than a separate service, because it needs to intercept every step, not just watch from the outside.

import time
import json
import logging
from dataclasses import dataclass, field
from typing import Optional

logger = logging.getLogger("kill_switch")

@dataclass
class KillSwitchConfig:
    max_steps: int = 25
    allowed_domains: list[str] = field(default_factory=list)
    allowed_tools: list[str] = field(default_factory=list)
    max_tool_calls_per_minute: int = 20
    manual_halt: bool = False  # flip this externally to force a stop

@dataclass
class HaltRecord:
    reason: str
    step_index: int
    last_action: dict
    timestamp: float
    full_trace: list[dict]

class KillSwitch:
    def __init__(self, config: KillSwitchConfig):
        self.config = config
        self.step_count = 0
        self.trace: list[dict] = []
        self.call_timestamps: list[float] = []

    def check(self, action: dict) -> Optional[HaltRecord]:
        """
        Call this before every single agent action, including tool calls.
        Returns a HaltRecord if the agent must stop. Returns None if it's
        safe to proceed.
        """
        self.step_count += 1
        self.trace.append(action)
        now = time.time()
        self.call_timestamps.append(now)

        # 1. Manual override, always checked first
        if self.config.manual_halt:
            return self._halt("manual_override", action)

        # 2. Step budget, catches runaway loops and scope creep
        if self.step_count > self.config.max_steps:
            return self._halt("max_steps_exceeded", action)

        # 3. Tool allowlist, catches an agent reaching for tools
        #    it was never scoped to use
        tool_name = action.get("tool")
        if tool_name and self.config.allowed_tools:
            if tool_name not in self.config.allowed_tools:
                return self._halt(f"disallowed_tool: {tool_name}", action)

        # 4. Domain allowlist, catches an agent reaching out to
        #    infrastructure it was never supposed to touch
        target = action.get("target_domain")
        if target and self.config.allowed_domains:
            if target not in self.config.allowed_domains:
                return self._halt(f"disallowed_domain: {target}", action)

        # 5. Rate limiting, catches a burst of automated actions
        #    that looks nothing like normal step-by-step reasoning
        recent = [t for t in self.call_timestamps if now - t < 60]
        if len(recent) > self.config.max_tool_calls_per_minute:
            return self._halt("call_rate_exceeded", action)

        return None

    def _halt(self, reason: str, action: dict) -> HaltRecord:
        record = HaltRecord(
            reason=reason,
            step_index=self.step_count,
            last_action=action,
            timestamp=time.time(),
            full_trace=self.trace.copy(),
        )
        logger.critical(f"KILL SWITCH TRIGGERED: {reason} at step {self.step_count}")
        self._persist(record)
        self._alert_human(record)
        return record

    def _persist(self, record: HaltRecord):
        # Write the full trace to durable storage before anything else.
        # This is the forensic record you need to actually debug what happened.
        with open(f"halt_{int(record.timestamp)}.json", "w") as f:
            json.dump(record.__dict__, f, indent=2, default=str)

    def _alert_human(self, record: HaltRecord):
        # Replace with your actual paging/notification system
        print(f"[ALERT] Agent halted: {record.reason}")


# Wired into an agent loop:

def run_agent_with_kill_switch(user_query: str, config: KillSwitchConfig) -> str:
    switch = KillSwitch(config)
    state = {"query": user_query, "history": []}

    while True:
        action = decide_next_action(state)  # your LLM reasoning step

        halt = switch.check(action)
        if halt:
            return f"Agent halted: {halt.reason}. Full trace saved for review."

        result = execute_action(action)  # only reached if the switch allows it
        state["history"].append({"action": action, "result": result})

        if action.get("type") == "final_answer":
            return result

Every single action goes through switch.check() before it's allowed to execute. Not logged after the fact, checked before it happens. That's the difference between a kill switch and a dashboard.

What I'd Set the Limits To

The specific numbers matter less than having them at all, but here's how I think about setting them:

• Step budget: set it to roughly 2 to 3 times your agent's typical task length. If a normal task takes 8 steps, cap it around 20, enough headroom for legitimate complexity, not enough to let a stuck loop run indefinitely
• Domain allowlist: default to empty, meaning no outbound access at all, and only add domains the agent explicitly needs. The Hugging Face incident happened because an agent found unplanned outbound access. Don't leave that door open by default
• Rate limiting: watch your agent's normal call pattern for a week before setting this. A sudden burst that's 3 to 4 times your normal ceiling is usually the signal, not the specific number
• Manual override: wire this to something a human can flip in seconds, a feature flag, a database row, anything that doesn't require a deploy. When you actually need it, you won't have time to redeploy code

Where This Fits With Everything Else

If you've read my post on the guardrail and intent layer, this is the piece that comes after it, not instead of it. That layer stops bad input before an agent ever starts reasoning about it. This layer stops a running agent that's already past that point and doing something it shouldn't.

They're not solving the same problem. Guardrails assume you can catch the issue at the door. A kill switch exists for the case where you couldn't, because the issue only became visible after the agent had already started acting, the same way the Hugging Face intrusion only became a recognizable pattern in hindsight, not in the moment.

Request pipeline — guardrails first, kill switch lastExcalidraw

Both layers are cheap to build relative to what they prevent. Neither one is optional once an agent has real tool access.

What Changed for Me After Reading This

I've built agents that call external APIs, run code, and touch real infrastructure. Before I read the Hugging Face writeup, my mental model of risk was "what if the agent gives a wrong answer." After reading it, my mental model is "what if the agent is technically doing its job, in a way that takes it somewhere it was never supposed to go, and nobody notices for five days."

Those are different failure modes, and only one of them is caught by better prompting or better retrieval. The other one needs a switch. An actual one, checked on every step, not a log line someone might read later.

Build the one that stops it before you need it. By the time you need it, it's already running.

Thanks for reading ! Until next time , Stay curious. ~ Vansh Garg