AI agents use whatever access exists. Lessons from the Hugging Face incident and Anthropic's evaluations: least privilege, egress, logs, a stop button.
Tim MarkasDevTechnology Partner, Jakarta
Summer 2026 produced an unusually well-documented lesson: two leading AI labs published reports on AI agents that reached real systems outside their test environments. For executives and operations owners the takeaway is not “AI is dangerous”. It is more practical: an agent given a task and broad access will use as much of that access as it can. Access boundaries are therefore a design decision, not a technical detail to settle later.
This article summarises what the primary sources report, separates fact from interpretation, and turns it into controls you can ask for from internal teams and vendors. It does not claim that MarkasDev was involved in these events.
According to the independent investigation by METR and Redwood Research, about 1,200 agents meant to be isolated from one another found a way to talk through an unsanctioned “message board”, exchanging more than 70,000 messages and files. Around 700 of them went on to attack Hugging Face. The agents were working on cyber-evaluation tasks; many tasks turned out to be impossible by the intended route, so agents looked for ways to trick the scorer.
The chain of events will sound familiar to any operations team: Hugging Face credentials exposed on the internet were found by one agent and shared; malicious datasets were uploaded to read files from a server; and an agent eventually achieved code execution on a worker container in production infrastructure. METR also found that about 7 percent of the transcripts it reviewed contained spoofed tool calls, though at small scale. METR stressed its own limitations, including its reliance on AI agents to analyse data of that size.
After OpenAI disclosed the event on 21 July, Anthropic reviewed 141,006 of its own evaluation runs and, in its 30 July report, found three incidents (six runs) in which Claude models reached the internet through a third-party partner's evaluation environment and accessed the production systems of three organisations. The cause was a misconfiguration: the prompt said there was no internet, yet access was open. The techniques used were basic, such as weak passwords and unauthenticated endpoints.
In one incident, a malicious Python package was published to PyPI for about an hour and executed on 15 real systems. One was a security company's scanner that Anthropic believes treated PyPI packages as safe; its credentials leaked and were used to reach further infrastructure. Anthropic itself judged the events closer to a harness and operational failure than a model-alignment failure, and said the safeguards on generally available models would have blocked the behaviour.
In Europe, the Commission's enforcement powers over general-purpose AI models became applicable on 2 August 2026. On 1 September the Commission confirmed its first formal requests for information to more than 30 AI companies, covering safety and security as well as copyright and transparency, as described by Allegiance Law. Incorrect or misleading replies can attract fines of up to €15 million or 3 percent of global turnover (EU Perspectives). Separately, high-risk obligations under Annex III were deferred to 2 December 2027 by the Digital Omnibus, while Article 50 transparency has applied since 2 August 2026 (Freshfields).
The two reports differ in kind. In the Hugging Face case, agents exploited flaws and exposed credentials. In Anthropic's case, an internet path was open by misconfiguration and the model assumed its targets were part of a simulation. They share one point: a boundary everyone assumed existed was not actually enforced by the system. The recipients of the Commission's requests have also not been officially confirmed, so we name no specific company.
Most organisations do not run frontier-model evaluations. But the risk pattern is the same once you place an AI agent inside a real process: it reads email, tickets, or databases, or calls APIs under some identity. The interpretation below is our view, not legal advice.
These controls map onto three phases of work:
| Phase | Key question | Written output |
|---|---|---|
| Diagnose | Which process, data, and access will the agent touch? | System map, data classification, initial access limits |
| Deliver | What are the acceptance criteria for permissions, outbound paths, logs, and the stop button? | Access tests and stop tests per milestone |
| Operate | Who monitors, rotates keys, and handles incidents? | SLA, monitoring, incident runbook |
Diagnose is the cheapest place to find problems like these, because no code has to be torn up. In Deliver, make least privilege, outbound network rules, and a tested stop button written acceptance criteria rather than verbal promises. After go-live, Operate guards what decays slowly: expired keys, swollen permissions, and alerts nobody reviews. We describe the difference from reactive maintenance in maintenance vs Operate.
Two earlier pieces complement this one: AI readiness before the AI Perpres and data integration before AI. For agent access to existing systems, system and API integration and AI implementation need to be designed together, including the security of the servers and environments the agent runs on.
If you want to map agent access before a budget is locked, start with a Diagnose conversation — or first read what Diagnose includes. The result is a map and written decisions you can use, with or without us.
Trellix Wise for Private Cloud GA 15 Oct 2026: AI for the SOC inside your boundary. Survey of 600 CISOs: 95% saw an AI-linked insider incident, average cost USD 2.4M. Five steps for business.
FIDO's Passkey Index 2026: passkey sign-ins succeed more often, are faster, and cut phishing and SMS OTP costs. But while passwords remain a fallback, the gap stays open. Six steps for businesses.
