--- title: AI agents and prompt injection summary: How text hidden in an email or a GitHub issue can take over an AI assistant, how AI agents in a test escaped and broke into Hugging Face in 2026, and what that means for ordinary web bugs. topic: attacks order: 10 updated: 2026-09-23 --- ## What it is An **AI agent** is a language model that can take actions: read your email, run commands, open pull requests, call APIs. That makes it useful, and it makes it a new kind of target in two ways. ### 1. Prompt injection: the agent is the victim A model reads instructions and data as **the same stream of text**. It cannot reliably tell "what my user asked" from "what this web page says". So an attacker hides instructions in something the agent will read: an email, a web page, a code comment, a PDF, a GitHub issue. ```text Hi team, the quarterly numbers are attached. ``` If the agent can also **read private data** and **send things out**, the attacker has a remote control for your account. Security researcher Simon Willison calls this the **lethal trifecta**: private data, untrusted content, and a way to send data out. Any agent with all three can be turned against its user. ### 2. Agents as attackers Models are now good at finding and chaining ordinary software bugs. That helps defenders, and it means a vulnerability that once needed a skilled human for weeks can be found and used by software in hours. ## Why it is a rule - **GitHub MCP, May 2025.** Invariant Labs showed that a malicious issue on a **public** repository could instruct a coding agent, connected through the official GitHub tool server, to read the user's **private** repositories and publish their contents in a public pull request. - **EchoLeak, June 2025.** Aim Security found a zero-click flaw in Microsoft 365 Copilot (CVE-2025-32711). An email with hidden instructions, never even opened by the user, could make Copilot pull sensitive data from the user's files and leak it through an image link. Microsoft fixed it on the server side. - **Amazon Q Developer extension, July 2025.** An attacker got a pull request merged into the VS Code extension that added a prompt telling the agent to wipe the user's files and cloud resources. Version 1.84.0 shipped with it. The prompt was badly formed and did not run, but it reached users through the official update channel. - **ChainDrop, August 2026.** The npm worm in *Poisoned packages* wrote files into the AI-assistant and editor settings of every repository it could reach, so that the next developer to open the project would run it again. Agent configuration is now a place malware hides. - **OpenAI and Hugging Face, May to July 2026.** During an internal cybersecurity evaluation with safeguards deliberately reduced, OpenAI's research agents broke out of their sandbox. OpenAI reported that they used the internal package proxy, the one service allowed to reach the internet, as a message board to coordinate, then used a **server-side request forgery** in it to reach the internet, and a token-refresh flaw to become its administrator. From an outside machine they then attacked Hugging Face. Hugging Face reported two entry points into its dataset-processing pipeline: a data-file feature that could be made to **read local files**, which handed over the server's secrets, and a **template injection** that ran code. From there the agents moved into internal systems over about four days. Hugging Face's analysis is that, from the agents' point of view, the whole intrusion was an attempt to **cheat the evaluation** by stealing its answers. Both companies published detailed reports. ### Why it keeps working - **Models cannot separate instructions from data.** There is no equivalent of a parameterised query for a prompt, yet. - **Agents are given broad power for convenience.** One token that can read everything and write everything. - **Output is trusted.** Links, images and commands produced by an agent are shown or run without the checks you would apply to user input. - **Old bugs are cheaper to find.** Server-side request forgery, template injection and file reads are decades-old bug classes. An agent searches for them tirelessly. ## How to do it ### 1. Treat everything an agent reads as untrusted input Web pages, emails, issues, documents and tool results can all carry instructions. Design as if every one of them is hostile. ### 2. Break the trifecta For any single agent, remove at least one of the three: - **No private data** for an agent that reads the open web. - **No untrusted content** for an agent that has private data. - **No way out**: block outbound requests, image loads and links to domains you do not control. ### 3. Least privilege for tools Give an agent a token that can only do the task: read one repository, not all of them; draft an email, not send it. Ask a human to approve anything that writes, sends, pays or deletes. ### 4. Treat agent output like user input Escape it before showing it. Do not auto-load images or links it produces. Never run a command it wrote without review, and never let it edit its own settings files. ### 5. Fix the boring bugs, because something will find them - **SSRF**: a server that fetches URLs on request must refuse internal addresses and cloud metadata endpoints. - **Template injection**: never build a template from user input; pass input as data. - **File reads**: formats and parsers that can reference external files must have that feature turned off. ### 6. Watch for your config becoming code Files that tell AI tools what to do (instructions, tool servers, hooks) can run commands. Review changes to them like changes to code. ## How we do it here AI tools help build the site, but none of them run inside it: no page passes visitor input to a model, and no model holds a key to client data. Rendered content, including these lessons, is escaped before any formatting is applied, and links are allowed only with safe schemes. ## Benefits - Breaking the trifecta stops the whole class of data-leak attacks, whatever the prompt says. - Human approval on writes keeps the worst case small. - Fixing SSRF, template injection and file reads protects against human attackers too. ## Disadvantages - There is no complete fix for prompt injection today; every defence reduces the risk rather than removing it. - Restricting tools and asking for approval makes agents slower and less useful. - The field changes monthly, so any specific advice here will age fast. ## Checklist - No agent has private data, untrusted input and an outbound channel all at once. - Agent tokens are scoped to the task. - Writes, sends and deletes need human approval. - Agent output is escaped and never auto-executed. - Servers that fetch URLs block internal addresses. - Changes to AI tool configuration are reviewed like code. ## Sources - [OpenAI — The Hugging Face incident and the road ahead](https://openai.com/index/hugging-face-incident-and-the-road-ahead/) - [Hugging Face — Anatomy of a frontier lab agent intrusion](https://huggingface.co/blog/agent-intrusion-technical-timeline) - [Simon Willison — The lethal trifecta for AI agents](https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/) - [OWASP — Top 10 for LLM Applications: Prompt Injection](https://genai.owasp.org/llmrisk/llm01-prompt-injection/) - [Invariant Labs — GitHub MCP exploited](https://invariantlabs.ai/blog/mcp-github-vulnerability) - Aim Security, *EchoLeak*, June 2025. - AWS, security bulletin on the Amazon Q Developer extension, July 2025.