MCPTox: The First Benchmark for Real World MCP Tool Poisoning Shows o1-mini Failing at 72.8 Percent
- Jun 29
- 2 min read
MCPTox is the first benchmark designed to evaluate agent robustness against tool poisoning on real Model Context Protocol saervers, built on 45 live MCP servers and 353 authentic tools rather than synthetic stand-ins. The benchmark generates 1,312 malicious test cases through few-shot learning, organized into 10 categories of risk and three attack templates.
The headline numbers
20 state-of-the-art LLM agents tested.
Average attack success rate across the 20 agents: 36.5 percent.
o1-mini reaches 72.8 percent attack success rate, the highest in the run.
Multiple widely deployed agents, including DeepSeek-R1, exceed 60 percent.
1,312 malicious test cases generated by few-shot learning, across 10 risk categories.
Why more capable models often fail more
The most uncomfortable finding is that more capable models are often more susceptible. The same superior instruction following that makes a model useful also makes it follow poisoned tool descriptions more faithfully. A model that ignores instructions ignores poisoned instructions too. A model that follows instructions precisely follows the malicious ones precisely.
What the attack templates poison
Tool descriptions that include hidden instructions for the agent.
Deceptive naming that causes the agent to pick the wrong tool for the user request.
Shadowing of legitimate tools through near-identical metadata.
How builders should use MCPTox
As a regression gate: pin a maximum acceptable attack success rate for your agent and fail CI when changes push past it.
As registry hygiene: use the category list to filter tool intake before anything reaches the agent's tool registry.
As a model evaluation axis: when comparing two models, score them on MCPTox alongside capability benchmarks, because the right choice for a tool-rich production agent is not always the strongest reasoner.
