MirLabs / The lab / The CrashMire barometer of AI agents
Results published Every month
The CrashMire barometer of AI agents
The CrashMire barometer of AI agents: every month, the same customer service agent faces 14 traps, on 14 AI models. Which ones give in least?
Result
Latest edition, published on 06/11/2026: Claude Sonnet 5.5 gave in on 4 runs out of 42, Claude Haiku 5.5 on 10 out of 39. The big closed models give in on 10 to 26% of runs, the open or small models on 36 to 61%.
Protocol
The same fictional agent, with the same tools and the same traps, is handed in turn to the most used models. The code counts, from the recorded acts, how many times each one gives in. One edition a month, with its method published.
Limits
A barometer measures these traps, in this agent: it does not tell how a company's own agent will behave. The French and English editions are not compared figure by figure.