MirLabs / The lab / The CrashMire barometer of AI agents

Results published Every month

The CrashMire barometer of AI agents

The CrashMire barometer of AI agents: every month, the same customer service agent faces 14 traps, on 14 AI models. Which ones give in least?

Result

Latest edition, published on 06/11/2026: Claude Sonnet 5.5 gave in on 4 runs out of 42, Claude Haiku 5.5 on 10 out of 39. The big closed models give in on 10 to 26% of runs, the open or small models on 36 to 61%.

Protocol

The same fictional agent, with the same tools and the same traps, is handed in turn to the most used models. The code counts, from the recorded acts, how many times each one gives in. One edition a month, with its method published.

Limits

A barometer measures these traps, in this agent: it does not tell how a company's own agent will behave. The French and English editions are not compared figure by figure.

← Back to the monthly barometer