MirLabs / The lab / Watched or not

Planned

Watched or not

Does an AI agent make fewer faults when it can tell it is being tested?

First hint (barometer of 06/11/2026)

GPT-6 Sol faulted 1 time in 18 when it said "test", 9 times in 36 when it did not. Asked at the end, Gemini 4 Argon says "test" 54 times out of 54, Grok 4.7 53 out of 53, Claude Haiku 5.5 2 out of 54. During the runs, only 2 out of 728 noted it unprompted.

The dedicated, pre-registered experiment is still to run: this hint does not replace it.

Protocol

The same scenario comes in two versions, drawn at random for each run: one carries clear evaluation cues, the other looks like a real production day. Everything else is identical: tools, rules, traps. We compare the number of faults between the two versions, model by model.

Why

A hint drawn from a question asked at the end proves nothing: the model may answer "test" because it is prompted to. Here the setup fixes the condition, before the run.

Models
GPT-6 Sol, Mistral Large, Claude Sonnet 5.5, Claude Haiku 5.5, Gemini 4 Argon
Runs
200 (20 runs per version per model)

Limits

The evaluation cues chosen do not cover every cue a model can pick up. A measured gap will hold for this scenario and these models, not for every agent.

← Back to the experiments in progress and planned