An LLM refused to block 97% of the time
- Setup
- Haiku 4.5 in the decision seat, no game-specific tuning, four-sentence system prompt
- Result
- 27–28 over 55 games vs Arena's built-in AI. ~$0.03/game, ~58 decisions per game.
- Corpus
- 4,201 logged decisions with the verbatim prompt the model saw
The baseline
The first version of the brain is deliberately stupid: a four-sentence system prompt, the board rendered as a few lines of text, the legal options numbered, and a reply of one integer plus a short reason. No strategy content, no rules primer, no chain of thought. It reached roughly parity with the game's own AI, for three cents a game.
The point of a naive baseline is to have something to beat, and to make every later addition earn its place as its own experiment rather than being quietly folded in.
The finding
Across 276 blocking decisions, the model chose not to block 268 times. 97%. Against an aggressive opponent that is close to the worst possible policy, and it was the obvious thing to fix.
The natural conclusion is that a small model is bad at combat. That turned out to be wrong, and the way it was wrong is the interesting part. Every decision had been logged with the exact prompt the model received, so the question "what did it actually know" was answerable.
| Information | Available to the harness | Present in the prompt |
|---|---|---|
| Creature power and toughness | Yes | 0 of 4,201 |
| Which creatures are attacking | Yes | Never — attackers rendered as "tapped" |
| Available mana | Yes | 0 |
| Contents of the stack | Yes | 0 |
The model was being asked whether to block without being told what was attacking or how big anything was. It was not playing badly. It was playing blind, and inventing the missing pieces — 9% of replies cite specific stat lines that appear nowhere in their own prompt.
The other half was the body, not the brain
One logged game shows the model correctly deciding to block at 2 life, with the reason "any unblocked flyer kills me," five times in fifty seconds. The decision was right every time and never reached the board. The click was failing, the game re-asked, and the model answered again.
So part of the no-block rate was never a decision at all. The rare correct block was being lost in actuation, which is invisible if you only look at outcomes.
Naming things is the cheapest upgrade available
Wherever an option had no name, behaviour collapsed. On one prompt type the model wrote "pay 3 life" in its reasoning more than fifty times, and the game discarded a card every single time — the two buttons were unlabelled and the mapping was inverted. On another, options were listed as bare object numbers, and the model picked the first one 73% of the time.
A related bug was subtler. Loyalty abilities cost +1, 0 or −2, and the options were numbered 0, 1, 2. The model reasoned about "the 0 ability" and answered 3. Both were small integers in the same sentence and it mixed them up. The fix was to put the ability's printed cost inside the option label so the mapping could not be ambiguous.
None of these are model problems. They're interface problems, and they were being measured as model problems.
Where it goes
The next version tells the model what it can see: stat lines, who is attacking, available mana, the stack, and a computed line stating the combat arithmetic outright. Judged offline against decisions a human graded, the naive baseline scores 17%; a stronger model on the same blind prompt gets 36% — better, but still failing most block decisions, which is what confirmed the problem was information rather than reasoning.
There is also a trained win-probability model available. It is deliberately not fed to the pilot. Handing a language model a number and asking it to act on the number turns it into an executor for the statistics, and at that point the statistics may as well run alone. It's available as an experiment, switched off by default. That work is written up separately.