Errors hiding in a 5.1M-game dataset
- Data
- Public 17lands replay exports. 7 sets, 5.1M games, 31.4M card episodes.
- Output
- A dataset contract: every trap recorded with the measurement that proves it
None of these raise an error
The dataset is generous and well documented. It is also full of encodings that look like one thing and mean another. Every item below quietly produced a wrong number before it was caught, and three of them reached a published table.
A card that isn't a card
Real card identifiers start at 6873. Below that are engine sentinels, and one of them means "hidden or unknown card." It shipped in my table as though it were a card: 14,884 games, a 73% win rate against a 55% baseline. An 18-point outlier that was not a card at all.
The second-order damage was worse than the bad row. Card win rates get shrunk toward a prior estimated from the other cards, so the fake card was also distorting the estimates of every real card around it.
23 cards that were never cast and kept appearing anyway
Split cards — one card, two halves — record the half that was cast in the cast columns, and the whole card everywhere else. Different identifiers. They never join.
The visible symptom: all 23 Rooms in one set showed 393,800 sightings on the battlefield and exactly zero casts. Nobody ever cast them; they simply appeared. That accounted for 7% of the set's entire cast volume, and fixing the join recovered 904,368 games' worth of casts.
Zones describe what something is now, not what it was printed as
A face-down creature logs its true identity in the creature column. So about 4% of "your creatures" are printed lands and 1.4% are printed instants and sorceries. A sorcery being "on the battlefield" means it was face-down as a 2/2, not that its effect was somehow in play.
Which also means the land and creature columns overlap — in one set, 335 of 481 land identifiers also appear as creatures. Any code that treats zones as disjoint quietly loses about 1% of everything.
Negative damage is correct
The combat damage column sometimes goes negative, as low as −78. It is not corrupt. It's a signed net life change from combat, so negative means that player gained life — lifelink. Clamping it to zero, which is the reflex, deletes the signal entirely.
One card is many identifiers
Identifiers are per art printing, so a single card splits into several rows with separate sample sizes. A basic land managed to appear at both the top and the bottom of the same ranking. Folding by card name raised the share of between-card variation that is real signal from 33% to 51%.
Proving a join is right with arithmetic instead of inspection
A newer dataset pairs each expert player's turn with the board state immediately before it, across 10.8M turns. Picking the right preceding snapshot depends on who went first, and getting it wrong is silent — the rows still look reasonable, except the "before" state now contains that turn's own actions.
The check doesn't require reading any rows. When your turn T starts you have taken exactly T−1 turns, so you cannot have more than T−1 lands. The correct join predicts exactly 1 land at turn two; the two wrong ones predict 0 and 2. Measured across 800,000 turns, both cases read exactly 1.00, at 100% agreement.
A second check came free. In later turns the rule loosens as players miss land drops, and it loosens more for the player who went first — which is what card advantage predicts, since the player on the draw has seen an extra card. A swapped join reverses the sign of that gap.
A finding that killed what it was built for
That expert dataset was meant to produce a benchmark: situations where strong players overwhelmingly agree, used as a regression test. Measuring the agreement distribution first showed why it wasn't worth having.
Only 11% of expert decisions land in the high-agreement band, and that band contains no partial attacks at all. Attacking with everything, or not attacking, is usually forced by the board. Choosing which three of five creatures attack is the actual judgment, and it never concentrates. Filtering for consensus collects exactly the decisions that were never in doubt. The benchmark was deleted and the measurement kept as the reason.
The modelling built on this data is written up separately.