← Writing

Reading 500 obligations out of a contract

Status
Production
Result
92% gap recall, 0% false positives, on a hand-graded set with negative controls
Role
Sole engineer

What it does

A long-form commercial agreement goes in. Out comes 500+ discrete obligations in a structured form, checked against the standard template, as a coverage matrix with every gap routed to whoever owns that subject. Review used to be manual, reviewer time capped how many contracts could be checked, and omissions surfaced late.

The decision that made it usable

The instinct is to hand the whole judgment to a model: here's the contract, here's our template, tell me what's missing. It demos well and it fails on the only question that matters in legal review — why did it decide that?

So the work is split by whether a rule can be stated without hedging. Which side's number governs when both specify one. Whether a cure period falls inside a notice period. Whether a clause binds the entity under review at all. Those became roughly 150 plain functions with unit tests.

The model does one thing: decide whether this paragraph and that template concept express the same requirement, in different words, under a different heading. No code does that, and no amount of code will.

The payoff is that a reviewer who disputes an outcome gets pointed at a named function and its tests rather than a paragraph of English that gave a different answer last week. A wrong rule is a code change plus a regression test and it stays fixed. A wrong prompt is a negotiation with a probability distribution.

The rubric that couldn't see the problem

The most expensive bug produced no false alarm. It produced a clean bill of health.

A strictness check scored both sides of a comparison 5 out of 5 and marked the row green, while the actual penalty figure underneath had doubled. The check had no access to the numbers. It could not be wrong, because it could only ever be irrelevant.

The cause was two stages upstream: the extraction step had no field for numeric values, so no figure ever reached the assessor. Every hour spent improving the assessor would have changed nothing. When an automated check comes back clean, the question worth asking is whether it could see the thing that would have made it dirty.

Limits

← Writing