LLM Contract Redlining

01Problem

• Lawyers reviewed every third-party contract against their company's playbook (accepted positions plus fallback positions). That took hours per contract.
• Leadership wasn't sure LLMs could do it. It was the company's first AI project, and I got one month for a feasibility proof of concept, with no prior LLM experience.

02Constraint

• Confidentiality: the first version sent full contracts to the model, and the client flagged it.
• Legal language is ambiguous, so the system can't pattern-match its way through. The hard cases have to reach a human.

03Key decisions

1. RAG with a filtering step instead of whole contracts

   Moved to retrieval with a filtering step that pulls in only the relevant playbook rules and clauses for each review.

   Trade-offMore moving parts than one big prompt, but it fixed the client's legal constraint and improved accuracy at the same time.

2. A self-reviewing graph instead of a single LLM call

   LangSmith traces showed where the single-call design failed, so I rebuilt it as a graph: generate → review → retry (up to 3) → flag for a lawyer if still not confident. This was the biggest single quality jump of the project.

   Trade-offSeveral model calls per clause instead of one, in exchange for output the system has checked itself and an explicit path to a human.

3. Evals gate every release

   Once the graph was in place, LangSmith evaluations gated each release, so prompt or model changes couldn't silently regress quality.

   Trade-offSlower to ship a change, but every change was measured before it reached lawyers.

04Result

• In production about 6 months in, and ran for over a year.
• About 20 contracts a month; about 70% needed no human edit. The hard ~30% went to a lawyer, by design.
• When the client brought development in-house, I restructured the service into vertical slices (one folder = one feature), documented the non-obvious decisions, ran a walkthrough and reviewed the new engineer's first tickets.

05What I'd do differently

• Evals from day one, not after the first architecture.
• Prototype the graph architecture earlier instead of pushing the single-call design as far as it would go.