They ran the most rigorous AI-vs-human study ever published. Then sold the business.

Why a controlled benchmark not a live deployment can still teach you more about AI’s real strengths than most splashy enterprise rollouts

The specific design choice (a shared rulebook of 30 legal issues) that let an AI system outperform lawyers with decades of experience

How to tell the difference between “AI is smarter than experts” and “AI is more consistent at a task experts already agreed how to grade”

A replicable method for identifying which parts of your own knowledge work are secretly rules-based, even though they feel like judgment

Why the company that ran this landmark study still ended up selling off its enterprise business five years later and what that tells you about the gap between winning a benchmark and building a business

In February 2018, legal-tech startup LawGeex published a study pitting its contract-review AI against 20 experienced US-trained lawyers, reviewing five real non-disclosure agreements for 30 predefined legal issues.

The AI scored 94% accuracy versus a lawyer average of 85%, and completed the task in 26 seconds against a lawyer average of 92 minutes.

The study’s credibility came from its design: an independent referee, law professors from Stanford, Duke, and USC, a shared rulebook of legal issues, and a statistical test (p = 0.0068) confirming the accuracy gap wasn’t chance.

The real mechanism wasn’t AI “understanding” contracts better than humans it was that the task had been pre-converted into a closed, gradable rule set that an algorithm could apply with perfect consistency, something individual lawyers vary on even when they’re experts.

LawGeex itself didn’t ride this study to lasting enterprise dominance its enterprise business was sold off to competitors in 2023, a footnote that matters as much as the headline result.