At ILTACON in Nashville this August, our founder and CEO, Jim Sullivan moderated a session on GenAI and human document review strategy with a panel of practitioners from across the industry. One thing became clear during the hour: the debate over whether GenAI can classify documents as well as a contract review team has largely ended. The questions that matter now are how to run GenAI review, how to measure it, and how to defend it. The panel answered all three, though not in that order. Here’s what stood out to us, and where we land.
Start with how you measure it: accuracy, and specifically, what to do when a reviewer and GenAI disagree. Panelists described testing GenAI against their own coding, then reading the model’s rationale on every conflict to see who was actually right. One panelist said that on those conflicts, he ended up deciding GenAI had been right about half the time. The point isn’t that GenAI and human reviewers split accuracy down the middle. It’s that a discrepancy no longer means the human wins by default, which is the assumption most review protocols were built on. Sullivan saw the same pattern testing on his own email, and he’d bet on GenAI over a contract review team on any sample set. There is no chance a review team classifies better. If a team’s only job is to mark documents as relevant or privileged, GenAI should do that work. People belong on the documents where judgment shapes the case, and on trial prep.
The rationale matters for a second reason. TAR hands you a score and, when someone asks why, you shrug. GenAI explains each call, so a reviewer can read the reasoning, fix the instruction, and run again. Validation shifts from checking a number to reading a reason. Sullivan also wondered how many classification errors TAR control sets hid over the years, since nothing explained the calls.
Defensibility comes from process, and the panel closed on that point. Their advice: document what you did and why, define who writes the instructions, who samples the output, and who signs off, and keep the case team, review team, and AI consultant in one conversation. We’d add that GenAI leaves a better record than earlier methods. Every call carries a rationale, every instruction has a version, and every sample has a result. Teams that keep those records will explain their process more easily than they ever did with TAR. Changing instructions mid-review doesn’t weaken the record either. Human teams rewrite their protocols on day ten of a review, and nobody objects. When people worry that GenAI runs too fast to adapt, Sullivan says you control the speed. Review 5,000 documents, look at the results, adjust, and go again. Run one document a day if that builds confidence.
Teams also need new yardsticks. Sullivan pointed out that a TAR project with 75% recall and 50-60% precision used to feel like a win, whereas the same numbers on a GenAI project would signal a problem. A panelist cautioned against agreeing to a fixed recall target before you see the data. Together, those points lead to one practice: look at your results first, set a target that’s reasonable and proportional for the matter, and don’t let an old benchmark set the bar.
Where a team starts matters too; Sullivan described investigation as the on-ramp. Chatting with your data carries little risk, because you aren’t defending anything to a judge. That comfort opens the door to first-pass responsiveness and privilege review, where the money sits, since review still takes around 70% of eDiscovery spend. He also described feeding pleadings or a case summary into the tool to get facts and issues before anyone opens a document—that changes strategy as well as cost. When first-level review drops from three months to three days, teams can ask whether they need to start it at all on a case likely to settle. We think teams should ask that question before they commit to review, not after.
Tool choice also carries privilege consequences. In February, a federal judge in New York held in United States v. Heppner that a criminal defendant’s conversations with a consumer AI chatbot did not qualify for privilege. The chatbot is not a lawyer, the provider’s privacy policy undercut any expectation of confidentiality, and sending the output to counsel afterward did not change that. The judge left room for a different result when counsel directs the AI work. A panelist raised the case, and the panel also mentioned a protective order that bars a receiving party from uploading produced documents to public AI tools. Our takeaway: read the terms of any tool that touches client material and keep that material out of consumer products. Sullivan’s view is that people in this industry bill enough to cover the $20 a month.
None of this means GenAI replaces every other tool. Date ranges, custodian negotiation, and search terms still matter, and hybrid TAR and GenAI workflows are common. The panel said TAR still shows up on large second requests, where volume drives the decision, and one panelist described hybrid workflows on those matters finishing in days instead of the three or four weeks TAR needed. Validation cost and what a court will accept belong in the choice too. Weigh it all matter by matter.
Sullivan’s closing advice gives every team a place to start. Take a dataset and review it for relevance. Have AI review the same set. Compare the results and learn how bad you are at eDiscovery. He called this the biggest opportunity he’s seen in his career, because someone without years of experience can build a skill that makes them far more valuable. Then read what you file. A panelist put the count of court filings with hallucinated citations at more than 1,700, and it has since passed 2,000 worldwide. Sullivan said it pains him to see citations that clearly nobody read before filing. Reading them is the job.
If you want to run that comparison on your own data, we’d be glad to help.
