Has Generative AI been overpromised and underdelivered?

By Jim Sullivan, Founder and CEO, eDiscovery AI

I was attending a session at a legal conference recently, and the moderator stated that most attorneys had high hopes for Generative AI in eDiscovery, but the reality has been disappointing as it has so far failed to deliver.

I’m sure my gasp was heard across the room.

WHATTTT?!?!?!?

About 17 years ago, I was sitting in a small stuffy room at the top of a fancy building, doing document review for hours each day. Of course, many discussions came up among the small review team, and a college football discussion came up. Do you know what I said?

“Ehhhh, Tim Tebow just seems a little bit soft.”

And his response was very clear “hearing you say that tells me that you know absolutely nothing about college football.”

And he was right! I was making a claim about a sport based on impressions I had made without seeing the man play the sport I was judging him on.

So what does this have to do with eDiscovery?

Tim Tebow is not soft. He was one of the toughest players in the game. Suggesting he is soft was a clear indication that I was not knowledgeable about college football.

Generative AI is the biggest leap forward in the history of eDiscovery. Suggesting it has been disappointing is a clear indication the speaker was not knowledgeable about using Generative AI in eDiscovery.

HAVE YOU USED RELATIVITY AIR?!? In 5 minutes, you can code your entire review set with a level of accuracy that destroys any human review.

I would ask “What were you expecting,” but the problem isn’t that Generative AI isn’t good enough. It’s that there is no chance the speaker had legitimately used AI before making that claim.

Don’t believe me? There is only one way to know for sure. TRY IT!

Give it 1 hour. One hour of your time to determine if it is as good as people claim. Let me show you how good it can be

1. 1. Identify a review project that is currently in progress. Ensure that first-level reviewers are still actively coding, so any adjustments to classifications won’t cause issues.
   2. Search for documents that the first-level review team has already classified for responsiveness.
   3. Randomly select a sample of 2,400 documents (95/2 sample).
   4. Ensure you are using a secure version of ChatGPT with data sharing turned off. Copy the reviewer protocol, and input it into ChatGPT with the following prompt: “You are a senior attorney using AI for document classification. Review the protocol below and create 10 prompts to help identify documents related to each issue in this case.”
      {PASTE YOUR REVIEWER PROTOCOL HERE}
   5. Review the AI-generated prompts and adjust them as needed.
   6. Run the refined prompts on the documents in your sample.
   7. Take a break and have a sandwich.
   8. Identify any documents where the reviewers classified a document as Relevant to any issue, but the AI did not—and vice versa.
   9. For each document identified in Step 8, review the classification to determine the correct designation. Track the outcomes by counting how often the AI’s classification aligns with the correct result versus the reviewers’ classification.

I will bet you anything the AI has more correct decisions than a review team at the end of this exercise.Humans cannot compete with the quality of AI in document classification.

Because the indisputable truth is that AI is INSANE. It can do things we didn’t believe were possible in the past. Not only can it review documents. It can draft memos. Draft briefs. Brainstorm arguments.Translate content. Review contracts. Summarize documents.

I get it. We have all gotten exhausted hearing about the next big thing in technology, but I assure you this one is real. This one is big. This is the biggest technological advance in our lifetime. Give it one hour and you will understand.