The study can be found here: https://arxiv.org/html/2401.16212v1
Many legal professionals have been skeptical about the quality of A.I. Review. A recent groundbreaking study out of New Zealand this year comparing LLMs/AI against contract attorneys sheds light on how well A.I. does. The principal finding is that advanced models match or exceed the quality of review of human reviewers, in a matter of seconds, and demonstrate a stunning 99.97% cost reduction over traditional methods.
The methodology of the study pitted junior associates and Legal Process Outsourcers (LPOs) against LLMs using Senior Associate decisions as the benchmark. A variety of LLMs were used for comparison. All were to read through a series of contracts and identify the relevant portions for decision making purposes. The analysis was carried out on two fronts; being able to determine the applicable legal issues, and being able to locate the information regarding those legal issues in the contracts.
The results of the study are very insightful. Of note:
- The human reviewers tended to be more precise at the expense of recall. They would under-suggest legal issues and relevant information preferring to focus on being more accurate, as opposed to capturing all the relevant information needed.
- The LLMs tended to have higher recall, capturing more of the overall relevant information and oversuggesting a little more than the human reviewers. Most of the LLMs had comparable or superior F-Measure scores (an average of Recall and Precision) than the human reviewer groups.
- However, it was in the time comparison in which AI review really shined. Considering that the results between the human and LLMs were largely comparable, the time it took each of the groups to complete the requisite tasks are below:
That’s right, the results were found in minutes.
When this information is quantified into a dollar amount for comparison, the per document rates portray a stark contrast.
The LLMs did as good of a job at a fraction of the cost. This is the benefit of A.I. Review.
As for the overall conclusions of the report, we can’t state it any better than the authors of the study did:
To see a demonstration of what A.I. review can do for you, contact us at eDiscovery AI Today!
