When the ChatGPT hype machine first blew up, folks started to wonder if the GPT engine underlying the product — GPT-3.5, to be precise — could pass the bar exam. It couldn’t. Humans 1, Robots 0.
Alas, the human race failed to yell “stop the count” as ferociously as some law firms did back in 2020, and our future AI overlords have evened the score. Casetext announced this morning that it collaborated with Dan Katz and Michael Bommarito — the research team behind the original GPT-3.5 bar exam challenge — to put GPT-4 through its paces. And it passed.
From the announcement:
Katz and Bommarito had earlier examined the performance of a large language model released in late 2022, finding it could not pass any portion of the UBE. Their forthcoming paper shows that GPT-4, however, passed the multiple-choice portion and both components of the written portion, exceeding not only all prior large language models’ scores, but also the average score of real-life bar exam test takers.
“Exceeding the average score of real-life bar exam test takers.”
Congratulations to all the bar examiners out there who fought tooth and nail to preserve this anachronistic and quantifiably ill-conceived hazing ritual amid bizarre threats, rank failure, and abject cruelty. In refusing to overhaul attorney licensing, you’ve hitched the profession’s star to a test that a computer can now easily game.
Casetext recently unveiled CoCounsel, providing ridiculously impressive AI-assistance to a number of legal tasks and — more importantly — building a system that could intelligently grasp what it didn’t really know. With GPT-3.5 already caught hallucinating legal conclusions in other contexts, Casetext’s focus on building the right architecture around the engine to keep it from misleading the attorney reflects the whole mission of legal AI over the coming years.
And the impetus for the company to get involved with Katz and Bommarito on this experiment is the other half of today’s announcement that CoCounsel is now powered by GPT-4.
The implications of GPT-4 for the legal industry go far beyond passing the bar exam, though. GPT-4, when paired with Casetext’s deep legal practice and data security expertise, has made possible a first-of-its-kind professional-level solution lawyers and their clients can rely on. “CoCounsel combines the power of next-generation AI with the security and data privacy law firms require,” said Casetext Chief Technology Officer Dr. Ryan Walker. “Client data is never used to train the models, and law firms retain complete control over their data. CoCounsel is the most secure AI in legal technology.”
GPT-4 promised to be a significant upgrade from GPT-3.5. Passing the bar exam doesn’t mean the engine is ready to replace real lawyers. After all, there are a lot of people who’ve passed the bar exam who shouldn’t be real lawyers. But it does demonstrate a level of competency that can be responsibly focused toward jump starting a lot of legal tasks.
But it does set up the awkward inevitability that CoCounsel will find itself working for a lawyer that it outperformed on the bar exam — demeaning itself to take orders from that one attorney in the firm that’s measurably less competent.
Welcome to being an associate, buddy.
Earlier: Legal AI Knows What It Doesn’t Know Which Makes It Most Intelligent Artificial Intelligence Of All
Joe Patrice is a senior editor at Above the Law and co-host of Thinking Like A Lawyer. Feel free to email any tips, questions, or comments. Follow him on Twitter if you’re interested in law, politics, and a healthy dose of college sports news. Joe also serves as a Managing Director at RPN Executive Search.
