A recent study at the University of Minnesota revealed that underperforming law students scored higher on two final exams when allowed to use artificial intelligence, while their high-achieving peers performed worse with this technology. Researchers compared the scores of 48 students on two final exams for courses, one covering Introduction to American Law and Legal Reasoning, and the other on Insurance Law.
Students first took the exams without the aid of artificial intelligence, then took two more exams with access to "ChatGPT-4", the latest chatbot developed by OpenAI. The researchers found that ChatGPT-4, which generates human-like text based on user prompts, significantly improved students' performance on multiple-choice questions.
Students' performance improved by approximately 29 percentage points compared to their scores on the exams without ChatGPT-4. Notably, underperforming students saw an increase of 45 percentage points when they used the AI. However, ChatGPT-4 failed to help students achieve higher scores on essay questions in both exams, and no improvement was observed for high-achieving students when using the artificial intelligence; in fact, their scores decreased by about 20 percentage points when utilizing ChatGPT-4.
The study, published this month by law professors Daniel Schwartz from the University of Minnesota and Jonathan Choi from the University of Southern California, indicated that this suggests AI may have a leveling effect on the legal profession, reducing the disparity between elite and non-elite lawyers. Schwartz commented to Reuters on Tuesday that the use of AI might have caused high-achieving students to feel somewhat complacent in the exams or that the technology might have impacted their logical reasoning skills.
He added, "Technology helps individuals at lower levels more than those at higher levels, partly because there is much more to gain." The study suggests that the use of artificial intelligence in the legal profession is likely to impact legal assistants and young lawyers, as AI can handle some of the tasks assigned to them.
