Artificial Intelligence Boosts Performance for Low-Performing Law Students, Raises Questions about High Achievers

A recent study conducted at the University of Minnesota has unveiled intriguing insights into the impact of artificial intelligence (AI) on law students' performance. The research reveals that while AI, specifically GPT-4, the latest iteration of OpenAI's language model, had a significant favorable influence on the final exam scores of low-performing law students, it led to a decline in performance for their high-achieving counterparts.
 

The study encompassed 48 students enrolled in two distinct courses—Introduction to American Law and Legal Reasoning and Insurance Law. The evaluation involved a comparison of their final exam scores in two scenarios: first, without the aid of AI, and then with the integration of GPT-4. This advanced AI model, known for generating human-like text based on user inputs, notably improved student performance on multiple-choice questions. The overall improvement was striking, with students achieving a remarkable 29 percentage-point enhancement over their original exam scores without AI assistance. Low-performing students experienced a particularly noteworthy 45 percentage-point surge in their exam scores when utilizing AI.
 
However, the introduction of GPT-4 yielded a different positive impact on the essay portion of the exams, and it had a contrasting effect on top-performing students. Surprisingly, their exam scores dropped by approximately 20 percentage points when employing GPT-4. The research paper, co-authored by Minnesota law professor Daniel Schwarcz and University of Southern California law professor Jonathan Choi, suggests that this observation indicates AI's potential to level the playing field within the legal profession by addressing disparities between elite and non-elite lawyers.
 
This study marks a significant addition to the growing body of research examining the role of AI in legal education. Previous investigations revealed that the earlier version of GPT-4 correlated with the performance of average law students. At the same time, a subsequent study demonstrated that GPT-4 could even achieve passing scores in bar exams. In a statement to Reuters, Schwarcz suggested that including AI might have inadvertently encouraged high-achieving students to rely more heavily on technology, potentially suppressing their independent legal reasoning skills.
 
Schwarcz's spring courses initially excluded the use of AI during final exams. A select group of students was then trained on the effective utilization of GPT-4 for legal analysis and prompts before undergoing a second final exam. The study meticulously compared the performance of participants between the two tests.
 
Notably, Schwarcz emphasized that AI's impact within the legal profession will likely influence paralegals and younger attorneys more significantly, as these emerging technologies can handle specific tasks traditionally performed by human professionals. In his view, the study's findings indicate that AI primarily benefits lower-performing individuals due to the more significant potential for improvement.
 
In closing, Schwarcz highlighted an interesting cognitive aspect: "Once someone frames an issue for you, you sort of lose the cognitive mindset of independently assessing it." This observation underscores the need for a balanced integration of AI within legal education and practice, raising essential questions about the extent of technology's role in shaping legal professionals' skill sets and approaches.