Vox PopulAI: Lessons from a global law firm's exploration of generative AI
Business leaders in every industry are seeking to identify the best uses for GenAI and where it can add the most value. This is as true in law firms as it is in other industries. Ashurst has been taking things a step beyond, using case identification and value prediction to truly understand and measure how GenAI might impact the way it works.
Vox PopulAI: Lessons from a global law firm's exploration of generative AI
Business leaders in every industry are seeking to identify the best uses for GenAI and where it can add the most value. This is as true in law firms as it is in other industries. Ashurst has been taking things a step beyond, using case identification and value prediction to truly understand and measure how GenAI might impact the way it works.
This is an edited version of an in-depth report. You can read it in full here.
From November 2023 to March 2024, Ashurst's Chief Data Officer team led three global GenAI trials involving 411 partners, lawyers, and staff across 23 offices in 14 countries. These trials aimed to test hypotheses on engagement and value derived from GenAI, using only publicly available data and adopting rigorous guardrails. The findings highlighted the following.
The initial value of GenAI for legal work lies in helping lawyers create first drafts quicker and more efficiently. Trial data indicated significant time savings: 80% for UK corporate filings, 59% for industry-specific research reports, and 45% for legal briefings. GenAI also facilitated non-legal tasks, such as drafting job descriptions and social media posts, saving up to two hours per task.
Participants found GenAI useful in pulling together relevant source material and providing a first draft to refine. However, accuracy is crucial as incorrect initial outputs could lead to more time spent on corrections. Therefore, proper safeguards and expert reviews are necessary.
GenAI-generated content had a greater variance in accuracy (1 to 4 out of 5) compared to human output (3 to 4 out of 5). On average, human-generated content scored higher (3.5 out of 5 vs. 3 out of 5 for GenAI). Interestingly, experts misidentified 50% of GenAI outputs as human-generated or couldn't tell the difference, suggesting that high-quality GenAI output can be indistinguishable from human work.
The study revealed language choice, tone and structure are indicators of human work. Well-structured summaries, appropriate vocabulary, and proper grammar and spelling were identified as human traits. Understanding these distinctions can help organisations better navigate the use of GenAI.
Quality in legal context is influenced by both objective and subjective factors. Experts had varying preferences and frustrations when evaluating GenAI output. Errors were sometimes acceptable if the overall efficiency gains were significant. For instance, despite some inaccuracies, 67% of respondents still recommended GenAI tools for their perceived advantages, such as providing a starting point and acting as a "second pair of eyes."
The trials indicated quality should be assessed using both numerical and subjective measures, recognising minor errors might be acceptable depending on the context and overall value provided.
GenAI trials uncovered value beyond legal tasks, helping staff across the organisation with day-to-day activities. Participants reported increased productivity in meetings, saving about 10 minutes per 30-minute call. GenAI provided consistent assistance, especially outside regular business hours, making participants feel more supported in managing their workload.
“…the trials have highlighted the importance of continuous experimentation, feedback, and upskilling to leverage GenAI effectively. Sharing experiences and collaborating within the industry will be crucial for promoting meaningful change and maximising the benefits of GenAI.”
Notably, 88% of respondents felt more prepared for the future using GenAI technology. This broader impact suggests GenAI can enhance overall efficiency and support staff in various roles.
The legal industry must carefully navigate the "jagged technological frontier" of GenAI to avoid leveraging it inappropriately, which could result in poorer performance. Trials highlighted the need for a sustained strategy to embed GenAI capabilities across the organisation, facilitating continuous dialogue about experiences and upskilling staff in both technical and soft skills.
Understanding and engaging with GenAI is crucial, as mere identification of use cases won't drive progress. Ashurst's trials emphasised the importance of hands-on experience and continuous feedback to uncover the myriad ways GenAI can add value.
Recommendations for running GenAI trials
- Design Trial Activities and Environment: Tailor activities to explore different aspects of GenAI use and create an environment that encourages experimentation.
- Ensure Direct Access: Provide participants with direct access to GenAI tools to explore their potential.
- Use Trial Leads: Appoint leads to guide the trial process and help address any challenges.
- Run "Art of the Possible" Sessions: Organise sessions to demonstrate potential applications and inspire participants.
- Hold People Accountable and Create Competition: Encourage engagement and innovation by holding participants accountable and fostering healthy competition.
Ashurst's learnings show an experience-led approach is crucial for understanding how GenAI can benefit users and the organisation. Engaging staff as stakeholders in the process ensures the technology is implemented effectively, addressing real needs and pain points.
Conclusion
These trials have provided valuable insights into the potential and challenges of using GenAI in the legal industry. While not all questions have been answered, the trials have highlighted the importance of continuous experimentation, feedback, and upskilling to leverage GenAI effectively. Sharing experiences and collaborating within the industry will be crucial for promoting meaningful change and maximising the benefits of GenAI.