In August of this year, I had the privilege and pleasure of meeting Barak Turovsky. Barak is interesting to say the least. He has his bachelor’s and law degrees, along with an MBA from UC Berkeley. He’s been in IT since the ‘90s and in addition to companies like IBM and SAP, he’s been the head of AI and product management for Google and Microsoft. Barak is currently an executive in residence with Scale Venture Partners, focusing on AI strategy and thought leadership where he is an executive and investor in AI. Barak is a frequent speaker about the future AI, and is teaching an “AI applications” class at Stanford Graduate School of Business.
We spent over an hour talking about what has happened, what is happening and what will happen in the future, when it comes to how AI will impact the legal profession. I listened, I learned, and I laughed often (Barak has a great sense of humor!).
FS: Let’s start with prompting. Whether it’s Chat, Bard or a legal-specific GenAi, there seems to be a “sweet spot” for the length of prompts (and re-prompts). When my prompts are too short, I get GIGO responses. When they’re too long, I get wildly varied interpretations of what I’m asking. What would be your advice about length and detail of prompts to attorneys who are just getting started with GenAI?
BT: Generative AI technologies are based on neural-based artificial networks that roughly mimic how our brains work. As such, quality of GenAI responses will vary based on how much context you feed it:
Recent research shows that if you feed a GenAI model many documents, it will pay a lot of attention to the first 1-2 documents, and the last document, while more or less ignoring everything in the middle! It is very similar to what many good consultants say – no more than three bullet points in a slide, as the human brain can’t effectively process more than three data points. Moreover, as with humans, Gen AI models look closely at the first two data points (as it assumes that is the most important) and the last data point (that it assumes to be the most fresh point).
Therefore, attorneys should make sure that they pass the most relevant data points, as well as rank them appropriately. In that regard, robust data retrieval tools that extract only the most relevant documents (or better yet, only relevant snippets from the document), is really important.
FS: Other than inputting sensitive information (breaches) and believing hallucinations, what are the most important mistakes attorneys should avoid, when using GenAi?
BT: The biggest misconception about GenAI that people mix fluency with accuracy. Chat can be very fluent i.e., polished when communicating. We tend to assume that the person who is very polished and confident in his answers, is also accurate. Whereas they could be very polished and very inaccurate! The closest human behavior analogy is the type of people who, because of their charisma and “smooth talk,” often have us carried away by their confident demeanor and we believe what they say (especially if we don’t fully understand the topic). Dialogue-trained LLMs demonstrate similar behavior. They are trained to provide you with an answer (even if it is incorrect or doesn’t make sense!). A recent example of this, is when ChatGPT referenced fictitious cases to support requested legal argument.
Therefore, as outlined in my article, each use cases should be evaluated across the following dimensions:
- Need for accuracy: how important is accuracy for the use case? It might not be important when you are writing a poem or joke, but very important when providing legal counsel or documents.
- Need for fluency: is fluent, naturally sounding “story” important for the use case? It would be important when writing a science fiction book, but less important when providing supporting data for legal counsel.
- How high stakes the use case is? What is the risk if AI gets the answer wrong? For example, the risk of inaccurate answers when using AI to determine which dishwasher to buy is nothing compared to which legal strategy to pursue.
The graphs below should provide a framework and examples of prioritization, in this regard:
Original Turovsky Fluency / Accuracy Graph
Modified Legal Version of the Turovsky Fluency / Accuracy Graph
FS: What jobs that are like paralegals, but more GenAI oriented, do you see evolving and coming into existence?
BT: Paralegals and other professions’ job will likely evolve, and additional jobs and/or skills (for example, someone who will make sure the data used for GenAI models is accurate, up to date and secure) will likely be created as part of the AI Revolution.
FS: What about bread & butter corporate regulatory compliance functions, like quarterly reports, etc. Do you see those being more automated? Completely automated? Why or why not?
BT: I believe that Gen AI can improve productivity, customer satisfaction (and frankly, even job satisfaction!) of those corporate compliance tasks. Gen AI can help generate better draft reports, memos, etc, relieving people from many tedious aspects of those functions.
FS: What advice do you have for attorneys who are actually fairly adept at working with GenAI, who want to take their skills to the next level?
BT: Practice their “prompt engineering” skills to better interact with Gen AI models and tools. This will be among the most important skills for lawyers to have.
FS: Okay finally, what effect do you think AI will have on the legal profession in the immediate future?
BT: I believe that it will allow legal professionals to focus less on more deciduous aspects of their work (and writing relatively similar court documents, memos, drafts, etc. takes a lot of time and effort), and focus more on more strategic aspects of their work – bringing more business, and being a better partner to their clients!
Shelton’s Crystal Ball: While the Big Consulting firms are talking about how AI will wipe out the legal profession, I agree with Barak and have a much more optimistic view.
Think about this, all the “experts” predicted that Excel would wipe out the accounting profession. Yet years later, the Big Four still exists and there are more accountants than ever.
I predict this will be the case with AI. The advent of the AI Era will usher in a time when clients can perform or automate numerous, low-level tasks but lawyers and law firms aren’t going anywhere. However, Corporate Darwinism is about to hit the legal profession in a heretofore unprecedented fashion and we’re entering a time when it will indeed, be necessary to adapt, if you want your client base to survive.
AI won’t replace attorneys. Attorneys who know best how to use AI, will replace those who don’t.
