How useful are Large Language Model (LLM) AI tools in legal work? Judge Newsom of the 11th Circuit returned to the question in today's opinion in United States v. Deleon. Deleon had robbed a convenience store cashier at gunpoint but never touched the cashier. Deleon reached over the counter at one point, but stayed on his side. The encounter was over in about a minute. Deleon was convicted of robbery under the Hobbs Act and received a "two-level physical restraint enhancement" of his sentence, for an additional 7 years.
Under 11th Circuit precedent, the panel was bound to hold that the robbery constituted "physical restraint." Still, all three judges wrote that it was a case that called for reconsideration of the precedent by the full 11th Circuit.
Judge Kevin Newsom wrote (for the second time in recent months) an extended concurrence describing his resort to LLMs for help in interpretation. He found that typing in the same query to the same LLM would "sometimes provide subtly different answers to the exact same question." He concluded that (1) the LLMs are designed to produce "creativity" in their responses, so that the varying responses are a feature, not a bug; and (2) "ordinary meaning" as ordinarily understood often/always has a range of meanings, so LLMs should mimic those slight variations. There's not always a "right" meaning, especially for a phrase of more than one word. He uses the example of "passed out" – (1) "the teacher passed out the writing assignment"; and (2) "the teacher passed out on the floor." "Creativity" can be adjusted in the LLMs, which call it "temperature." Dialing it down gives less variation.
He concludes that LLMs can be useful for finding ordinary meaning of a word or phrase but shouldn't be expected to be perfect and unvarying.
If you're brainstorming business ideas, summarizing large masses of information, or (dare I say) "writing" a term paper, then variety is, so to speak, the spice of life. Mindless repetition is bad; a little imagination is good. So, while I'll confess that I find it a little frustrating that I can't explain, at a granular technical level, exactly why the models changed their successive responses, however slightly, it now makes sense to me that they did so. Again, that's how they're designed.
The content of this article is intended to provide a general guide to the subject matter. Specialist advice should be sought about your specific circumstances.