In general it is not surprising. Even if this particular study is bad.
There are certain areas of law work that are about analyzing large amounts of texts, drawing conclusions and writing other texts based on that and nothing more. That is literally the bread of LLMs.
Those types of lawyers should be the first in line for unemployment, not programmers, not even close.
"That is literally the bread of LLMs." correct. However, programming has a large number of advantages RE LLM use compare to law:
You can execute the logic, and set up loops from the output. You can set up more useful RL. It's easier to generate synthetic training data. It naturally supports tool use and agent parallelism. It's easier to integrate with APIs (with what few APIs the court systems provide). Programming explicitly encodes abstractions at the function, module levels etc that are easier to KG/reason/build upon than text chunks.
'Bread *and butter'. The English expression requires the second part—but otherwise fits perfectly in your well-stated point, with which I wholeheartedly agree.
Thank you! As a non native speaker I was not sure if “and butter” is a mandatory part but didn’t want (nor had time) to llm the comment for the sake of authenticity :) TIL
Just because it is theoretically the bread and butter of LLMs does not mean LLMs are capable of doing the job. It still needs to be proven, setting prior beliefs aside. Law is a life-critical system and deserves our highest level of scrutiny.
I see the same problem with AI in both programming and law though.
AI is like a scab on a wound: it's a temporary filler, it rushes in to fill a void, but it's not going to be the final solution.
Models showed us that there was huuuge unmet demand for literacy, both in software and in law. But now we have a choice to either address the systemic causes of the unmet demand, or just try to paper over them with layers and layers of AI scab.
> But now we have a choice to either address the systemic causes of the unmet demand, or just try to paper over them with layers and layers of AI scab.
Yeah, but in my experience it won't come down to "which is the better solution" but "which is cheaper/easier"
So I look forward to lots of layers of papered over AI scabs in the future. It won't be cheaper in the long run, but it will pump someone's quarterly numbers enough that they get a promotion before the problem they introduce come back to them
LLMs answered student questions of the top of their heads, without any refresher look into the case law. And systems that were primied with the case law like NotebookLM underperformed when compared to baseline LLMs that you'd as anything about anything.
It's not about what LLMs can or are suited to do. This study shows strengths of what's already in them, innately.
These are academics. Not to disparage them or their work at all but it is very different to the transaction or litigation work that is done in BigLaw. It is a lot more focused on analysing and summarising existing texts, which are themselves more easily available for LLMs to train on (statutes, case law, legal journals, textbooks). As such it is probably the easiest legal work to LLM-ify but also the least valuable, because I assume law professors aren't getting paid nearly as much as BigLaw lawyers. So this approach won't scale. Not to say AI won't crack BigLaw but it will be a different challenge.
> analyzing large amounts of texts, drawing conclusions and writing other texts based on that and nothing more
The same could be said about programming. Or if you want to be even more reductive, looking at a screen and pressing buttons to make the correct lights light up https://xkcd.com/722/
> It is hopelessly self-centered to claim that they were made in God's image.
Much less self centered than assume we’re developed from a mud on our own, without any external intervention.
> It is hopelessly self-centered to claim that He made the world just for us, and put us above all animals.
Not a bit more than think that we’re ultimate authority to judge and decide who is above who
> It is hopelessly self-centered to literally centre the Earth, and have the Sun rotate around us.
Geometrically it is an arbitrary choice of the coordinates system. Cosmologically and historically it was only an intermediate view held by many nations regardless of their religious views. Generally, the Earth is a truly unique planey unless found otherwise.
> Much less self centered than assume we’re developed from a mud on our own, without any external intervention.
No. "From mud" is strictly less egocentric than "in God's image" for any possible interpretation.
> Not a bit more than think that we’re ultimate authority to judge and decide who is above who
No such thought necessary.
> Geometrically it is an arbitrary choice of the coordinates system.
It is absolutely not. Have a quick peek into https://en.wikipedia.org/wiki/Deferent_and_epicycle. If you put Earth in the middle, then all the other planets move along squiggly lines and do not travel on the clean ellipses prescribed by gravity. Atheists joke about 'why not intelligent falling?' in response to 'intelligent design', which I think is a bit much. But yeah, put us back in geocentrism and you do need something like intelligent falling to fill in the gaps.
Yeah it’s funny to compare yourself now when you touched some real thinkers and real wisdom and years ago when you thought charlatans like Dawkins were worth trusting :)
Because modern so called atheists is a popular religious cult of its own, they tend to cancel anyone challenging it, because of lack of better arguments.
Natural scientist by education and degrees and raised in implicitly atheistic soviet family in my pursuit of truth I started noticing more and more evidence of the world picture without God not making much sense and fundamentally incomplete, flawed in many ways. Also it is heavily pushed down modern western people’s throats with very little real scientific support for its foundational claims.
Also noticing people pursuing their own agenda and manipulating others with “atheism” religion.
A lot of thinking, logical reasoning and a sprinkle of personal unusual experiences eventually made obvious for me that there is much more to all this than meets the eye. And there must be some deity.
Nobody knows for certain what or who he is, but world with him is much more credible model of reality than without. And by definition he can’t be “proved” so I believe in him technically, though it feels like I know.
Also on a personal note (as you see from many comments here) - many so called atheists are arrogant people (in same way as religious radicals), while most true believers I met are more humble about their faith. Though it is definitely a biased perspective. But still Id rather be associated with best religion thinkers (like for example vast majority of mideveal thinkers that shaped our modern civilization) rather than with so called atheists who tend think they are the center of the universe.
> This leads me to believe that most people either get lucky and then apply a framework in retrospect to justify their luck
Yes, congratulations on finding the truth.
This is the pattern 95% of business, psychology and other pseudoscience is built upon.
The 2 main system reasons behind it: 1) any complex system cannot be really calculated farther in the future than a very short timeframe 2) natural human brain tendency to organize the observed universe into patterns.
The good news is that if you keep buying lottery tickets your chances of winning at least once also grow.
There are certain areas of law work that are about analyzing large amounts of texts, drawing conclusions and writing other texts based on that and nothing more. That is literally the bread of LLMs.
Those types of lawyers should be the first in line for unemployment, not programmers, not even close.
reply