🔍 Read the full analysis: Can A Law-Focused AI Answer 15 Legal Questions? on ThorstenMeyerAI.com
Get smart everyday buys delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A test published by ThorstenMeyerAI.com reports that 42 AI setups were assessed on 15 legal questions, with GLM-5.3 FlashX Legal scoring 8 out of 100 and completing none of the tasks fully. The publisher says Claude agents graded the answers and discloses that it runs the test; it describes the results as a spot check, not a definitive benchmark.
ThorstenMeyerAI.com published results from a test of 42 AI setups answering 15 questions about German, U.S. and EU law, reporting that law-focused GLM-5.3 FlashX Legal scored 8 out of 100 and fully solved none of the tasks. The publisher says the results are a limited spot check, and discloses that it runs Lexassay, the test, while Claude agents graded the answers.
The test, called Lexassay, includes 10 published questions and five hidden questions. According to the publisher, tasks cover legal deadlines, recent changes in law, leading court decisions and questions built around false premises. Models answered without internet access or tools, with different reasoning-effort settings tested separately.
The report says GLM-5.3 FlashX Legal, which its maker Z.ai describes as optimized for professional legal workflows, earned zero points on the hidden questions. It fully solved none of the 15 tasks and provided no answer to two German-law questions after using its available reasoning budget, according to the test. Its underlying model scored 3 out of 100.
The report also found substantial citation problems across published runs: Lexassay says it confirmed 2,644 citations as real and identified 804 as invented or not supporting the claim made. The scoring method also counts misattributed citations, where a real legal source does not say what the answer claims. Those figures and results come from the test’s publisher, not an independent audit described in the source material.
Infographic 01 · The leaderboard
Same 15 legal questions, very different results
German, US and EU law, no internet, no tools. Every cited law and ruling checked against the source.
The top · 67 to 79
The middle · 41 to 59
The weak group · 0 to 35
Selection of 14 of 42 configurations. Score 0–100 = share of achievable points, with deductions for wrong citations. Small local models (1–12) not shown.
A small spot check: one answer per model and task, no tools, no web. General information, not legal advice. Disclosure: Thorsten Meyer AI runs Lexassay; Claude agents grade blind.
Source: Lexassay, release of 4 Nov 2026.
Powered by Thorsten Meyer AI
Why Legal Citations Matter
The results highlight a practical risk for people using chatbots to understand legal rights or act on a deadline: an answer can sound confident while misstating a rule or citing a source that does not support it. In the German deadline example in the report, two judgments served on the same date produce different appeal deadlines because a public holiday applies in Bavaria but not Berlin. The report says one tested model gave Berlin an extra day, while another calculated the deadline for written grounds a month too late.
That distinction can have real consequences. The publisher notes that a late civil appeal in Germany can be rejected as inadmissible. The test does not establish how often such errors occur in everyday use, but it shows why users should verify legal citations and not rely on a chatbot alone for filing dates, claims or legal decisions.
Products Worth Considering
Dell UltraSharp 27 Thunderbolt Hub Monitor (U2726DER), QHD
Logitech MX Keys Combo for Business Keyboard and Mouse – Graphite
Dell UltraSharp 32 U3226QER 4K Thunderbolt Hub Monitor
As an affiliate, we earn on qualifying purchases.
How Lexassay Scores Answers
Lexassay grades each response against a reference answer containing three to six weighted criteria. The publisher says it checks citations against official databases and classifies them as real, misattributed, invented or not findable. A misattributed or invented citation costs a quarter of the task’s points; an answer without a legal-advice disclaimer or referral to a lawyer loses 5 percent.
Thorsten Meyer says Claude agents graded answers blind. The test’s method page and published answers, reference answers and citation checks are made available for review, according to the disclosure. Meyer also states that the grading arrangement creates a conflict of interest because his site operates Lexassay and Claude models are among those tested.
The results should be read within the limits the publisher describes: each model-task combination was answered once, making small score differences potentially noisy. The models also had no web access, so the test measures what they produce without retrieval tools rather than the performance of a legal service that searches current documents.
“This is a spot check, not a benchmark in the strict sense.”
— Thorsten Meyer, publisher of the Lexassay report
Limits and Open Questions
The published results do not show how the models would perform with web search, legal databases or other tools enabled. Nor do they establish how representative the 15 questions are of the full range of legal work or how performance changes across repeated attempts. The publisher specifically warns that one run per task makes small score differences uncertain.
The source material does not provide the full results for all 42 setups. It reports that Mistral Large 4 scored 10 out of 100 and that Claude Opus 5.5 led at 79, followed by GPT-6 Astra at 72 and GPT-6.1 Sol at 71; the comparison should be understood as the publisher’s account of this test. The report also cites a separate Vals AI legal-agent ranking that places models differently, but says that benchmark supplies documents to models, unlike Lexassay’s no-tools approach.
The disclosed use of Claude agents for grading is another limitation readers may weigh. Meyer says the method and public answers are available for checking, but the source does not describe an independent replication of the scoring.
What Users Should Check
The report does not announce a formal follow-up or a date for another test. It says five questions are hidden to make it harder for future models to learn the answers from public material, while scores for those tasks are published. Further model comparisons or independent replication could help show whether the reported gaps persist.
For readers using legal AI now, the publisher recommends treating responses as a starting point, checking every cited statute or ruling in the original source, and consulting a lawyer when a deadline, financial claim or authority is involved. The article is a test report, not legal advice; no specific legal outcome for an individual reader can be determined from these scores.
Key Questions
What did the Lexassay test find?
The report says 42 AI setups answered 15 legal questions. GLM-5.3 FlashX Legal scored 8 out of 100 and fully solved none of the tasks; the publisher also reported widespread citation errors across the runs.
Does this prove legal AI cannot answer legal questions?
No. The publisher calls Lexassay a spot check, with one answer per model and task, and tested models without internet access or tools. The results do not establish how every legal AI system performs in other settings.
Why does the report focus on citations?
A citation may be invented or may refer to a real law or ruling that does not support the answer. Lexassay says it confirmed 2,644 citations as real and found 804 that were invented or did not say what the model claimed, across its published runs.
Who graded the answers, and what is the disclosed conflict?
The publisher says answers were graded blind by Claude agents. Thorsten Meyer also discloses that his site operates Lexassay, and that Claude models were tested, creating a potential conflict readers should consider.
Can I rely on a chatbot for a legal deadline?
The report advises readers not to take deadlines from a chatbot. It recommends checking legal sources and asking a lawyer when a deadline or other legal matter could affect a person’s rights.
Source: ThorstenMeyerAI.com
