🔍 Read the full analysis: What Makes Z.ai’s Legal AI Rank Near The Top On Benchmark Tasks? on ThorstenMeyerAI.com
Get smart everyday buys delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Z.ai’s GLM-5.3-FlashX-Legal scored 19.17% on Vals AI’s Harvey Legal Agent Benchmark leaderboard, placing seventh among 81 models in the archived results. The score means it passed every rubric criterion on about one in five tasks, although it passed 92.5% of individual criteria. Z.ai says it improved 12.5 percentage points over its base model; the leaderboard supports that comparison, but its “state-of-the-art among open-source models” claim lacks a specified comparison set.
Z.ai’s GLM-5.3-FlashX-Legal ranked seventh of 81 models on Vals AI’s archived results for Harvey’s Legal Agent Benchmark, posting a 19.17% all-criteria pass rate. The result gives a measured comparison with other legal AI systems, but it means the model satisfied every requirement on only about one in five tasks—not that it achieved 19.17% accuracy across all legal work.
The leaderboard’s scoring rule is strict: a task passes only if every rubric criterion passes. In the archived Version 1 results, GLM Legal’s task score was 19.17%, while its individual-criterion pass rate was 92.5%. A task could therefore fail because of one missed item, such as a deadline or citation, even if the model met most other requirements. Vals AI reports that its criterion rate counted an item as passed only when both of its judges passed it.
The result appears on Vals AI’s public leaderboard, which uses Harvey’s grading protocol and a private holdout set. Vals lists the holdout as 120 tasks, with internet access disabled. Two language-model judges grade each submission, and the reported task score averages their pass rates. Z.ai’s October 13 announcement names Vals as the independent evaluator; Vals lists the model release as October 12.
Z.ai says the model improved by 12.5 percentage points over its base model. The archived leaderboard supports that arithmetic: GLM-5.3-Flash scored 6.67%, compared with 19.17% for GLM Legal. Vals listed the legal model at $0.15 per test, versus $0.57 for the base model. Z.ai separately lists API pricing of $0.37 per million input tokens and $1.25 per million output tokens; those are different measures from the leaderboard’s per-test cost.
What the Score Means for Legal Work
The result highlights both the potential and the limits of using a legal AI agent on complex, multi-part work. A 92.5% criterion pass rate indicates that the model met many individual requirements, but its lower all-pass score shows how often at least one shortcoming can leave a complete work product below the benchmark’s standard. For legal teams, a nearly complete answer may still need substantial checking before it can be relied on.
Cost is another point of comparison. Vals listed GLM Legal at $0.15 per test, below the listed costs for the higher-ranked systems in the archived table. That makes the result relevant to organizations comparing benchmark performance with test costs, but the figures do not establish overall value, production costs, or suitability for a particular firm’s matters. Procurement decisions also depend on data handling, security, workflow integration, and human review, which the ranking does not measure.
Products Worth Considering
Apple MacBook Air 15-inch (2026) with M5 chip, 16GB memory, 512GB SSD, Silver
Canon PIXMA TR160 Wireless Portable Color Inkjet Printer
Dell XPS 13 9350 Copilot+ Laptop, 13.4-inch 120Hz Display, Intel Core Ultra 7 256V, 16GB RAM, 512GB SSD, Ubuntu Linux 24.04 LTS
As an affiliate, we earn on qualifying purchases.
How Harvey Grades Legal Agents
Harvey released its Legal Agent Benchmark as open source on May 6, 2026. The benchmark contains 1,250 tasks across 24 legal practice areas. Each task combines an instruction and client-matter materials with a requested work product; more than 75,000 expert-written rubric criteria cover items such as facts, conclusions, citations, recommendations, deadlines, amounts, and formatting.
Harvey describes the all-pass method as a way to assess whether an agent finishes a task to the required standard, rather than awarding a task pass for partial completion. Its published example notes that a deal-team report identifying eight of ten risks would still be materially incomplete. Leaderboard scoring uses a private holdout set intended to mirror the public benchmark’s mix. Z.ai describes GLM-5.3-FlashX-Legal as built on GLM-5.3-Flash and trained further for legal workflows, including research, contract review, and drafting.
Z.ai calls its result “state-of-the-art performance among open-source models” on the evaluation. The company materials cited in the source do not identify the comparison set supporting that wording, and the legal model’s published weights were not found in the source review. The model’s placement on a leaderboard alone does not establish that it is open source under a particular definition.
“A task is marked complete only if every criterion passes.”
— Harvey, describing its benchmark scoring rule
Limits of the Published Results
The leaderboard is a benchmark result, not evidence of how the model performs across live legal matters. The available information does not establish its reliability across jurisdictions, practice areas, changing law, or different client documents. A source review described a four-task spot check in which the legal model cited rulings the reviewer could not find; that small check is not a benchmark and cannot establish a general error rate. The reviewer also found that two Claude models cited a real ruling for the opposite of its holding, underscoring that citation errors are not unique to one system.
The published materials described the model’s outputs as drafts for legal professionals, but the source review found no “not legal advice” notice on the Z.ai pages it examined. Details about client-data processing, applicable safeguards, and responsibility for citation checks were not established in the benchmark results. Nor does the available information explain exactly which models Z.ai included in its “open-source” comparison. The documentation page itself is undated, and the leaderboard figures cited here are from an archived version, so later rankings may differ.
What Firms Should Verify Before Use
Z.ai’s documentation recommends using the model as an agent connected through MCP services to sources such as laws, cases, and company data. Its AutoClaw blog post, dated October 13, announced a beta available by application. The company describes the model as suited to research, contract review, and drafting through initial drafts; those stated uses do not amount to a finding that it can replace professional review.
For firms evaluating it, the next practical steps are to test it on representative work, verify every legal citation against authoritative sources, and check how client information is handled. Human review remains central given the benchmark’s strict all-criteria results and the unverified citation errors noted in the small spot check. Further benchmark updates, more detail on the comparison behind Z.ai’s open-source claim, and clearer information on deployment and data policies could help buyers judge whether the system fits their workflows.
Key Questions
What does GLM Legal’s 19.17% score mean?
On Vals AI’s archived leaderboard, 19.17% of benchmark tasks passed every required rubric criterion, averaged across two AI judges. A task with even one missed criterion counted as a failure.
How can the model have a 92.5% criterion pass rate but a 19.17% task pass rate?
The figures use different units. The model passed 92.5% of individual criteria, but a task passed only when all its criteria passed. Missing one requirement can fail an otherwise mostly complete task.
Did GLM Legal outperform Z.ai’s base model?
Yes, in the archived leaderboard comparison: GLM Legal scored 19.17% on tasks, compared with 6.67% for GLM-5.3-Flash. That is a 12.50-percentage-point difference, consistent with Z.ai’s stated 12.5-point improvement.
Does the ranking show that GLM Legal is ready to give legal advice?
No. The score measures performance on a specific benchmark, not suitability for unsupervised legal practice or performance on every real case. The materials describe outputs as drafts for legal professionals, and citations and work products require review.
What remains uncertain about Z.ai’s open-source claim?
Z.ai calls the result state of the art among open-source models, but the cited materials do not specify the comparison set. The source review also found no published weights for the legal model; the benchmark ranking does not settle what access or licensing terms apply.
Source: ThorstenMeyerAI.com
