🔍 Read the full analysis: A Home AI Experiment In Editing Local Models, With A Zero Legal Score on ThorstenMeyerAI.com
Get smart everyday buys delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A home experiment removed a measured looping direction from the weights of local model Ornith 1.5 35B-A3B. In a comparison using the same setup, completed answers rose from 4 of 15 to 12 of 15, but the legal score remained zero; a separate run using the model’s recommended sampling settings completed 13 of 15 without editing.
A home experiment reported November 5 found that editing a local AI model’s weights helped it finish more legal benchmark answers but did not improve its score: Ornith 1.5 35B-A3B completed 12 of 15 questions after the edit, compared with 4 of 15 in the matched comparison, while scoring zero in both runs. The result suggests the intervention reduced looping, not that it gave the model legal knowledge or made its answers reliable.
The experiment ran from Wednesday evening to Thursday morning and was inspired by a report from four MIT CSAIL researchers on using one AI system to edit another model’s weights. The home test did not reproduce that method. Instead, a Claude Opus 5.5 session wrote code to run Ornith, measure a direction associated with its looping, and remove that direction at half strength from the output weights of 27 of the model’s 40 layers. The authors say there was no additional training, legal data or AI system reading the weights as text.
In the matched comparison, the edited model produced 12 completed answers out of 15, up from four. But its legal score remained at 0. The 12 edited answers contained 62 wrong or invented citations, an average of 5.2 per answer. Across 75 checked criteria, the edited run met seven, compared with three before the edit. The source describes the answer content as weak.
A separate run also complicates the apparent improvement: with the model card’s recommended sampling settings and no weight edit, the original model completed 13 of 15 answers and still scored zero. That is not the same as the matched comparison, so the figures should not be treated as a direct head-to-head test under identical settings. The project says a change in local setup alone moved the unedited model from 10 finished answers to four.
Infographic 01 · What we changed
One loop direction, weakened at half strength
Inspired by the MIT post, but a simpler, classic technique (directional ablation): code written by a Claude Opus 5.5 session measured one loop direction by running the model and projected it out of the weights. No training, and no AI reading weights as text.
The six steps
- Write test material that is not the benchmark: 24 legal development questions plus 8 general controls, 0 overlap with benchmark prompts
- Find the true loop onset at token level: from there, at least 90% of the next 1,500 positions repeat earlier 32-token sequences
- Run the model and measure one loop direction per layer: average internal signal inside the loop minus the reasoning just before it
- Weaken it in the weights:
W ← W − α·r·(rᵀW)on the MLP output matrices of all 256 experts and the shared expert - Bake an edited copy; the original files stay untouched and the edited weights stay internal
- Benchmark original and edit on the same local setup, 15 tasks, graded blind by three Claude agents
Quick test: share of pure copying
Restart 20 tokens before a known loop onset, 8 cases, 3,000 tokens. Lower is better.
The cost of the edit
- Full strength: 23 of 24 legal questions finished, but control answers broke: tokens dropped out of short formulas, and month counting went wrong
- Half strength: 23 of 24 finished, 9 of 9 checked control facts right, and one German control question was answered in Chinese once
- Before the edit: 8 of 24 legal development questions did not finish; all 8 controls did
One run per setting. The loop detector was revised three times during the experiment, so the development numbers come from a moving yardstick. The direction is a statistical description, not a single “loop neuron”.
Sources: our experiment records, 4–5 Nov 2026 (executing Claude Opus 5.5 session; review by GPT-6 Astra in Codex); Ornith-1.5-35B-A3B model card.
Powered by Thorsten Meyer AI
More Finished Answers, No Legal Accuracy
The test highlights a distinction that matters to anyone evaluating AI tools: a model can produce a complete response without producing a correct one. If a system gets stuck repeating itself, an intervention that reduces loops may make it easier to use. But completion counts alone do not establish that it knows a subject, cites valid authorities or can be trusted with consequential work.
That distinction is particularly important for legal tasks, where a fabricated statute or court decision can make an answer misleading even when it is fluent and finished. In this experiment, the edited model completed more answers in the matched setup, yet its score stayed at zero and the edited answers contained dozens of incorrect or invented citations. The report therefore offers evidence of a possible change in response behavior, not a demonstration of improved legal performance.
The result also shows why simple settings changes need to be tested alongside changes to model weights. The unedited model completed 13 answers under recommended sampling settings, more than the 12 in the edited matched run. Because the conditions differed, neither number alone establishes which approach performs better overall. The reported figures are a small experiment, not a general performance guarantee.
Products Worth Considering
Apple MacBook Air 15-inch with M5 chip, 24GB RAM, 512GB SSD, Midnight
Dell XPS 13 9350 Copilot+ Laptop, 13.4-inch 120Hz Display, Intel Core Ultra 7 256V, 16GB RAM, 512GB SSD, Ubuntu Linux 24.04 LTS
Apple MacBook Pro MPHF3LL/A 14.2-Inch Laptop | M2 Pro 12-Core Chip 16GB Unified Memory 1TB SSD | 19-Core GPU Liquid Retina XDR Display Space
As an affiliate, we earn on qualifying purchases.
A Different Route From MIT’s Edit
The MIT CSAIL work described by the source calls its approach “weight-space programming”: a general-purpose model reads another network’s weights and edits them directly, without a conventional training loop. In a Qwen example, the researchers attributed looping to a direction in the model’s internal signals. Their report said that removing it was associated with a maths score rising from 7.9% to 48.1%. The home experiment was prompted by that result but used a more established technique, directional ablation, based on measurements taken while running the model.
The source quotes the MIT researchers as cautioning that an edit could remove an obstacle hiding an ability, but could not add an ability the model lacked. The home test was framed as a way to examine that distinction: would reducing looping help a model answer legal questions, or would lack of legal knowledge remain the limiting factor?
Ornith was selected from four local models because the experimenters observed looping in its benchmark responses. The model card describes it as a mixture-of-experts system with about 35 billion parameters in total and about 3 billion active for each token, designed for coding and agent work rather than law. The project’s legal benchmark, Lexassay, asks 15 questions about German, U.S. and EU law without internet access or tools, then checks citations against primary sources. The source says the benchmark includes 10 published tasks and five hidden ones.
Small Test, Several Confounders
The findings are limited by the scale and design of the experiment. The source reports one run per setting, a small development set and a loop detector that changed during the work. It also says three graders from the same model family carried out blind grading, while Claude both performed the edit and supplied those graders. GPT-6 Astra, from a different company, reviewed the code, numbers and claims and identified faults; that review is a check, not a second experiment.
The comparison is further complicated by the reported change in local setup, which moved the unedited model from 10 completed answers to four, and by the separate run using recommended sampling settings. The source does not provide enough information here to determine how much of the difference in completion counts came from the weight edit versus settings or setup. It also does not establish whether the edit would have similar effects on other models, tasks or hardware.
The benchmark figures described in the article are from these experimental runs; the source says the public Lexassay site still displays only the original Ornith result. The test does not establish that any answer is suitable for legal use. The authors explicitly describe the work as an AI experiment, not legal advice.
Further Testing Must Separate Effects
The immediate next step would be a controlled replication that holds the local setup, sampling settings, token budget, grading method and loop detector constant while comparing edited and unedited versions across repeated runs. That would help separate an effect of the weight change from ordinary variation in model output. Testing on the benchmark’s hidden questions would also help show whether any apparent benefit carries over beyond the tasks used during development.
The report does not announce a scheduled replication or a confirmed update to Lexassay. It says the public benchmark page still shows the original Ornith result. For now, the result remains a limited home experiment: the edit was associated with more completed answers in one matched comparison, but the legal score remained zero and a settings-only run completed 13 of 15.
Key Questions
What did the experiment change?
A Claude Opus 5.5 session wrote code that measured a direction associated with looping in Ornith 1.5 35B-A3B, then removed it at half strength from the output weights of 27 of 40 layers. The report says the model was not retrained and no legal data was added.
Did the edited model become better at law?
Not on the reported score. The edited model scored zero, the same as the matched unedited comparison. It met seven of 75 checked criteria, compared with three before editing, and produced 62 wrong or invented citations across 12 completed answers.
How many answers did the model finish?
In the matched comparison, the edited model finished 12 of 15, compared with four of 15 before the edit. In a separate run using recommended sampling settings with no edit, the original model finished 13 of 15. Those separate conditions are not a like-for-like comparison.
Was this the same method used by the MIT researchers?
No. The home test used directional ablation: code ran the model, measured a direction linked to looping and adjusted weights. The MIT work described in the source involved an AI reasoning directly about another model’s weights and editing them.
Can the result show that the edit works on other models?
No. The report describes one model, one experiment per setting and several limitations, including a changing loop detector and local-setup differences. Further controlled testing would be needed to establish whether the effect generalizes.
Source: ThorstenMeyerAI.com
