🔍 Read the full analysis: One Edit, 21 More Correct: GPT-6 Astra’s Qwen JSON Escaping Test on ThorstenMeyerAI.com
Get smart everyday buys delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
An experiment reported by ThorstenMeyerAI.com found that a rank-one edit to one matrix in Qwen3-4B-Instruct-2507 increased exact answers on a synthetic JSON-escaping test from 49 of 120 to 70. The edited model fixed 21 cases without breaking any on that test, but still failed 50 cases, and its score on a separate table-extraction task remained zero out of 30.
OpenAI’s GPT-6 Astra, running as a Codex agent, made a targeted change to one matrix in the open Qwen3-4B-Instruct-2507 model and raised its exact-answer score on a synthetic JSON-escaping test from 49 of 120 to 70 of 120, according to results published by ThorstenMeyerAI.com. The gain was limited to that task: the model still missed 50 test cases, and its score on a separate table-extraction task stayed at zero out of 30.
The test asked the model to copy short text containing characters such as tabs, line breaks, quotation marks or backslashes into a JSON object. A response counted as correct only if parsing the JSON returned text identical to the original. For example, a tab must be represented with the JSON escape sequence \t; valid JSON that changes or drops a character still fails the test.
The report says the edited model answered 70 of 120 cases exactly, compared with 49 before the edit. That is an increase of 21 correct cases, or 17.5 percentage points. The authors report 21 cases fixed and none broken on this final test. They say the edit was a calculated rank-one update to the output projection of the model’s last MLP layer. The other 397 stored tensors were unchanged, and the process did not use gradient training.
GPT-6 Astra, set to extra-high reasoning effort in the Codex session, planned the experiment, inspected the model weights, prepared and froze the tests, calculated the edit and wrote the report, according to the site. ThorstenMeyerAI.com says the final results are public on Lexassay.com and that its figures refer to 10 published Lexassay tasks. The report identifies Claude as uninvolved in this experiment; a Claude agent later drafted the article and checked its figures against the stored results.
Infographic 01 · The measured result
Where the edit helped, and where it did not
Qwen3-4B-Instruct-2507, original versus one edited matrix, on synthetic texts that must survive the trip into JSON without losing a character.
Final test · 120 cases by group
- Texts with tabsthe target failureOriginal19/30Edit30/30
- Line breaksalready perfectOriginal30/30Edit30/30
- Mixed special characterstabs, quotes, backslashesOriginal0/30Edit10/30
- Second table rowstill unsolvedOriginal0/30Edit0/30
Development, validation and controls
- Developmentused to fit the edit · not independentOriginal24/40Edit40/40
- Validationnew cases · 9 wins, 0 lossesOriginal10/40Edit19/40
- Random directioncontrol on validation · same strengthOriginal10/40Edit10/40
- Plain-text tabscontrol · real tabs must surviveOriginal10/10Edit10/10
Exact answer = after normal JSON parsing, the text field matches the source text character for character. Same prompt, temperature 0, seed and output limit for both models; saved and reloaded edit, no hooks, no grammar constraints, no repair. Final test, validation and controls frozen before any independent output of the edited model. Fifty of 120 final answers are still wrong.
Sources: our experiment records, 5 Nov 2026 (Lexassay research, internal): experiment 3 result report and final result by GPT-6 Astra working as a Codex agent (OpenAI). Synthetic test cases, not published.
Powered by Thorsten Meyer AI
A Narrow Gain in Exact JSON Copying
The result offers a specific example of editing a model’s stored weights to target a known failure, rather than changing only the prompt. In software systems, JSON is commonly used to pass structured data between services and tools. If a model mishandles tabs or other special characters, a response may look plausible while silently altering the underlying text. Exact-copy tests can expose that kind of failure.
But this test does not establish that the edit makes Qwen broadly more reliable, or that it improves performance in real applications. The reported gain applies to a narrow, synthetic task. Half of the final cases were still wrong, and the separate table-row extraction result did not improve. The finding is best read as a measurable change on one frozen benchmark, not evidence of general capability or dependable production performance.
Products Worth Considering
FLEXISPOT Electric Adjustable Workbench, 48 x 30 in, Maple
Dell XPS Tower 8960 Desktop, Intel Core i7-13700, NVIDIA GeForce RTX 4060, 16GB DDR5, 512GB SSD
Dell UltraSharp U2724DE 27-Inch WQHD IPS Black Monitor
As an affiliate, we earn on qualifying purchases.
A Follow-Up to an Earlier Failure
This was the third experiment in a week of work on editing model weights, according to ThorstenMeyerAI.com. In an earlier experiment, the site says Claude edited a local model called Ornith: it answered all 10 published benchmark tasks rather than four, while its legal score remained zero. GPT-6 Astra reviewed that work and identified reported problems including a resume bug, a broken command-line interface and side effects from weight conversion.
Before the Qwen JSON test, GPT-6 Astra had tried another edit to the same model, targeting extraction of fields such as email addresses and due dates from synthetic business messages. The frozen final test showed 97 correct answers out of 200 for both the original and edited versions. The experiment report says both versions also produced 47 invalid answers, many involving raw tabs in JSON strings. That earlier result supplied the clue for the new, targeted test; this was not a broad search across every possible model weakness.
For the JSON-escaping experiment, the site says a clearer prompt alone scored 16 of 32 screening cases, the same as the original model with the initial prompt. The weight edit was evaluated against a separately frozen final set, according to the report. The model is described as an open model under the Apache-2.0 license.
“The edited model got 70 of 120 test cases exactly right instead of 49.”
— ThorstenMeyerAI.com experiment report
Independent Checks Are Still Pending
The site says the experiment has not yet been reviewed by a second AI, unlike the earlier Ornith work, which GPT-6 Astra checked. The published account also does not establish whether the score would hold across other prompts, different test sets or real-world data. The test cases are synthetic, and the available material does not show whether independent researchers have reproduced the result.
The report gives a result for one edited model and one frozen test, but the supplied material does not describe a broader evaluation of possible side effects beyond the named table-extraction task. It is also unclear how the model performs on other formats or tasks that depend on exact text copying. The figures should not be treated as evidence that the model is generally more accurate.
Reproduction and Broader Testing
The experiment’s runs are publicly available on Lexassay.com, according to the November 9 update, giving other researchers a way to inspect the reported work. The next useful steps would be an independent review of the procedure and results, reproduction of the edit, and tests on additional frozen cases to check whether the improvement persists.
Any broader claim would also require evaluating other tasks and checking for unintended changes beyond the reported benchmarks. Until those checks are available, the confirmed result in the source is limited to the reported comparison: 49 to 70 exact answers out of 120 on a synthetic JSON-escaping test, with no reported losses on that test.
Key Questions
What did GPT-6 Astra change?
According to ThorstenMeyerAI.com, it applied a calculated rank-one update to one matrix: the output projection of Qwen3-4B-Instruct-2507’s last MLP layer. The report says the other 397 stored tensors were unchanged and that the edit did not use gradient training.
How much did the JSON test score improve?
The edited model scored 70 of 120 exact answers, compared with 49 before the edit. The report describes that as 21 more correct cases, or a 17.5-percentage-point gain, with no cases broken on this final test.
Does this mean Qwen is now reliable at handling JSON?
No broad reliability conclusion is established by the report. The test was synthetic and narrow, and the edited model still got 50 of 120 cases wrong. Performance on real-world applications was not established in the supplied material.
Did the edit improve the model’s other tested task?
No. The report says the model’s result for extracting a table row remained 0 of 30. A previous field-extraction experiment also showed no difference between the original and edited model on its frozen final test.
Has another researcher independently verified the result?
The source says the experiment has not yet received an independent review. It reports that the runs are public on Lexassay.com, but the supplied information does not confirm independent reproduction.
Source: ThorstenMeyerAI.com
