Why I Didn’t Choose The Best-Performing AI Setup For My News Pipeline

🔍 Read the full analysis: Why I Didn’t Choose The Best-Performing AI Setup For My News Pipeline on ThorstenMeyerAI.com

Before you orderOffer from Amazon

Get smart everyday buys delivered free with Prime

• Fast, free delivery on millions of items
• Prime Video, Amazon Music and more included
• Member-only deals all year

Start your free Prime trial →Free trial for eligible customers · Cancel anytime

As an affiliate, we earn on qualifying purchases.Hide

Thorsten Meyer says he will use GPT-6 Luna for his news pipeline after testing 71 model configurations and finding that the higher-scoring GPT-6 Astra cost about 100 times more per article. The results, published October 13, describe a test of article writing from real news sources in English and German, including checks for source fidelity and fabricated details.

In the standard test, GPT-6 Astra scored 4.00 out of 5 in both English and German and received a perfect source-fidelity score. In a follow-up run of 12 articles, it averaged 3.93. Meyer reports that its score interval did not overlap with Luna’s in that run, suggesting a clearer difference than the small initial sample alone could establish.

GPT-6 Luna averaged 3.67 in the robustness run, with source fidelity of 4.71 out of 5. Every Luna article in the test produced output the pipeline could use after its normal repair step. Meyer reports a cost of about $0.0012 per article during testing, compared with about $0.12 for Astra. In production over the prior week, Luna averaged about $0.0013 per article and took a median of 17 seconds; Astra took about 50 seconds.

The evaluation covered two real sources: a 6,340-character product filing and a 619-character political news item. Each configuration wrote in English and German. Two AI judges, GPT-6.1 Sol and Claude Opus 5.5, scored the articles on source fidelity, fabrication, clarity about uncertainty, news value, structure and polish, with German language quality also assessed. Meyer says the strongest configurations then faced 10 additional source-based articles, mostly short ones.

Cost and Accuracy in Daily Publishing

The results highlight a practical trade-off for publishers producing articles at volume: the strongest score may not justify the added cost if a cheaper model clears the required quality bar consistently. In Meyer’s figures, Luna’s per-article cost was a small fraction of Astra’s, while its follow-up score remained below Astra’s. That is a measured difference in this test, not proof that Luna will suit every newsroom or source type.

The evaluation also suggests that a model’s standing on one benchmark may not predict its performance in another job. Meyer reports that Claude Opus 5.5, which he describes as the best available coding model on the FrontierCode benchmark, scored 3.52 in the news-writing robustness run. Claude Sonnet 5.5 scored near Luna in the standard language tests, but its estimated cost was about 30 times higher. The findings support task-specific testing before adopting a model for a production workflow.

For readers, source fidelity matters because invented quotations and unsupported details can make a news article misleading. Meyer’s test treats valid output and fidelity as practical requirements, alongside writing quality. Those checks cannot establish that every future article will be accurate, but they make model failures visible in the sample.

A Test Built Around Real Sources

Meyer says he used the same production prompt and repair step as his news pipeline. The study included API, subscription and free-tier models, as well as open-weight models run on his own hardware. Its stated scope was a model’s ability to turn a supplied news source into a finished article, rather than a general measure of intelligence.

The test used only two sources for its standard comparison, so Meyer set a rule that score gaps below about 0.3 points across those articles should be treated as noise. He says the follow-up run is a more reliable basis for ranking the strongest configurations. The reported robustness scores include 95% bootstrap intervals, a way to show uncertainty around estimated results.

Meyer also tested GPT-6 Luna at six reasoning-effort settings. The reported English and German scores varied, but he says the differences were within the noise. The highest setting cost about 12 times as much and took about 12 times as long as the no-effort setting, without a corresponding score gain. He kept the production setting at none.

“For a high-volume task, the right model is the cheapest one that clears your quality bar reliably, not the one at the top of the chart.”

— Thorsten Meyer, in the October 13 report

Limits of the Published Comparison

The standard comparison used two source articles, and the available report describes only a 10-article follow-up for the strongest configurations. It does not establish how the rankings would hold across a larger range of topics, longer periods, different prompts or other publishers’ workflows. The scores come from two AI judges; the source material does not describe an independent human review of every article.

The report gives model costs and timing for the tested setup, but does not specify whether those figures include every operational expense, such as infrastructure, monitoring or editorial review. It also does not provide a complete account of the final fallback arrangement: the supplied source says GPT-6 Luna now runs first, then ends mid-sentence while describing what follows.

The test’s judgments about model rankings, quality thresholds and whether score differences represent noise are Meyer’s interpretations of his results. The report does not establish that the same model choice would be appropriate for every publication or use case.

Luna Remains the Production Choice

Meyer says GPT-6 Luna remains in production at the no-reasoning setting. He reports that two backup models were removed from the news route on October 7 after the test exposed failures. GPT-4.1 nano scored 1.62 in English and 1.68 in German, with a source-fidelity score of 1.00; the judges flagged invented quotations attributed to an industry analyst and an anonymous company spokesperson. The locally run gpt-oss-120b produced no valid articles among four attempts and generated named quotations the source did not support, according to Meyer.

The published account does not specify the current replacement or full sequence of fallback models. Further results would be needed to show whether Luna continues to meet the pipeline’s quality threshold across a broader set of sources and over time. For now, Meyer’s reported decision is based on the test and production measurements available through October 13.

Key Questions

Why did Meyer choose GPT-6 Luna if GPT-6 Astra scored higher?

Meyer reports that Astra scored higher, but cost about $0.12 per article, compared with roughly $0.0012 for Luna in the test. He says Luna’s follow-up score and source-fidelity results were sufficient for his production needs.

Did higher reasoning effort improve Luna’s articles?

In Meyer’s six-setting test, scores varied but he judged the differences to be within the noise. The maximum setting cost and took about 12 times as much as the no-effort setting, with no corresponding gain in the reported scores.

What sources were used in the evaluation?

The standard test used a 6,340-character product filing and a 619-character political news item. Meyer says the additional robustness run used 10 more real sources, mostly short ones.

Why were two fallback models removed?

Meyer says GPT-4.1 nano generated unsupported quotations, while gpt-oss-120b failed to produce valid output in all four attempts and also generated named quotations that were not supported by the source.

Does the test show Luna is the best model for every newsroom?

No. The findings concern Meyer’s prompt, sources, languages, judges and production setup. The report does not establish how the results would transfer to other workflows or a broader set of articles.

Source: ThorstenMeyerAI.com