📊 Full opportunity report: Minerva. The opposite path. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
Italy’s Minerva-3B, a sovereign LLM trained from scratch with 50% Italian data, outperforms multilingual models but scores just 4.9% on Italian school exams. This reveals scaling limits in country-specific AI investments.
Italy’s Minerva-3B, a fully open-source sovereign language model trained from scratch on over 2.5 trillion tokens, scored only 4.9% on the INVALSI Italian school-exam benchmark, despite outperforming comparable multilingual models on Italian benchmarks. This development challenges assumptions about the relationship between training scale and language-specific performance, and has significant implications for Europe’s AI strategy.
Minerva was developed by Sapienza University of Rome’s NLP group, led by Roberto Navigli, with support from Italy’s national supercomputing infrastructure and public funding. The project trained models from scratch on a dataset composed of roughly 50% Italian content, totaling 1.14 trillion tokens, and published weights and code openly. While Minerva’s models outperform multilingual counterparts on standard benchmarks, its 3B version scored just 4.9% on the INVALSI Italian school exams, a result considered near chance, highlighting a disconnect between benchmark performance and real-world language understanding.
Researchers noted that larger dataset size and parameter count are critical for handling complex language tasks, suggesting that even substantial native-language investments may be insufficient at current scales. The findings raise questions about the effectiveness and necessary scale of sovereign-language models, especially in European contexts where language-specific data is limited and costly to produce.
Minerva.
The opposite
path.
Italy spent years building a European sovereign LLM from scratch. Then Minerva-3B scored 4.9% on the INVALSI Italian school exam.
Where AMÁLIA layered Portuguese specialization onto a multilingual foundation, Minerva trained from scratch on 2.5 trillion tokens with approximately 50% Italian content. Where AMÁLIA’s weights are not yet public, Minerva published weights, training data, and code as truly-open from day one. By every institutional measure, the Italian approach worked. But the empirical results contain a finding the press coverage has been quiet about — and it has implications that extend well beyond Italy.
Same problem. Opposite path.
European sovereign-LLM development has two primary architectural approaches. Italy chose from scratch with substantial native-language foundation. Portugal chose continuation pre-training of a multilingual model. The structural comparison surfaces what each commitment actually requires operationally.
The comparison is not “Italy did it better than Portugal.” Both projects respond to the same structural problem with different architectural strategies under different institutional and economic constraints. Italy’s national-AI investment is structurally larger by an order of magnitude — and Minerva is the visible artifact of that scale.
Rebooting the Machines: A New Human Vision for Artificial Intelligence
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
4.9% on INVALSI. The bitter lesson surfaces.
In June 2024, researchers evaluated Minerva-3B on the Italian school-exam benchmark. The result was unambiguous. This is not a critique of Minerva — it is a critique of the public discourse around what Minerva’s empirical results actually demonstrate.
Mastering Small Language Models: A Practical Guide to Building Lightweight NLP Systems with Python, Transformers, and Quantization Techniques
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
350M to 7B. Four parameter scales, one architecture.
The Minerva model family covers four parameter tiers, each with specific training corpora. Each scale level reveals what the from-scratch path actually requires at different operating points.
Italian + English
100B English
~50% English
+ 200B code
KNAT Full Study Guide: Smart Edition Academy Kaplan Nursing Entrance Exam Study Manual with 4 Full Length Practice Tests + 500 Realistic Questions + … + Online videos + Online Flashcards
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three answers. Same question.
Minerva, AMÁLIA, and OpenEuroLLM represent the three operational answers to the European sovereign-LLM question. Each makes different architectural and institutional bets. The strategic discourse benefits from treating all three as data points in the same empirical experiment.
AI Model Evaluation
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three standards the movement should adopt.
The structural critique generalizes beyond Minerva. The European sovereign-LLM movement benefits from internalizing these lessons across every subsequent national project. Italy modeled the openness standard; the movement should adopt it as norm.
Minerva is one valid answer to the European sovereign-LLM question. AMÁLIA is another. OpenEuroLLM is potentially a third. The strategic discourse benefits from treating all three as data points in the same empirical experiment rather than as competing national-prestige projects. More analysis like this is needed. Not less.
Implications for European Sovereign-Language AI Strategies
The results from Minerva demonstrate that high-quality, country-specific language models require more than just large datasets and parameters; they demand significant native-language investment at scale. Despite Italy’s substantial effort and resources, the model’s poor performance on complex academic tests indicates that current approaches may not produce the depth of country-knowledge needed for practical applications. This challenges the assumption that training models from scratch on national data alone can meet the demands of complex language understanding, and suggests that European AI strategies must account for the substantial resource commitments required.
For policymakers and researchers, the findings underscore the importance of scaling and data quality over mere dataset size. The case of Minerva highlights the need for a nuanced understanding of the costs and benefits of sovereign-language AI projects, and the potential need for international collaboration or hybrid approaches to achieve meaningful language competence at the national level.
European Sovereign-LLM Development and the Scaling Dilemma
Italy’s Minerva project represents one of the most ambitious efforts to develop a European sovereign language model from scratch, utilizing extensive computational resources and public funding. Unlike Portugal’s AMÁLIA, which incorporated multilingual pretraining with a focus on European Portuguese, Minerva trained on a vast dataset with a high proportion of Italian content, aiming to produce a model tailored for Italian language tasks. Previous European projects have debated the merits of continuation pretraining versus training from scratch, but Minerva’s results suggest that scale remains a fundamental challenge. Despite achieving impressive benchmarks, the poor performance on real-world academic tests exposes the limits of current resource investments in sovereign-language AI at the parameter levels used.
“Minerva’s results challenge the narrative that training from scratch on national data alone can produce deep country knowledge at current scales.”
— Thorsten Meyer
Unresolved Questions on Scaling and Performance Limits
It remains unclear whether further scaling of native-language data and parameters would improve Minerva’s performance on complex language tasks, or if fundamental limitations exist. The ongoing evaluations and iterations by the research team may shed light on these issues, but definitive conclusions are yet to be reached. Additionally, the broader applicability of these findings to other European languages and models is still under discussion.
Future Developments in European Sovereign-Language AI
The Minerva team plans to continue refining their models, including experiments with increased scale and different training methodologies. Policymakers and researchers will closely monitor these developments to assess whether larger investments or hybrid approaches can bridge the performance gap. European AI agencies may reevaluate funding strategies based on these empirical results, emphasizing the importance of scale and data quality in future projects.
Key Questions
Why did Minerva perform poorly on the Italian school exams?
Despite extensive training on Italian data, Minerva’s 3B model scored only 4.9%, indicating that current scale and data quality are insufficient for deep language understanding in complex academic contexts.
Does this mean sovereign-language models are not viable?
Not necessarily. It suggests that current approaches may need to be scaled up significantly or combined with other strategies to produce effective models for complex language tasks.
How does Minerva compare to multilingual models?
Minerva outperforms comparable multilingual models on standard benchmarks but struggles with real-world, language-specific tasks, highlighting the gap between benchmark success and practical language understanding.
What are the implications for European AI policy?
The findings suggest that European AI initiatives must consider substantial resource commitments and realistic expectations regarding native-language model performance at current scales.
Source: ThorstenMeyerAI.com
