TL;DR
Get smart everyday buys delivered free — and shop member deals
- Fast, free delivery on millions of items
- Access to Prime Big Deal Days deals on October 6–7
- Prime Video, Amazon Music and more included
A live experiment by Firmulate demonstrates that AI models can diagnose problems but often fail to complete actions necessary for business success, as detailed in the original analysis. The ongoing company reveals insights into automation’s challenges and limitations, which are explored in more detail in the original analysis.
Firmulate has launched a live experiment where a synthetic workforce of 13 AI models manages a software company, exposing critical gaps between diagnosis and execution that impact real-world business outcomes.
In this ongoing experiment, the company faces a monthly burn of €105,000 against €2,300 in recurring revenue, with every workday versioned and publicly documented. The AI models identify crises, analyze problems, and produce recommendations, but only a few manage to close deals or execute critical actions that generate revenue, illustrating the importance of the original analysis.
Notably, models that follow detailed evidence trails succeed in securing deals, while thorough analysis alone does not guarantee better management. For example, two models identified the same customer opportunity, but only those that traced deeper into documents closed a €55,000 deal, adding €4,583 in monthly recurring revenue. Meanwhile, models that focus solely on analysis without completing actions underperform or fail to generate revenue.
The experiment also tested trust and decision-making discipline, with all models refusing fake approval requests, emphasizing the importance of evidence retrieval and disciplined execution over superficial analysis. The final leaderboard ranked GPT-5.6-SOL first with a score of 95, while Opus 4.8, despite producing the most rules and analysis, finished last with a score of 73, highlighting that more analysis does not equal better management.
Implications of AI Management Limitations for Businesses
This experiment underscores that AI systems can diagnose problems convincingly but often struggle to carry out decisive actions that impact revenue and organizational health. For companies adopting AI automation, success depends not just on analysis but on the ability to execute and complete critical tasks reliably. The live experiment offers a transparent view of the challenges and risks of relying solely on AI for management decisions, emphasizing the importance of disciplined execution and evidence-based action.
As an affiliate, we earn on qualifying purchases.
Live Experiment Demonstrates AI Management Challenges
Firmulate’s experiment is unprecedented in its transparency, connecting AI decision-making directly to cash flow, customer outcomes, and organizational memory. The company runs every workday with versioned decisions, exposing failures and successes publicly. Past AI management tests often focused on isolated tasks, but this experiment pushes the boundary by managing an entire company, revealing that thorough analysis alone does not guarantee success. The results challenge assumptions that more analysis automatically leads to better management, showing instead that disciplined execution is critical.
“Insight mattered only when it survived the full journey from discovery to disciplined execution.”
— an anonymous researcher
As an affiliate, we earn on qualifying purchases.
Unresolved Questions About AI’s Role in Business Management
It remains unclear how much of the observed performance gap is due to AI model limitations versus organizational or process factors. The long-term scalability of such AI-driven management approaches and their integration into real businesses are still untested and uncertain. Further results are needed to determine whether improvements in AI discipline and evidence retrieval can close the execution gap.
AI decision-making support systems
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Next Steps for Evaluating AI-Managed Business Operations
Firmulate plans to continue the live experiment, with upcoming updates on whether AI models can improve in completing actions that lead to revenue. Additionally, broader industry interest is growing in testing AI’s capacity to manage real companies, which may influence future automation strategies. Observers will watch for whether the experiment’s insights translate into practical, scalable management solutions.
As an affiliate, we earn on qualifying purchases.
Key Questions
What is the main purpose of Firmulate’s live experiment?
The experiment aims to test whether AI models can manage an entire company effectively, focusing on the gap between diagnosis and execution and how that impacts business outcomes.
What are the key findings from the experiment so far?
While AI models can identify problems and produce recommendations, success depends on their ability to complete actions. Thorough analysis alone does not guarantee better results; disciplined execution is critical.
Why is this experiment significant for businesses considering AI automation?
It highlights that AI’s value lies not just in diagnosing issues but in reliably executing decisions that drive revenue and organizational success. Failure to complete actions can undermine potential gains.
What remains uncertain about AI’s role in management?
It is still unclear whether AI can consistently overcome execution gaps at scale, and how organizations can best integrate AI decision-making with disciplined follow-through in real-world settings.
Source: ThorstenMeyerAI.com
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
