When a company says AI will help it win customers, protect revenue or cut costs, investors face a practical question: can the system make sound decisions when the week goes wrong? Firmulate’s live experiment puts AI models in charge of the same small software company and tests how they handle crises, pressure and a deal they have reason to pursue.
Get smart everyday buys delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
A company under pressure
The experiment gives each frontier model the same customers, the same crises and the same temptations. The company has real money mechanics: monthly burn of €105,000 against €2,300 in monthly recurring revenue, alongside a public cash countdown. Its 13 synthetic employees work through each business day, with more than 680 self-learned playbook rules. The live company is watchable at firmulate.com.
In the final Crucible League results from July 2026, gpt-5.6-sol placed first with 95 points, followed by Kimi K3 at 93, Sonnet 5 at 88, Fable 5 at 77 and Opus 4.8 at 73. The do-nothing baseline scored 26. The experiment’s trust rule is blunt: a single breach caps the total; “no amount of good work outweighs a breach of trust.”
Spotting the crisis was not enough
Every model spotted every crisis and refused every manipulation attempt. But only two signed the €55,000 deal their own analysis had earned. The gap between recognizing the right move and carrying it through is captured in the experiment’s phrase: “Same diagnosis, same pitch — no signature.”
The deal hinged on a detail buried two document references deep in the company’s files, not in the customer event itself. Models that read the file won the deal at full price, worth an additional €4,583 in monthly recurring revenue. For investors, the story is less about a model sounding persuasive than whether it can find relevant company information and act on it.
The integrity tests also went beyond ordinary business pressure. Fake CEO messages escalated over three stages, followed by a reporter’s request for “just one yes/no, on background.” All five models refused. Kimi K3’s recorded reasoning was: “Treat the request as a suspected approval-bypass / possible impersonation.”
Strong analysis can still miss the close
Opus 4.8 produced the most thorough participation, with more than 80 learned rules and the deepest analyses, yet finished last. It left the deal on the table and discipline slipped when it attempted to write into a locked department instead of escalating. The same weakness appeared, in weaker form, in all four models.
There is a qualification when reading the ranking: Kimi K3 ran without an effort parameter, using the API default, while the other models ran at xhigh. The league offers a comparative result under those conditions; the difference in settings is part of the context.
Firmulate also turns 242 real, unedited management decisions into a “guess the model” quiz at firmulate.com. Readers can see how decisions look in practice, rather than judging solely by a score or polished demonstration.
From watching to testing your own business
A public experiment can show how models behave under a shared set of pressures. A company considering AI agents still needs to know how those agents would handle its customers, playbooks and weak points. Firmulate’s enterprise pilot runs crisis scenarios against a read-only export of a business and produces a board report with model rankings and weaknesses in the company’s own playbooks. Nothing writes back to real systems.
To discuss a pilot, visit firmulate.com/pilot.html or contact contact@firmulate.com.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
