🔍 Read the full analysis: The Benchmark That Ensures AI Managers Never Hit Zero on ThorstenMeyerAI.com
Get smart everyday buys delivered free — and shop member deals
- Fast, free delivery on millions of items
- Access to Prime Big Deal Days deals on October 6–7
- Prime Video, Amazon Music and more included
TL;DR
Firmulate has launched a new benchmark measuring AI managers’ ability to handle a company’s worst week. Results show models scoring up to 95, with a minimum baseline of 26 points for minimal effort, highlighting the importance of trust and task completion.
Firmulate has introduced a new benchmark designed to measure the real-world effectiveness of AI managers during a company’s worst week, with the final standings revealing top scores of 95 and a minimum baseline of 26 points for minimal effort. This development shifts focus from traditional language model metrics to practical management capabilities, emphasizing trust and task completion under pressure. For more context, see the original analysis.
The benchmark involved four frontier AI models managing a simulated small software company facing seven days of crises, customer demands, and trust challenges. This approach is discussed in detail in the original analysis. Each model’s decisions were fully auditable, ensuring transparency. The top performer, gpt-5.6-sol, scored 95 points, while the baseline, representing minimal management effort, scored 26 points. Notably, no model achieved a perfect score of 100, which the designers see as a red flag indicating unmeasured gaps or overly optimistic assessments.
The scoring system reflects a core principle: partial progress and trustworthiness are valued over superficial performance. A single breach of trust, such as failing to escalate or verify critical information, caps the total score, underscoring the importance of integrity in AI management. The results reveal that models capable of reading and referencing internal documentation—specifically, those that found hidden data— secured higher deals and revenue, demonstrating the importance of thoroughness.
During stress tests, all models successfully refused social engineering attempts, indicating strength in trust management. However, performance varied in follow-through tasks; some models failed to escalate or complete tasks, highlighting that thoroughness and follow-through are distinct skills. The benchmark’s design ensures that models are assessed on their ability to finish what they start, stay honest, and handle pressure without breaking trust.
Firmulate Benchmark · July 2026
The Benchmark That Ensures AI Managers Never Hit Zero
Four frontier AI models were dropped into a simulated small software company on its worst week — seven days of crises, customer demands, and trust tests. Every decision auditable. Every score earned.
01 · The League Table
How the Week Ended
The final standings reward trust and task completion over conversational polish. Thoroughness with internal documentation — especially hidden data — translated directly into higher deals and revenue. Follow-through, however, proved to be a separate skill entirely.
| Model / Baseline | Score | Performance | Docs Deep-Dive | Refused Social Eng. | Follow-Through |
|---|---|---|---|---|---|
| gpt-5.6-sol | 95 | ✓ Found hidden data | ✓ Passed | ~ Mostly complete | |
| Frontier Model B | — | ~ Partial | ✓ Passed | ~ Mixed | |
| Frontier Model C | — | ~ Partial | ✓ Passed | ✗ Failed escalation | |
| Frontier Model D | — | ✗ Missed | ✓ Passed | ✗ Tasks dropped | |
| Baseline (minimal effort) | 26 | ✗ None | ~ N/A | ✗ None |
One breach of trust — a failure to escalate or verify critical information — caps the total score. Integrity is non-negotiable in the Firmulate scoring system.
02 · The Simulation
Anatomy of a Worst Week
The benchmark replaces language-model metrics with a seven-day operational stress test. Each model managed the same simulated company through escalating pressure.
Crisis Influx
Seven days of escalating incidents, outages, and urgent customer demands arrive simultaneously.
Documentation Dig
Models must read and reference internal docs — hidden data separates the thorough from the superficial.
Trust Gauntlet
Social engineering attempts and unverified claims test whether the manager stays honest under pressure.
Audited Scoring
Fully auditable decisions feed a scoring system where partial progress beats polished conversation.
03 · The Scoring Philosophy
Between 26 and Suspicion
The benchmark recognizes a floor: even minimal management — triaging crises, reading emails — earns 26 points. At the other end, a perfect 100 is treated as a warning sign of unmeasured gaps or overly optimistic assessment.
04 · Why It Matters
What Changes for Business
For companies deploying AI in CRM, support, or decision-making, the July 2026 results reframe selection criteria around three practical pillars.
Integrity Under Pressure
Trustworthiness outranks fluency. The scoring system discourages overestimating AI based on conversation quality and rewards models that stay honest when stakes are high.
Finish What You Start
Thoroughness and follow-through are distinct skills. Some models excelled at discovery yet failed to escalate or complete tasks — a gap invisible to traditional benchmarks.
Auditable by Design
Every decision in the simulation is fully auditable, setting a new standard for transparent, standardized evaluation of AI managing real operations.
05 · Key Questions
The Essentials, Answered
How the benchmark works — and the open questions about translating simulated scores into real-world management.
Why a minimum of 26 points?
It recognizes that even minimal efforts — triaging crises or reading emails — have value, establishing a baseline for basic effective management.
Why did no model score 100?
Flawless performance in complex, trust-dependent tasks remains unmeasured or unrealistic. Designers treat a perfect score as a red flag, not an achievement.
How does scoring emphasize trust?
A single breach — failing to escalate or verify critical information — caps the total. Integrity gates everything else.
What’s next for evaluation?
Future iterations may add complex scenarios, broader stress tests, and longer evaluation periods — while results shape procurement and industry standards.
Why Trust and Completion Matter in AI Management
This benchmark emphasizes that in deploying AI for real business processes, trustworthiness and task completion are more critical than superficial language capabilities. The scoring approach discourages overestimating AI performance based solely on conversation quality, instead rewarding models that demonstrate integrity, thoroughness, and reliability under stress. For companies integrating AI into CRM, support, or decision-making, these results highlight the importance of selecting models capable of finishing tasks and maintaining trust, especially during crises.
The approach also introduces a new standard: a minimum baseline of 26 points for minimal management effort, recognizing that even doing the bare minimum has value. Conversely, achieving a perfect score of 100 is considered suspicious, as it could indicate unmeasured or overly optimistic evaluation metrics. This scoring system aims to promote honesty and transparency in AI performance assessments, aligning AI management with real-world business priorities.
As an affiliate, we earn on qualifying purchases.
The Evolution of AI Benchmarks in Management Tasks
Traditional AI benchmarks primarily measure language proficiency, such as conversational ability or problem-solving in controlled environments. However, as AI tools are increasingly integrated into operational management, the focus has shifted toward their ability to handle real-world tasks—reading documentation, making decisions, managing crises, and maintaining trust. The Firmulate benchmark responds to this shift by creating a scenario where models are tested under the pressures of a simulated company’s worst week, emphasizing practical management skills over theoretical capabilities.
Previous efforts to evaluate AI management performance often lacked transparency or did not account for trust and follow-through. The July 2026 results mark a step forward, establishing a standardized way to measure how well AI models can manage ongoing business operations, especially when stakes are high. This aligns with broader industry trends toward responsible AI deployment and operational reliability.
Leading up to these results, many experts have called for benchmarks that reflect real-world management demands, including handling crises, making decisions based on internal data, and maintaining ethical standards. The Firmulate league’s transparent, auditable scoring system responds directly to these needs, setting a new standard for AI management evaluation.
As an affiliate, we earn on qualifying purchases.
Unanswered Questions About Benchmark Application
It is not yet clear how these benchmark scores will translate to real-world business environments, especially across different industries or larger organizations. The models’ performance in the simulated scenario may differ when faced with actual operational complexity and unpredictability. Additionally, the long-term impact of emphasizing trust and task completion over language prowess remains to be seen, as does the potential for models to improve in these specific areas over time.As an affiliate, we earn on qualifying purchases.
Next Steps for AI Management Evaluation
Following the July 2026 results, industry stakeholders are expected to adopt or adapt the benchmark for their evaluation processes. Companies considering AI tools will likely scrutinize models’ ability to finish tasks and maintain trust under pressure, using the benchmark as a reference. Additionally, further iterations of the league may incorporate more complex scenarios, broader stress tests, and extended evaluation periods to better mirror real-world management challenges.
Developers of AI models will also focus on improving skills related to documentation reading, follow-through, and trust preservation, aiming to score higher in future benchmarks. Meanwhile, researchers and industry leaders will debate the implications of the scoring system, especially the minimum baseline and the absence of perfect scores, shaping future standards for responsible AI deployment.
Finally, broader adoption of such benchmarks could influence regulatory standards and best practices, pushing the industry toward more transparent, trustworthy AI management solutions that are resilient under pressure.
As an affiliate, we earn on qualifying purchases.
Key Questions
What is the main purpose of the new AI management benchmark?
The benchmark aims to evaluate how well AI models can manage a company’s operations during a worst-case scenario, focusing on trustworthiness, task completion, and integrity under pressure.
Why is there a minimum score of 26 points for minimal effort?
The score recognizes that even minimal management efforts—such as triaging crises or reading emails—have value. It establishes a baseline for what constitutes basic effective management.
Why is no model scoring 100 in the benchmark?
The absence of perfect scores indicates that truly flawless performance in complex, trust-dependent management tasks remains unmeasured or unrealistic at this stage.
How does the scoring system emphasize trust?
The system caps scores if an AI breaches trust, such as failing to escalate or verify critical information, highlighting that integrity is non-negotiable in management scenarios.
How might this benchmark influence real-world AI deployment?
It encourages organizations to prioritize models that can reliably finish tasks and maintain trust, potentially guiding procurement decisions and setting new industry standards for responsible AI use.
Source: ThorstenMeyerAI.com
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
