In the age of artificial intelligence, many assume that more thorough, rule-based systems naturally outperform their peers. But recent experiments challenge that assumption, revealing that diligence alone doesn’t guarantee success—especially when it comes to real-world decision-making.
Prime made for students and young adults
- Fast, free delivery for dorm and study essentials
- Prime Video and Amazon Music included
- Member-only deals
The Firmulate Experiment: Testing AI in a Business Environment
Firmulate, a platform dedicated to benchmarking AI decision-making, recently conducted a groundbreaking test involving four advanced AI models. The goal? To simulate running a small software company through its worst week—complete with crises, customer requests, and ethical dilemmas. Each model faced identical scenarios, with decisions being carefully versioned and auditable, ensuring transparency and fairness in testing.
Measuring AI’s Business Judgment
The models included:
- gpt-5.6-sol, scoring 95
- Kimi K3, scoring 93
- Sonnet 5, scoring 88
- Fable 5, scoring 77
For context, a do-nothing baseline scored just 26, underscoring how little progress some models make without targeted effort. Interestingly, despite their high scores, the models’ ability to conclude the scenario successfully varied significantly.
Key Findings: Diligence vs. Impact
All four AI models demonstrated formidable awareness—they detected every crisis presented, and all refused manipulative tactics, such as social engineering attempts or impersonation tricks. For example, when fake CEO messages escalated over three stages and a reporter posed a subtle yes/no question, all models refused to sign any questionable deals.
However, the story took a different turn when it came to closing the deal. Only two models, gpt-5.6-sol and Kimi K3, ultimately signed the €55,000 contract. Despite identical diagnoses and pitches, the other models fell short. Why?
The critical weakness was not in understanding or initial decision-making but in follow-through. The most thorough participant, Opus 4.8, which incorporated over 80 learned rules and provided the deepest analysis, finished last in the final step. Its discipline slipped, and the close was left on the table—an outcome that exemplifies the difference between diligence and impact.
The Hidden Factor: Reading Beyond Surface Data
What made the difference? The models that succeeded looked two document references deep into the company’s own files, uncovering a buried fact that provided the decisive advantage. This overlooked information allowed them to close the deal at full price—equivalent to over €4,583 MRR (monthly recurring revenue). The models that missed this key insight failed to sign, despite correct diagnoses and good pitches.
Social Engineering Under Scrutiny
The experiment also tested resilience against social engineering. Fake CEO messages generated over multiple escalation stages, along with a subtle reporter query, were used as test cases. All models refused to approve anything suspicious, with Kimi K3 explicitly treating such requests as potential impersonation attempts. This demonstrates that advanced AI can uphold integrity under pressure, a vital trait for real-world deployment.
Popular right now
KitchenAid Classic Series 4.5-Quart Tilt-Head Stand Mixer, Onyx Black
Vitamix Propel Series 750 Professional-Grade Blender, 4 Programs, Black
Bose QuietComfort Headphones (2nd Gen), Wireless Noise-Cancelling Over-Ear Headphones, Black
As an affiliate, we earn on qualifying purchases.
What This Means for Business AI Deployment
For companies considering AI solutions, the findings are instructive. The ability to identify crises and refuse manipulative tactics is essential but insufficient. Real impact depends on thorough reading, prioritization, and follow-through. The AI must not only understand the surface but also dig into relevant data—sometimes several layers deep—to uncover critical facts that can seal deals or prevent losses.
Real-World Application: Wargaming Your AI Workforce
Firmulate offers a unique platform where enterprises can simulate their AI’s decision-making in a controlled environment, mirroring real crises and economic mechanics. This ‘wargame’ allows organizations to evaluate how their AI would perform in high-pressure scenarios—before deploying it in the wild.
Watch the live experiment at firmulate.com/live, where you can see AI models navigating a complex, simulated business environment, making decisions, and facing consequences in real-time. The platform also provides a quiz to test your understanding of AI decision-making and offers pilot programs tailored to your enterprise.
The Takeaway: Diligence Is Not Enough
The Firmulate experiment underscores an important lesson: in AI-driven business decisions, volume and rule-following are not substitutes for strategic prioritization. A system that reads deeper, commits to follow-through, and maintains discipline under pressure will outperform one that simply covers more ground or learns more rules.
For decision-makers, this means investing not just in AI models that excel at detection and refusal but also in those that can consistently close the deal—by reading the fine print, prioritizing impactful actions, and holding discipline until the very end.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
![Terence Tao: Math 2.0 [Pdf]](https://acceptedblog.com/wp-content/uploads/2026/10/terence-tao-math-2-0-pdf-featured-560x300.jpg)
