🔍 Read the full analysis: OpenAI’s Agent Training In Software: A Closer Look At Ironclad’s Terms on ThorstenMeyerAI.com
Get smart everyday buys delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
OpenAI’s October 6 post describes training GPT-6 Astra on tasks inside Ironclad’s contract-management software, using hosted test environments and synthetic tasks based on public SEC-filed contracts. Astra met an average 55% of evaluation criteria; OpenAI says its time estimates were simulated, and the results do not establish customer productivity gains or readiness for unsupervised use.
OpenAI reported on October 6 that it trained and evaluated GPT-6 Astra on 11 legal, commercial and procurement tasks inside Ironclad’s contract-management software. The results offer a look at how OpenAI is teaching models to use specialized business applications, but the company’s reported 55% average rubric score is not a task-completion rate, and its time estimates were simulated rather than measured customer savings.
Ironclad staff and OpenAI employees who use the product selected 11 workflows, including creating nondisclosure agreements, setting up procurement approval processes and changing a reusable contract clause based on a requester’s jurisdiction. OpenAI estimated that an experienced user would take 30 to 40 minutes to complete each task. Evaluators scored each task against a rubric of 8 to 50 criteria, depending on its complexity.
Ironclad provided hosted copies of its software for model practice. OpenAI says it created synthetic training tasks from contracts publicly filed in the SEC’s EDGAR database, filtered to remove personal information. The company says it did not use OpenAI customer data, its internal contracts or non-public Ironclad customer data. The post identifies GPT-6 Astra as the first frontier model trained through this approach.
OpenAI reported that Astra met an average 55.0% of the criteria, compared with 41.6% for GPT-5.6 Sol at a high setting; an internal model used during Astra’s development scored 63.7%. Astra met about 94% of the criteria on one showcased task. The reported average is the share of rubric requirements met across the evaluation, not the percentage of tasks completed successfully. OpenAI also estimated 19.2 minutes per Astra attempt, versus 37 minutes for GPT-5.6 Sol, but says these figures are simulated from assumed processing and generation speeds.
OpenAI is training agents inside your software. Read the fine print on Ironclad.
Several AI trackers guessed “Ironclad” was a hardened agent framework. It’s a contract-management software company — and the post describes OpenAI training its frontier model inside a vendor’s real product, then inviting other vendors to do the same.
legal, commercial & procurement — e.g. NDAs, approval flows, jurisdiction clauses
per task, experienced user (OpenAI estimate)
criteria per task — a rubric, not pass/fail
public SEC filings; no customer or non-public Ironclad data
The average share of rubric criteria met — not tasks completed. In contracting, partial credit isn’t partial value: a workflow that skips one required approval is the exact failure the system exists to prevent.
37.0 → 19.2 minutes are “simulated estimates … not measured customer time savings,” per OpenAI’s own footnote. Credit to OpenAI for saying so plainly.
Its hardest customer problems get built into the next frontier model; agents that work well in its product make the product more valuable.
Every improvement makes the model better at operating the vendor’s interface. Taken far enough, the agent becomes the interface.
Averages hide missed approvals.
Narrowest access; no self-escalation.
METR found agents spoofing tool-call records.
Measure the whole loop.
Public filings, not your contracts.
Modest numbers, significant method. A frontier lab is moving from general computer use to training inside specialised business software, with the vendor’s help — agents learning their trade the way people do. Today: just over half of a contracting workflow’s requirements, in simulated time, on 11 research tasks.Software vendors are becoming training grounds for the agents that may one day operate their products for them.
Why Contract Workflow Accuracy Matters
The evaluation points to a challenge beyond whether an agent can operate a software interface: it must follow multiple business rules at once. A procurement workflow might require Finance approval above a spending threshold, a Security review for certain requests and Legal review for nonstandard terms. Missing one required condition could send a purchase through without an approval, even if the system completed other steps correctly.
That makes a partial rubric score hard to translate into usable work. OpenAI’s own post says a model losing track of a business rule limits what software companies can confidently ask it to do, and argues that human oversight remains necessary. Ironclad CTO Sunita Verma similarly emphasized the need for agents to preserve the controls teams rely on. The results show progress on a test, not evidence that agents can safely run these workflows without review.
For software providers, partnering could help expose weak points in difficult workflows and potentially make their products more useful. It also raises a business question: as customers give instructions to agents rather than interact with screens, a vendor’s value may depend more on its rules, records, audit trails and controls than on its interface. That is an implication of the approach, not a reported outcome of this evaluation.
Products Worth Considering
FLEXISPOT Standing Desk with Monitor Riser & Cable Management
Apple iMac 2024 All-in-One Desktop with M4 Chip, 24‑inch Retina Display, 16GB Unified Memory, 256GB SSD, Silver
Apple iPad Air (2025) 13‑inch Wi‑Fi 256 GB – Space Gray (M3 chip)
As an affiliate, we earn on qualifying purchases.
How OpenAI Tested Ironclad Workflows
The October 6 post paired a model-training account with a call for other software companies to work with OpenAI on tasks current agents cannot reliably complete. The proposed partners would provide a concrete example of a failure, people with deep knowledge of the work, a secure testing environment and data that can be used safely for research. The post describes a research and partnership effort; it does not announce a general release of an agent for Ironclad customers.
The choice of legal and procurement tasks matters because their requirements can be consequential and specific to a company’s own rules. OpenAI says the Ironclad work aimed to train models to understand business rules, complete multi-step workflows in specialized software and check completed work against the original requirements. The evaluation’s 11 selected tasks provide a limited test set, not a measure of performance across Ironclad’s full product or every customer’s processes.
OpenAI’s stated data boundaries also define what the post confirms: its synthetic tasks were based on public filings, and the company says it excluded the listed categories of private customer and internal contract data. The post does not provide a broader independent audit of the training process or evaluation.
What the Evaluation Does Not Establish
The post does not show how Astra would perform across Ironclad’s broader workflows, different customer configurations or live production conditions. It also does not report customer deployments, independently verified results or whether the model’s errors clustered around particular high-risk requirements. The 55% figure is an average criteria score; without a detailed account of which criteria failed on each task, readers cannot determine how often a missed requirement would block a workflow or create a material problem.
The estimated task times are not observed work times, and OpenAI says they do not represent measured customer savings. The post provides no evidence that Astra was faster than an experienced user at producing a correct, fully checked result. It also does not specify when partner projects will begin, which companies will participate or what safeguards would govern any later customer-facing use.
What OpenAI’s Partner Invitation Requires
OpenAI says it is inviting a small number of software companies to partner on tasks that today’s agents cannot reliably complete. Interested companies are asked to bring a specific failure case, knowledgeable staff, a secure testing environment and research data that can be used safely. The post does not give a schedule or name additional partners.
For businesses considering agents in contract, finance or customer-record systems, the immediate practical issue is not a promised productivity gain but how performance will be checked. Buyers can ask vendors to disclose the evaluation criteria, identify which requirements failed, explain how approvals and exceptions are protected, and specify when a human must review an action. OpenAI’s post does not answer those operational questions for customer deployments; further partner testing and clearer deployment details would be needed to judge readiness.
Key Questions
What did OpenAI and Ironclad test?
They evaluated models on 11 legal, commercial and procurement workflows in hosted copies of Ironclad’s contract-management software. Examples included creating nondisclosure agreements and configuring procurement approvals.
Does Astra’s 55% score mean it completed 55% of the tasks?
No. OpenAI says 55% is the average share of rubric criteria met. It is not a task-completion rate, and the post does not establish that a workflow with some criteria met is safe or usable without review.
Did OpenAI measure customer time savings?
No measured customer savings were reported. OpenAI describes the 19.2-minute Astra estimate as simulated, based on assumed processing and generation speeds, and limited to the research tasks.
What data did OpenAI say it used?
OpenAI says its synthetic tasks were based on public contracts filed in the SEC’s EDGAR database, with personal information filtered out. It says it did not use OpenAI customer data, OpenAI internal contracts or non-public Ironclad customer data.
Can companies use Astra to run contract workflows without supervision?
The post does not establish that. OpenAI says human oversight still matters, and the reported average criteria score does not show readiness for unsupervised production use.
Source: ThorstenMeyerAI.com
