Revised Results for OpenAI’s o3 AI Model: A Higher Price Tag
When OpenAI unveiled its o3 “reasoning” AI model in December, the company partnered with the creators of ARC-AGI, a benchmark designed to test highly capable AI, to showcase o3’s capabilities. Months later, the results have been revised, and they now look slightly less impressive than they did initially.
A Higher Computing Cost
Last week, the Arc Prize Foundation, which maintains and administers ARC-AGI, updated its approximate computing costs for o3. The organization originally estimated that the best-performing configuration of o3 it tested, o3 high, cost around $3,000 to solve a single ARC-AGI problem. Now the Arc Prize Foundation thinks that the cost is much higher — possibly around $30,000 per task.
A Proxy for o3 Pricing
The revision is notable because it illustrates just how expensive today’s most sophisticated AI models may end up being for certain tasks, at least early on. OpenAI has yet to price o3 — or release it, even. But the Arc Prize Foundation believes OpenAI’s o1-pro model pricing is a reasonable proxy.
For context, o1-pro is OpenAI’s most expensive model to date.
A Closer Comparison
“We believe o1-pro is a closer comparison of true o3 cost … due to amount of test-time compute used,” Mike Knoop, one of the co-founders of the Arc Prize Foundation, told TechCrunch. “But this is still a proxy, and we’ve kept o3 labeled as preview on our leaderboard to reflect the uncertainty until official pricing is announced.”
A High Price for o3 High
A high price for o3 high wouldn’t be out of the question, given the amount of computing resources the model reportedly uses. According to the Arc Prize Foundation, o3 high used 172x more computing than o3 low, the lowest-computing configuration of o3, to tackle ARC-AGI.
Pricing Rumors
Moreover, rumors have been flying for quite some time about pricey plans OpenAI is considering introducing for enterprise customers. In early March, The Information reported that the company may be planning to charge up to $20,000 per month for specialized AI “agents,” like a software developer agent.
Efficiency Concerns
Some might argue that even OpenAI’s priciest models will cost well under what a typical human contractor or staffer would command. But as AI researcher Toby Ord pointed out in a post on X, the models may not be as efficient. For example, o3 high needed 1,024 attempts at each task in ARC-AGI to achieve its best score.
Conclusion
The revised results for OpenAI’s o3 AI model demonstrate the high computing costs associated with advanced AI capabilities. While OpenAI has yet to officially price o3, the Arc Prize Foundation’s update suggests a potentially higher cost than initially estimated. As the company plans to introduce pricey plans for enterprise customers, the efficiency of these models will be crucial to their adoption and viability.
FAQs
Q: What is the revised estimated cost of o3 high to solve a single ARC-AGI problem?
A: The Arc Prize Foundation now estimates that o3 high may cost around $30,000 per task, significantly higher than the original estimate of $3,000.
Q: Why is the Arc Prize Foundation using o1-pro as a proxy for o3 pricing?
A: The Arc Prize Foundation believes that o1-pro is a closer comparison of true o3 cost due to the amount of test-time compute used. However, this is still a proxy, and the organization has kept o3 labeled as preview on its leaderboard to reflect the uncertainty until official pricing is announced.
Q: Will OpenAI’s priciest models be more cost-effective than human contractors or staff?
A: According to AI researcher Toby Ord, OpenAI’s priciest models may not be as efficient as human contractors or staff, which could impact their adoption and viability.
