The Elusive Quest for Artificial General Intelligence
In 2019, AI researcher François Chollet designed a puzzle game meant to be easy for humans but hard for machines. The game, called ARC, became an important way for experts to track the progress of artificial intelligence and push back against the narrative that scientists are on the brink of building AI technology that will outsmart humanity.
The Challenge of ARC
The game, which stands for Abstraction and Reasoning Corpus, tests the ability to quickly identify visual patterns based on just a few examples. To play the game, you look closely at the examples and try to find the pattern. Each example uses the pattern to transform a grid of colored squares into a new grid of colored squares. The pattern is the same for every example.
The Evolution of AI
For years, these puzzles proved to be nearly impossible for artificial intelligence, including chatbots like ChatGPT. AI systems typically learn their skills by analyzing huge amounts of data culled from across the internet. That meant they could generate sentences by repeating concepts they had seen a thousand times before. But they couldn’t necessarily solve new logic puzzles after seeing only a few examples.
The Breakthrough
In December, OpenAI said that its latest AI system, called OpenAI o3, had surpassed human performance on Mr. Chollet’s test. Unlike the original version of ChatGPT, o3 was able to spend time considering different possibilities before responding.
The Debate
Some saw it as proof that AI systems were approaching artificial general intelligence, or AGI, which describes a machine that’s as smart as a human. Mr. Chollet had created his puzzles as a way of showing that machines were still a long way from this ambitious goal. But the news also exposed the weaknesses in benchmark tests like ARC, short for Abstraction and Reasoning Corpus.
The New Benchmark
In response, Mr. Chollet teamed up with Mike Knoop, co-founder of the software company Zapier, to create what they called the ARC Prize. The pair financed a contest that promised $1 million to anyone who built an AI system that exceeded human performance on the benchmark, which they renamed "ARC-AGI." Companies and researchers submitted over 1,400 AI systems, but no one won the prize. All scored below 85 percent, which marked the performance of a "smart" human.
The Evolution of ARC
OpenAI’s o3 system correctly answered 87.5 percent of the puzzles. But the company ran afoul of competition rules because it spent nearly $1.5 million in electricity and computing costs to complete the test, according to pricing estimates. OpenAI was also ineligible for the ARC Prize because it was not willing to publicly share the technology behind its AI system through a practice called open sourcing.
The Future of AI
As OpenAI and other companies continue to improve their technology, they may pass the new version of ARC. But that does not mean that AGI will be achieved. Judging intelligence is subjective. There are countless intangible indicators of intelligence, from composing works of art to navigating moral dilemmas to intuiting emotions.
Conclusion
The quest for artificial general intelligence is ongoing, with researchers like Mr. Chollet pushing the boundaries of what is possible. While AI systems are improving, they still struggle with tasks that come naturally to humans, such as navigating the physical world. The journey to AGI will require continued innovation and creativity.
FAQs
Q: What is the ARC game?
A: The ARC game is a puzzle game designed to be easy for humans but hard for machines, created by AI researcher François Chollet.
Q: What is the goal of the ARC game?
A: The goal is to track the progress of artificial intelligence and push back against the narrative that scientists are on the brink of building AI technology that will outsmart humanity.
Q: How do AI systems learn?
A: AI systems typically learn their skills by analyzing huge amounts of data culled from across the internet.
Q: What is artificial general intelligence?
A: Artificial general intelligence, or AGI, describes a machine that’s as smart as a human.
