Coding Robots: The Future of Software Engineering
Agentic AI’s Breakthrough Moment
Last week’s GTC 2025 show may have been agentic AI’s breakout moment, but the core technology behind it has been quietly improving behind the scenes. That progress is being tracked across a series of coding benchmarks, such as SWE-bench and GAIA, leading some to believe AI agents are on the cusp of something big.
The Evolution of AI-Generated Code
It wasn’t that long ago that AI-generated code was not deemed suitable for deployment. The SQL code would be too verbose or the Python code would be buggy or insecure. However, that situation has changed considerably in recent months, and AI models today are generating more code for customers every day.
Benchmarks: Measuring AI Agents’ Capabilities
Benchmarks provide a good way to gauge how far agentic AI has come in the software engineering domain. One of the more popular benchmarks, dubbed SWE-bench, was created by researchers at Princeton University to measure how well LLMs like Meta’s Llama and Anthropic’s Claude can solve common software engineering challenges.
SWE-Bench: A New Era of Code Generation
When the authors submitted their paper, "SWE-Bench: Can Language Models Resolve Real-World GitHub Issues?" to the by the International Conference on Learning Representations (ICLR) in October 2023, the LLMs were not performing at a high level. "Our evaluations show that both state-of-the-art proprietary models and our fine-tuned model SWE-Llama can resolve only the simplest issues," the authors wrote in the abstract. "The best-performing model, Claude 2, is able to solve a mere 1.96% of the issues."
That changed quickly. Today, the SWE-bench leaderboard shows the top-scoring model resolved 55% of the coding issues on SWE-bench Lite, which is a subset of the benchmark designed to make evaluation less costly and more accessible.
GAIA: Measuring AI Agents’ Capabilities
Huggingface put together a benchmark for General AI Assistants, dubbed GAIA, that measures a model’s capability across several realms, including reasoning, multi-modality handling, Web browsing, and generally tool-use proficiency. The GAIA tests are non-ambiguous, and are challenging, such as counting the number of birds in a five-minute video.
BIRD: Measuring SQL Generation
H2O.ai’s software is involved in another benchmark that measures SQL generation. BIRD, which stands for BIg Bench for LaRge-scale Database Grounded Text-to-SQL Evaluation, measures how well AI models can parse natural language into SQL.
The Future of Software Engineering
The rapid progress in generating decent computer code has led some influential AI leaders, such as Nvidia CEO and co-founder Jensen Huang and Anthropic co-founder and CEO Dario Amodei, to make bold predictions about where we will soon find ourselves.
Conclusion
The future of software engineering is rapidly evolving, and AI agents are playing a crucial role in shaping that future. With the rapid progress in generating decent computer code, it’s clear that AI agents are on the cusp of something big. As the core capabilities of Gen AI get better, the number of reasons not to put AI agents into production decreases.
FAQs
Q: What is SWE-bench?
A: SWE-bench is a benchmark created by researchers at Princeton University to measure how well LLMs like Meta’s Llama and Anthropic’s Claude can solve common software engineering challenges.
Q: What is GAIA?
A: GAIA is a benchmark for General AI Assistants that measures a model’s capability across several realms, including reasoning, multi-modality handling, Web browsing, and generally tool-use proficiency.
Q: What is BIRD?
A: BIRD is a benchmark that measures SQL generation, specifically how well AI models can parse natural language into SQL.
Q: What is the future of software engineering?
A: The future of software engineering is rapidly evolving, and AI agents are playing a crucial role in shaping that future. With the rapid progress in generating decent computer code, it’s clear that AI agents are on the cusp of something big.
