Comparing AI Chatbots for Programming: A Comprehensive Review
The Surprising Results
I’ve been around technology long enough that little excites me, and even less surprises me. But shortly after Open AI’s ChatGPT was released, I asked it to write a WordPress plugin for my wife’s e-commerce site. When it worked, I was indeed surprised. That marked the beginning of my deep exploration into chatbots and AI-assisted programming. Since then, I’ve subjected 11 large machine models (LLMs) to four real-world tests.
The Verdict
Unfortunately, not all chatbots can code alike. It’s been 18 months since that first test, and even now, five of the 11 LLMs I tested can’t create working plugins. In this article, I’ll show you how each LLM performed against my tests. Two chatbots I recommend using, but they cost $20/month. The free versions of the same chatbots do well enough that you could probably get by without paying, but the rest are not so great. I won’t risk my programming projects with them or recommend that you do until their performance improves.
ChatGPT Plus: The Top-Notch Choice
ChatGPT Plus with GPT-4 and GPT-4o passed all my tests. Its dedicated Mac app is a game-changer, allowing me to have multiple screens open simultaneously. Additionally, its multi-factor authentication and email login make it a secure choice. While it’s not perfect, I recommend it for its solid coding results and overall performance.
Perplexity Pro: A Close Second
Perplexity Pro, on the other hand, is a close second. It uses a different LLM than ChatGPT, which is an advantage if you want to cross-check code. However, its email-only login and lack of a dedicated app are drawbacks.
The Rest: Chatbots to Avoid
I tested 11 LLMs, and six passed most of my tests. The other chatbots, including Meta AI, Meta Code Llama, Claude 3.5 Sonnet, Gemini Advanced, and Microsoft Copilot, each only passed one of my tests. Meta AI and Meta Code Llama failed three of my tests, while Claude 3.5 Sonnet and Gemini Advanced failed two. Microsoft Copilot failed all four.
Conclusion
The results of my tests were surprising, especially given the big investments of Microsoft and Google. However, this area of innovation is improving at warp speed, so we’ll be back with updated tests and results over time. Stay tuned.
FAQs
Q: What’s the best AI chatbot for programming?
A: ChatGPT Plus with GPT-4 and GPT-4o is the top-notch choice, followed by Perplexity Pro.
Q: Which chatbots should I avoid?
A: Meta AI, Meta Code Llama, Claude 3.5 Sonnet, Gemini Advanced, and Microsoft Copilot are chatbots to avoid for programming.
Q: Can I use other AI chatbots for programming?
A: Yes, but be aware that each chatbot has its strengths and weaknesses. Choose the one that best fits your needs.
