Incident Report: Cause of Last Week’s ChatGPT Outage and Measures to Prevent a Repeat
Services Impacted
- ChatGPT
- Sora video creation
- APIs: agents, real-time speech, batch, and DALL-E
Cause of the Outage
The recent outage was caused by a cloud provider data center failure, which impacted OpenAI databases. Although the databases are mirrored across regions, switching over to a backup database required manual intervention from the cloud provider to redirect operations to a backup data center in another region. This manual intervention was responsible for resolving the outage, but the reason it took so long was due to the scale of the project.
Failover and Infrastructure Changes
A failover is an automated process for switching to a backup system in the event of a system failure. OpenAI has announced that it is working towards creating infrastructure changes to improve responses to future cloud database failures. The company stated that it will add a layer of indirection under its applications and cloud databases to ensure that systems are resilient to extended outages in any region of its cloud providers.
Significant ChatGPT Outage
The ChatGPT outage was caused by a regional cloud provider database failure, but its effects were global, as evidenced by user reports on social media from across Europe and North America.
Google Trends Data
A Google Trends graph shows that this may have been the largest such event, with more people searching for information about it than for any previous outage.
Conclusion
OpenAI has published an incident report detailing the cause of last week’s ChatGPT outage and the measures it is taking to prevent a repeat. The outage was caused by a cloud provider data center failure, which required manual intervention to redirect operations to a backup data center. OpenAI is working to add a layer of indirection under its applications and cloud databases to improve responses to future cloud database failures.
Frequently Asked Questions
Q: What was the cause of the recent ChatGPT outage?
A: The cause was a cloud provider data center failure, which impacted OpenAI databases.
Q: What services were impacted by the outage?
A: ChatGPT, Sora video creation, APIs: agents, real-time speech, batch, and DALL-E.
Q: How long did the outage last?
A: The outage began on December 26th, 2024 at 10:40 AM and was mostly resolved by 3:11 PM, except for ChatGPT which was 100% recovered by 6:20 PM.
Q: What is OpenAI doing to prevent a repeat?
A: OpenAI is working to add a layer of indirection under its applications and cloud databases to ensure that systems are resilient to extended outages in any region of its cloud providers.
