Elevated Errors in ChatGPT: A Misconfigured Experiment Caused Service Degradation
Incident Summary
On February 19, 2025, ChatGPT experienced a service degradation, leading to a significant increase in failed conversation attempts. The issue was caused by a misconfigured internal experiment that unintentionally triggered a surge in traffic, overwhelming our inference infrastructure. This increase in load led to saturation of compute resources, causing failures in generating responses.
The Root Cause
The root cause of the issue was a misconfigured internal experiment that was not properly approved, resulting in an unexpected surge in traffic. This increased load led to a saturation of compute resources, causing failures in generating responses.
Response and Resolution
After identifying the root cause, we took immediate action by temporarily shedding load from free-tier users to stabilize the system. As capacity was restored, paid users gradually recovered, and the full service was restored by 11:19 AM PT.
OpenAI Continues to Work on Solutions
The incident response highlights the importance of building stronger safeguards around experiment changes and configurations. To prevent similar outages from happening in the future, we are working on the following solutions:
Stronger Safeguards
- Building better protections around experiment changes and configurations by moving from a uniform approval process to a risk-based model to ensure safer rollouts of experiments.
Faster Root Cause Identification
- Automating notifications for relevant changes and experiments to more quickly identify root causes of increased failures.
Conclusion
We apologize for the inconvenience caused by this incident and are committed to providing a reliable and high-quality experience for our users. We are continually working to improve our systems and processes to prevent similar issues from occurring in the future.
Frequently Asked Questions
Q: What caused the service degradation in ChatGPT?
A: The service degradation was caused by a misconfigured internal experiment that unintentionally triggered a surge in traffic, overwhelming our inference infrastructure.
Q: How did OpenAI respond to the incident?
A: We took immediate action by temporarily shedding load from free-tier users to stabilize the system and restored the full service by 11:19 AM PT.
Q: What measures is OpenAI taking to prevent similar outages in the future?
A: We are building stronger safeguards around experiment changes and configurations and automating notifications for relevant changes and experiments to more quickly identify root causes of increased failures.
