DigitalOcean Services Status
Cloud Control Panel and Multiple services
Incident
Report for DigitalOcean
Resolved
Postmortem
The Incident
Beginning at 04:58 UTC, January 29th, the DigitalOcean NYC3 data center region experienced an Internet connection outage. This prevented Droplets and Droplet-based services, App Platform, Load Balancers, and Spaces in that data center region from accessing the Internet during the outages. Additionally, users were unable to access the Cloud Control Panel and the DigitalOcean API during the outage and event processing failures were evident in all regions. Customers may have also seen some gaps in metrics and monitoring during the incident. This incident was triggered by a software bug in the border (edge) routers of the NYC3 region.
Our engineers were able to detect and begin an incident response for this issue within minutes of the first signs of impact. We quickly determined the root cause and began remediation efforts.
The root cause was discovered to be a bug in the operating system software of critical edge routing hardware, causing these routing systems to crash. The bug was triggered by interpretation of automatically propagated routes. Routers propagate routing information between redundant nodes as efficiently as possible to avoid errors. This behavior of standard routing protocols, coupled with the bug, caused clustered routers to crash nearly concurrently.
Without the routing service, the edge routers quickly degraded and stopped serving traffic. This cut off the DigitalOcean network behind the edge routers in question from the rest of the Internet during these crashes. Besides impact to customers, the lack of external network access complicated efforts to troubleshoot and fix problems and spread secondary impacts to nearly all DigitalOcean services. The core problem occurred in three waves. The initial outage consisted of 16 minutes of Internet connectivity loss, followed by two 15-minute periods of network congestion, and finally 50 minutes of congestion, as traffic was redirected to perform software upgrades.
To fix this issue, DigitalOcean engineers deployed emergency software upgrades to the impacted systems. These upgrades were the most time-consuming part of the response and, when completed, they resolved the bug in the impacted systems for the long term. Internet access at the expected performance levels was restored for Cloud, API, Droplets, Droplet-based services like Managed Kubernetes and Load Balancers, App Platform, Functions, Monitoring, and Spaces by the end of the emergency software upgrades. Some asynchronous customer-triggered events like power offs and resizes were stuck until DigitalOcean engineers completed manual cleanup of affected events after the incident.
Timeline of Events
UTC Timeline
04:58 - Both edge routers crash in NYC3, causing a complete outage for NYC3 for customer and internal services, as well as global impact to some services.05:05 - Internal incident response begins.
05:12 - Network Engineering team gains access to the edge routers in NYC3 and begins investigation.
05:13 - Network Engineering team sees the routing process crashed.
05:14 - Edge1 recovered and took traffic. Complete outage ends, congestion exists for customer and internal services.
05:21 - Multiple Engineering teams across DigitalOcean still reporting down alerts. Network Engineering team sees that memory is exhausted on Edge2.
05:27 - Multiple Engineering teams confirm alerts are clearing and that cloud.digitalocean.com is accessible again.
05:40 - Edge2 recovered after manual intervention caused by hardware failure. Congestion ends for customer and internal services.
06:06 - Public status page posted with information about the outage .
06:24 - Edge1 crashes for a second time, causing customers to be unable to access or experience latency with Droplets and Droplet-based services via public/private networks.
06:36 - Network Engineering team makes the decision to emergency upgrade software on Edge1 to a newer version to help mitigate the crashes.
06:41 - Edge1 recovers and is able to take traffic again.
06:52 - Edge2 crashes for a second time, causing customers to be unable to access or experience latency with Droplets and Droplet-based services via public/private networks.
06:59 - Network Engineering team confirms Edge2 needs to recover before starting the software upgrade for Edge1.
07:07 - Edge2 recovered and took traffic.
07:08 - Network Engineering team begins to drain traffic away from Edge1 in preparation for upgrade. Congestion to Droplets and Droplet-based services in NYC3 begins.
07:23 - Edge1 is rebooted to apply the software upgrade.
07:44 - Network Engineering team brings Edge1 online and it begins to take traffic.
07:56 - Edge1 upgrade complete, traffic restored. Congestion ends.
08:09 - Network Engineering team begins to drain traffic away from Edge2 in preparation for upgrade. Congestion to Droplets and Droplet-based services in NYC3 begins.
08:18 - Edge2 is rebooted to apply the software upgrade.
09:05 - Network Engineering team confirms both edge routers are upgraded and traffic is balanced between them. All customer impact confirmed over.
Future Measures
The current and ongoing network infrastructure upgrade scheduled to be completed in the first half of 2023 aims to improve overall network performance and stability. As of today, SYD1, SGP1, and BLR1 are the completed regions for this overarching upgrade and are running on the new network infrastructure.
As a result of these upgrades, users can expect a smoother and more efficient experience when using the network by increasing bandwidth and enhancing availability. To achieve this, the upgrade utilizes updated hardware and software, fixing any bugs that existed in the previous network setup.
In addition to the immediate benefits, there is a long-term plan to address potential vulnerabilities. This includes a proof of concept, scheduled for 2023, which focuses on implementing vendor diversity. This strategy will reduce the reliance on a single vendor, which can be a source of single point of failure. Reliance on a single vendor was a factor in this incident and the other routing crashes we experienced, as the software on the edge routers is from a single vendor. By diversifying vendors, the likelihood of a similar failure in the future is significantly reduced, ensuring the stability and reliability of the network for years to come.
In Conclusion
DigitalOcean recognizes and understands how disruptive networking incidents are to our customers and that repeated networking outages erode customer trust, as well as lead to business impact for our users. We let our users down and are working to earn customer trust back by taking steps to improve our networking stack as a whole, as well as reduce single points of failure in our networking vendor and in DigitalOcean internal services.
We’re committed to being as transparent as possible as networking upgrades happen and will communicate with users to ensure they can plan appropriately. We are also reviewing all deployment processes at DigitalOcean to ensure sufficient testing is done before any production change. DigitalOcean is confident in our commitment to excellence and that the future measures we’ve identified will drive better and better outcomes for our users.
Mar 07, 2023 - 22:11 UTC
-
Resolved
As of 10:00 UTC, our Engineering team has resolved the issue with the Cloud Control panel, services in the NYC3 region, Functions, Container registry, and Event processing in all the regions should now be operating normally. If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.
Jan 29, 2023 - 10:08 UTC
-
Monitoring
Around 04:58UTC, our Engineering team observed issues with the Cloud Control panel, services in the NYC3 region, Functions, Container registry, and Event processing in all the regions. Our Engineering team has mitigated these issues as of 05:12 UTC and currently monitoring the same. The users should no longer experience issues with the Cloud Control panel, Functions, Container Registry, and Event processing, and all the services in the NYC3 region will be functional. We are monitoring the situation and will post an update once we confirm the issue is fully resolved.
Jan 29, 2023 - 06:06 UTC
-
This incident affected: Regions (Global) and Services (Cloud Control Panel, Container Registry, Droplets, Event Processing, Functions).
The Incident
Beginning at 04:58 UTC, January 29th, the DigitalOcean NYC3 data center region experienced an Internet connection outage. This prevented Droplets and Droplet-based services, App Platform, Load Balancers, and Spaces in that data center region from accessing the Internet during the outages. Additionally, users were unable to access the Cloud Control Panel and the DigitalOcean API during the outage and event processing failures were evident in all regions. Customers may have also seen some gaps in metrics and monitoring during the incident. This incident was triggered by a software bug in the border (edge) routers of the NYC3 region.
Our engineers were able to detect and begin an incident response for this issue within minutes of the first signs of impact. We quickly determined the root cause and began remediation efforts.
The root cause was discovered to be a bug in the operating system software of critical edge routing hardware, causing these routing systems to crash. The bug was triggered by interpretation of automatically propagated routes. Routers propagate routing information between redundant nodes as efficiently as possible to avoid errors. This behavior of standard routing protocols, coupled with the bug, caused clustered routers to crash nearly concurrently.
Without the routing service, the edge routers quickly degraded and stopped serving traffic. This cut off the DigitalOcean network behind the edge routers in question from the rest of the Internet during these crashes. Besides impact to customers, the lack of external network access complicated efforts to troubleshoot and fix problems and spread secondary impacts to nearly all DigitalOcean services. The core problem occurred in three waves. The initial outage consisted of 16 minutes of Internet connectivity loss, followed by two 15-minute periods of network congestion, and finally 50 minutes of congestion, as traffic was redirected to perform software upgrades.
To fix this issue, DigitalOcean engineers deployed emergency software upgrades to the impacted systems. These upgrades were the most time-consuming part of the response and, when completed, they resolved the bug in the impacted systems for the long term. Internet access at the expected performance levels was restored for Cloud, API, Droplets, Droplet-based services like Managed Kubernetes and Load Balancers, App Platform, Functions, Monitoring, and Spaces by the end of the emergency software upgrades. Some asynchronous customer-triggered events like power offs and resizes were stuck until DigitalOcean engineers completed manual cleanup of affected events after the incident.
Timeline of Events
UTC Timeline
04:58 - Both edge routers crash in NYC3, causing a complete outage for NYC3 for customer and internal services, as well as global impact to some services.05:05 - Internal incident response begins.
05:12 - Network Engineering team gains access to the edge routers in NYC3 and begins investigation.
05:13 - Network Engineering team sees the routing process crashed.
05:14 - Edge1 recovered and took traffic. Complete outage ends, congestion exists for customer and internal services.
05:21 - Multiple Engineering teams across DigitalOcean still reporting down alerts. Network Engineering team sees that memory is exhausted on Edge2.
05:27 - Multiple Engineering teams confirm alerts are clearing and that cloud.digitalocean.com is accessible again.
05:40 - Edge2 recovered after manual intervention caused by hardware failure. Congestion ends for customer and internal services.
06:06 - Public status page posted with information about the outage .
06:24 - Edge1 crashes for a second time, causing customers to be unable to access or experience latency with Droplets and Droplet-based services via public/private networks.
06:36 - Network Engineering team makes the decision to emergency upgrade software on Edge1 to a newer version to help mitigate the crashes.
06:41 - Edge1 recovers and is able to take traffic again.
06:52 - Edge2 crashes for a second time, causing customers to be unable to access or experience latency with Droplets and Droplet-based services via public/private networks.
06:59 - Network Engineering team confirms Edge2 needs to recover before starting the software upgrade for Edge1.
07:07 - Edge2 recovered and took traffic.
07:08 - Network Engineering team begins to drain traffic away from Edge1 in preparation for upgrade. Congestion to Droplets and Droplet-based services in NYC3 begins.
07:23 - Edge1 is rebooted to apply the software upgrade.
07:44 - Network Engineering team brings Edge1 online and it begins to take traffic.
07:56 - Edge1 upgrade complete, traffic restored. Congestion ends.
08:09 - Network Engineering team begins to drain traffic away from Edge2 in preparation for upgrade. Congestion to Droplets and Droplet-based services in NYC3 begins.
08:18 - Edge2 is rebooted to apply the software upgrade.
09:05 - Network Engineering team confirms both edge routers are upgraded and traffic is balanced between them. All customer impact confirmed over.
Future Measures
The current and ongoing network infrastructure upgrade scheduled to be completed in the first half of 2023 aims to improve overall network performance and stability. As of today, SYD1, SGP1, and BLR1 are the completed regions for this overarching upgrade and are running on the new network infrastructure.
As a result of these upgrades, users can expect a smoother and more efficient experience when using the network by increasing bandwidth and enhancing availability. To achieve this, the upgrade utilizes updated hardware and software, fixing any bugs that existed in the previous network setup.
In addition to the immediate benefits, there is a long-term plan to address potential vulnerabilities. This includes a proof of concept, scheduled for 2023, which focuses on implementing vendor diversity. This strategy will reduce the reliance on a single vendor, which can be a source of single point of failure. Reliance on a single vendor was a factor in this incident and the other routing crashes we experienced, as the software on the edge routers is from a single vendor. By diversifying vendors, the likelihood of a similar failure in the future is significantly reduced, ensuring the stability and reliability of the network for years to come.
In Conclusion
DigitalOcean recognizes and understands how disruptive networking incidents are to our customers and that repeated networking outages erode customer trust, as well as lead to business impact for our users. We let our users down and are working to earn customer trust back by taking steps to improve our networking stack as a whole, as well as reduce single points of failure in our networking vendor and in DigitalOcean internal services.
We’re committed to being as transparent as possible as networking upgrades happen and will communicate with users to ensure they can plan appropriately. We are also reviewing all deployment processes at DigitalOcean to ensure sufficient testing is done before any production change. DigitalOcean is confident in our commitment to excellence and that the future measures we’ve identified will drive better and better outcomes for our users.
Mar 07, 2023 - 22:11 UTC
-
Resolved
As of 10:00 UTC, our Engineering team has resolved the issue with the Cloud Control panel, services in the NYC3 region, Functions, Container registry, and Event processing in all the regions should now be operating normally. If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.
Jan 29, 2023 - 10:08 UTC
-
Monitoring
Around 04:58UTC, our Engineering team observed issues with the Cloud Control panel, services in the NYC3 region, Functions, Container registry, and Event processing in all the regions. Our Engineering team has mitigated these issues as of 05:12 UTC and currently monitoring the same. The users should no longer experience issues with the Cloud Control panel, Functions, Container Registry, and Event processing, and all the services in the NYC3 region will be functional. We are monitoring the situation and will post an update once we confirm the issue is fully resolved.
Jan 29, 2023 - 06:06 UTC
-
This incident affected: Regions (Global) and Services (Cloud Control Panel, Container Registry, Droplets, Event Processing, Functions).