Networking - Multiple Regions
Incident
Report for DigitalOcean
Resolved
Our Engineering team has confirmed full resolution of the issue impacting network connectivity in multiple regions. The impact has been completely subsided and the network connectivity is back to normal for all the impacted services.
If you continue to experience problems, please open a ticket with our support team from your Cloud Control Panel.
Thank you for your patience and we apologize for any inconvenience.
If you continue to experience problems, please open a ticket with our support team from your Cloud Control Panel.
Thank you for your patience and we apologize for any inconvenience.
Monitoring
At this time, all services and regions impacted by this incident are recovered.
We're now monitoring to ensure stability.
If you are still experiencing any issues, please let our Support team know.
We're now monitoring to ensure stability.
If you are still experiencing any issues, please let our Support team know.
Update
Our Engineering team has confirmed that additional network routes are impacted, alongside the impact previously mentioned for our SFO datacenters.
Notably, routes between NYC and SGP were heavily impacted for a period of time, before we were able to shift traffic and improve performance.
Additionally, routes between SYD and NYC were seeing extreme latency. After shifting of traffic, performance has improved but continues to show higher-than-normal latency.
Due to there being a major issue with East to West North American traffic with at least two major upstream providers, DigitalOcean traffic on any routes following East to West North American traffic may be impacted. Our Engineering team is making traffic shifts as quickly as possible, where possible.
We continue to await more information from the upstream providers.
Notably, routes between NYC and SGP were heavily impacted for a period of time, before we were able to shift traffic and improve performance.
Additionally, routes between SYD and NYC were seeing extreme latency. After shifting of traffic, performance has improved but continues to show higher-than-normal latency.
Due to there being a major issue with East to West North American traffic with at least two major upstream providers, DigitalOcean traffic on any routes following East to West North American traffic may be impacted. Our Engineering team is making traffic shifts as quickly as possible, where possible.
We continue to await more information from the upstream providers.
Identified
Our Engineering team has confirmed upstream issues with two providers, leading to poor network performance in our SFO datacenters. Our team is engaging with those providers as well as exploring any possible routing configurations to improve performance.
At this time, users will see packet loss, latency, and possible connection errors for Droplets, Droplet-based services like Managed Kubernetes, and App Platform Apps.
We'll provide another update when we have further information or if our team is able to improve the situation via routing configurations.
At this time, users will see packet loss, latency, and possible connection errors for Droplets, Droplet-based services like Managed Kubernetes, and App Platform Apps.
We'll provide another update when we have further information or if our team is able to improve the situation via routing configurations.
Investigating
Our Engineering team is investigating an issue with networking in our SFO datacenters. At this time, there appears to be an upstream issue with a provider impacting networking performance on routes to and from SFO for DigitalOcean services, including Droplets and Droplet-based services like Managed Kubernetes and App Platform Apps.
We will post a further update as soon as we have more information.
We will post a further update as soon as we have more information.
This incident affected: Networking (SFO1, SFO2, SFO3).