Global Networking
Incident
Report for DigitalOcean
Resolved
Our Engineering team has confirmed full resolution of this incident.
From approximately 20:15 - 21:45 UTC, DigitalOcean experienced a global networking issue that impacted multiple services and products. Users saw increased error rates and latency for event processing, accessing our Cloud Control Panel/API, applying Cloud Firewall policies, accessing www.digitalocean.com and our Community site, and DNS resolution. Additionally, users saw timeouts/increased latency for networking requests to Droplets and Droplet-based services, as well as connections to existing App Platform Apps.
We sincerely apologize for the disruption. If you continue to experience issues or have questions, please reach out to our Support team by opening a ticket from within your account. Thank you.
From approximately 20:15 - 21:45 UTC, DigitalOcean experienced a global networking issue that impacted multiple services and products. Users saw increased error rates and latency for event processing, accessing our Cloud Control Panel/API, applying Cloud Firewall policies, accessing www.digitalocean.com and our Community site, and DNS resolution. Additionally, users saw timeouts/increased latency for networking requests to Droplets and Droplet-based services, as well as connections to existing App Platform Apps.
We sincerely apologize for the disruption. If you continue to experience issues or have questions, please reach out to our Support team by opening a ticket from within your account. Thank you.
Monitoring
Our Engineering team has observed recovery of all services. Some events for Droplets and Droplet-based services are being processed, so users may still see lingering events until all those events have been processed.
We are now monitoring the situation and will post a final update once we confirm this incident is resolved.
We are now monitoring the situation and will post a final update once we confirm this incident is resolved.
Update
Our Engineering team has mitigated the issue with TOR1 event processing and events are now flowing normally once again. Users should be able to create, destroy, perform power actions, resize, etc for Droplets and Droplet-based services.
Update
Our Engineering team has identified a lingering issue with event processing in our TOR1 region. Users may see latency or failures for events for Droplets and Droplet-based services in TOR1, including create, power cycle, power off/on, resizes, etc.
We're working to return event processing to normal rates and will share an update soon.
We're working to return event processing to normal rates and will share an update soon.
Update
Our Engineering team has taken action to improve the situation and we are seeing recovery across all services. Event processing, including actions like power cycle, power off/on, resize, etc, was stalled, but has recovered and users should start seeing those events process normally. Droplet and Droplet-based services networking should also be returning to normal latency/error rates.
We are ensuring all services have recovered and will post an update shortly.
We are ensuring all services have recovered and will post an update shortly.
Update
Our Engineering team continues to work on mitigation efforts and is seeing a return of stability in some regions, including NYC and SFO.
Services in SYD1 and TOR1 continue to be heavily impacted, as well as increased error rates for our Cloud Control Panel/API, www.digitalocean.com, DigitalOcean Community site, and latency/errors in applying Cloud Firewall policies for Droplets and Droplet-based services.
Services in SYD1 and TOR1 continue to be heavily impacted, as well as increased error rates for our Cloud Control Panel/API, www.digitalocean.com, DigitalOcean Community site, and latency/errors in applying Cloud Firewall policies for Droplets and Droplet-based services.
Identified
Our Engineering team has identified the root cause of a global issue with networking and are working to mitigate the situation. During this time, users may see a variety of timeouts or errors, including, but not limited to, accessing our Cloud Control Panel/API, Droplet and Droplet-based services network latency/timeouts, App Platform deployment errors, and DNS resolution timeouts.
We apologize for the disruption and will continue to provide updates as they are available.
We apologize for the disruption and will continue to provide updates as they are available.
Investigating
Our Engineering team is currently investigating multiple reports of issues, including increased error rates with our Cloud Control Panel/API, Droplet networking, App Platform app deployments/apps loading, and possible other impacts.
We are working to clarify impact and root cause and will post an update as soon as possible.
We are working to clarify impact and root cause and will post an update as soon as possible.
This incident affected: API, Cloud Control Panel, Networking (Global), and Event Processing (Global).