Networking and Event Processing in LON1 and SFO3
Incident
Report for DigitalOcean
Resolved
As of 02:45 UTC, our Engineering team has confirmed the full resolution of the issue that impacted network reachability and event processing in the LON1 and SFO3 regions. All services and resources should now be fully reachable and operational.
If you continue to experience problems, please open a ticket with our support team from within your Cloud Control Panel. Thank you for your patience and we apologize for any inconvenience.
If you continue to experience problems, please open a ticket with our support team from within your Cloud Control Panel. Thank you for your patience and we apologize for any inconvenience.
Monitoring
After further investigation, our Engineering teams have confirmed there was a concurrent event in SFO3 which caused impact to our internal network. This resulted in multiple monitoring systems alerting, as well as impacted users trying to perform events in SFO3, such as Droplet creates, power on/off, etc.
Confirmed impact:
01:35 - 01:50 UTC: Network connectivity loss in LON1 to Droplets and Droplet-based services like Managed Databases, Load Balancers, and Kubernetes Clusters, as well as unavailability to App Platform Apps and Functions. Users were also unable to perform events in LON1 during this time.
01:32 - 01:40 UTC: Users unable to perform events in SFO3 during this time.
All services are now operating normally and we will update the title of this incident to accurately reflect this outage.
We appreciate your patience and will post an update once we are confident the issue is fully resolved.
Confirmed impact:
01:35 - 01:50 UTC: Network connectivity loss in LON1 to Droplets and Droplet-based services like Managed Databases, Load Balancers, and Kubernetes Clusters, as well as unavailability to App Platform Apps and Functions. Users were also unable to perform events in LON1 during this time.
01:32 - 01:40 UTC: Users unable to perform events in SFO3 during this time.
All services are now operating normally and we will update the title of this incident to accurately reflect this outage.
We appreciate your patience and will post an update once we are confident the issue is fully resolved.
Update
Our Network Engineering team has confirmed an unexpected issue occurred in our LON1 region, which led to a loss of network connectivity in LON1 from 01:35 - 01:56 UTC. Network traffic began to recover at 01:56 UTC to all services in LON1.
We are continuing to investigate impact in other regions and will post an update soon.
We are continuing to investigate impact in other regions and will post an update soon.
Investigating
At approximately 01:36 UTC, we received multiple alerts for services in multiple regions. So far, we've confirmed issues in LON1, NYC1, NYC3, SFO2, SFO3, AMS3, and FRA1. We are working to determine impact and are seeing some services self-recovering.
Users may have experienced and may still be experiencing issues with network connectivity to Droplets and Droplet-based services, event failures like Droplet creates and power on/off, Functions unavailability, and other issues with DigitalOcean services.
As soon as we have further information to share, we'll post an update here.
Users may have experienced and may still be experiencing issues with network connectivity to Droplets and Droplet-based services, event failures like Droplet creates and power on/off, Functions unavailability, and other issues with DigitalOcean services.
As soon as we have further information to share, we'll post an update here.
This incident affected: Regions (Global) and Services (Droplets, Event Processing, Managed Databases, Networking, Functions).