LON1 Networking
Incident
Report for DigitalOcean
Resolved
From 20:20 - 23:25, we experienced a partial outage of VPC traffic in our LON1 region. This outage was triggered by an underlying bug uncovered by the maintenance activities in https://status.digitalocean.com/incidents/pqsgdmyzyy0k.
Public networking was unaffected, but Load Balancers, Managed Kubernetes Clusters, Managed Databases, App Platform, Functions, and Droplet-to-Droplet traffic flowing via VPC was impacted.
We sincerely apologize for the disruption triggered by the maintenance activities we performed. We've successfully run this maintenance in multiple regions without any impact, but going forward, we will ensure the bug discovered here is not present before rolling out the upgrades to the remaining regions.
Public networking was unaffected, but Load Balancers, Managed Kubernetes Clusters, Managed Databases, App Platform, Functions, and Droplet-to-Droplet traffic flowing via VPC was impacted.
We sincerely apologize for the disruption triggered by the maintenance activities we performed. We've successfully run this maintenance in multiple regions without any impact, but going forward, we will ensure the bug discovered here is not present before rolling out the upgrades to the remaining regions.
Monitoring
Our Engineering team has confirmed the mitigation was successful and traffic is flowing normally again. Services dependent on VPC traffic were impacted, which include Load Balancers, Managed Databases, Managed Kubernetes Clusters, App Platform, and Functions. We've also confirmed the impact from this outage was triggered by the activities in the maintenance here: https://status.digitalocean.com/incidents/pqsgdmyzyy0k
We're now monitoring and will post an update once we confirm this issue is fully resolved.
We're now monitoring and will post an update once we confirm this issue is fully resolved.
Update
Mitigation efforts are now 100% complete and we are seeing restoration of connectivity. We're confirming with multiple Engineering teams that the mitigation was successful and will post an update soon.
Identified
Our Engineering team has identified the root cause of the issue and is now mitigating it. These efforts are approximately 35% complete. We'll continue to post updates.
Update
Our Engineering team has confirmed impact to Load Balancers, Managed Databases, Managed Kubernetes Clusters, and App Platform. Connectivity directly to Droplets does not appear to be affected.
Investigating
Our Engineering team is currently investigating reports of connectivity issues in our LON1 region. Maintenance is ongoing as per https://status.digitalocean.com/incidents/pqsgdmyzyy0k and we are working to determine if the root cause is related to that maintenance. At this time, users may see timeouts and errors when connecting to services in LON1.
We'll post an update with further information as soon as possible.
We'll post an update with further information as soon as possible.
This incident affected: Regions (LON1) and Services (Networking).