Network Connectivity in BLR1
Incident
Report for DigitalOcean
Resolved
Impact: minor;
From 09:05 - 11:57 UTC, we experienced a networking issue in our BLR1 region. We initially believed the issue was due to the ongoing performance issues with the APAC fiber faults, but further investigation from our Network Engineering team revealed this was a separate issue.
The root cause was tracked to an unexpected hardware issue that resulted in network unavailability for a large subset of services in the BLR1 region.
User impact included being unable to reach Droplets and Droplet-based services like Kubernetes clusters, Managed Databases, and Load Balancers.
Users with Block Storage Volumes and Snapshots in BLR1 also experienced intermittent failures with CRUD (create, read, update, delete) operations. This could have resulted in API errors or errors shown in the Cloud Control Panel when working with those services.
Users utilizing App Platform also saw image pull failures and deployment latency/failures during the course of this incident.
Our Network Engineering team has an open case with our upstream provider in BLR1 to determine the cause behind this hardware failure. We will also be working internally to determine quicker remediation tactics for this type of failure.
If you continue to experience problems, please open a ticket with our support team from within your Cloud Control Panel. Thank you for your patience and we apologize for any inconvenience.
Affected Components: Services - API Services - App Platform Services - Block Storage Services - Droplets Services - Event Processing Services - Kubernetes Services - Load Balancers Services - Managed Databases Services - Networking Services - Spaces Regions - BLR1
Status: resolved
2023-03-29 12:47 UTC
-
Our Engineering team has successfully mitigated the issue impacting multiple products in BLR1. At this time, network connectivity to Droplet-based services including Droplets, Managed Kubernetes, and Managed Database in BLR1, should be operating normally.
We'll continue to monitor the situation to confirm this incident is fully resolved and will post an update soon.
Affected Components: Services - API Services - App Platform Services - Block Storage Services - Droplets Services - Event Processing Services - Kubernetes Services - Load Balancers Services - Managed Databases Services - Networking Services - Spaces Regions - BLR1
Status: monitoring
2023-03-29 11:41 UTC
-
As of 09:05UTC, our Engineering team is currently investigating an issue impacting multiple products in the BLR1 region. During this time, users may have experienced packet loss/latency, timeouts, and related issues with Droplet-based services in these regions, including Droplets, Managed Kubernetes, and Managed Database. We will share an update once we have further information.
Affected Components: Services - API Services - App Platform Services - Block Storage Services - Droplets Services - Event Processing Services - Kubernetes Services - Load Balancers Services - Managed Databases Services - Networking Services - Spaces Regions - BLR1
Status: investigating
2023-03-29 11:20 UTC
-
Incident Created at: 2023-03-29 11:20 UTC Resolved at: 2023-03-29 12:47 UTC
From 09:05 - 11:57 UTC, we experienced a networking issue in our BLR1 region. We initially believed the issue was due to the ongoing performance issues with the APAC fiber faults, but further investigation from our Network Engineering team revealed this was a separate issue.
The root cause was tracked to an unexpected hardware issue that resulted in network unavailability for a large subset of services in the BLR1 region.
User impact included being unable to reach Droplets and Droplet-based services like Kubernetes clusters, Managed Databases, and Load Balancers.
Users with Block Storage Volumes and Snapshots in BLR1 also experienced intermittent failures with CRUD (create, read, update, delete) operations. This could have resulted in API errors or errors shown in the Cloud Control Panel when working with those services.
Users utilizing App Platform also saw image pull failures and deployment latency/failures during the course of this incident.
Our Network Engineering team has an open case with our upstream provider in BLR1 to determine the cause behind this hardware failure. We will also be working internally to determine quicker remediation tactics for this type of failure.
If you continue to experience problems, please open a ticket with our support team from within your Cloud Control Panel. Thank you for your patience and we apologize for any inconvenience.
Affected Components: Services - API Services - App Platform Services - Block Storage Services - Droplets Services - Event Processing Services - Kubernetes Services - Load Balancers Services - Managed Databases Services - Networking Services - Spaces Regions - BLR1
Status: resolved
2023-03-29 12:47 UTC
-
Our Engineering team has successfully mitigated the issue impacting multiple products in BLR1. At this time, network connectivity to Droplet-based services including Droplets, Managed Kubernetes, and Managed Database in BLR1, should be operating normally.
We'll continue to monitor the situation to confirm this incident is fully resolved and will post an update soon.
Affected Components: Services - API Services - App Platform Services - Block Storage Services - Droplets Services - Event Processing Services - Kubernetes Services - Load Balancers Services - Managed Databases Services - Networking Services - Spaces Regions - BLR1
Status: monitoring
2023-03-29 11:41 UTC
-
As of 09:05UTC, our Engineering team is currently investigating an issue impacting multiple products in the BLR1 region. During this time, users may have experienced packet loss/latency, timeouts, and related issues with Droplet-based services in these regions, including Droplets, Managed Kubernetes, and Managed Database. We will share an update once we have further information.
Affected Components: Services - API Services - App Platform Services - Block Storage Services - Droplets Services - Event Processing Services - Kubernetes Services - Load Balancers Services - Managed Databases Services - Networking Services - Spaces Regions - BLR1
Status: investigating
2023-03-29 11:20 UTC
-
Incident Created at: 2023-03-29 11:20 UTC Resolved at: 2023-03-29 12:47 UTC