Volumes Availability and Downstream Services in FRA1
Incident
Report for DigitalOcean
Resolved
Our Engineering team has confirmed full resolution of this incident.
From 17:06 - 17:39 UTC, we experienced an availability outage on an internal storage cluster, due to an issue with a networking component. Users may have seen degraded performance with Volumes, issues connecting to Managed Kubernetes clusters, issues creating/deleting Mongo clusters, and delayed deploys/updates to existing Apps in our FRA1 region.
If you continue to experience problems, please open a ticket with our support team from within your Cloud Control Panel. Thank you for your patience throughout this incident.
From 17:06 - 17:39 UTC, we experienced an availability outage on an internal storage cluster, due to an issue with a networking component. Users may have seen degraded performance with Volumes, issues connecting to Managed Kubernetes clusters, issues creating/deleting Mongo clusters, and delayed deploys/updates to existing Apps in our FRA1 region.
If you continue to experience problems, please open a ticket with our support team from within your Cloud Control Panel. Thank you for your patience throughout this incident.
Monitoring
Our Engineering team has confirmed the issue with the networking component of the internal storage cluster was the root cause and the remediation steps taken were successful.
Users should no longer be seeing issues with operations on Volumes, connecting to Managed Kubernetes clusters, operations with Mongo Managed Databases, nor deploys/updates to Apps in Frankfurt.
We will monitor this issue for a short period to ensure it's fully resolved and will post a final update at that time.
Users should no longer be seeing issues with operations on Volumes, connecting to Managed Kubernetes clusters, operations with Mongo Managed Databases, nor deploys/updates to Apps in Frankfurt.
We will monitor this issue for a short period to ensure it's fully resolved and will post a final update at that time.
Identified
The team has identified an issue with a networking component of the internal storage cluster and has taken steps to remediate the issue. At this time, we're seeing Volumes operations returning to pre-incident thresholds.
Our App Platform team identified that users with Apps in Frankfurt may have also seen delays in deploys/updates to existing Apps.
We're watching metrics closely to confirm operations return to normal and are confirming the network issue was the root cause of the issue.
Our App Platform team identified that users with Apps in Frankfurt may have also seen delays in deploys/updates to existing Apps.
We're watching metrics closely to confirm operations return to normal and are confirming the network issue was the root cause of the issue.
Investigating
Our Engineering team is investigating an issue with a drop in availability for an internal storage cluster in our FRA1 region. At this time, users with Volumes in FRA may experience slower than expected operations, as well as I/O stalls. Users with Managed Kubernetes clusters may see issues connecting to clusters. This issue is also impacting Mongo Managed Database operations, including creates and deletes, but does not impact already running Mongo clusters.
We will post an update as soon as possible.
We will post an update as soon as possible.
This incident affected: Kubernetes (FRA1), Managed Databases (FRA1), App Platform (Frankfurt), and Volumes (FRA1).