Kubernetes Clusters in AMS3
Incident
Report for DigitalOcean
Resolved
As of 13:53 UTC, our Engineering team has confirmed the full resolution of the issue that impacted Kubernetes Clusters in our AMS3 region. All resources should now be fully reachable.
If you continue to experience problems, please open a ticket with our Support team from within your Cloud Control Panel. Thank you for your patience and we apologize for any inconvenience.
If you continue to experience problems, please open a ticket with our Support team from within your Cloud Control Panel. Thank you for your patience and we apologize for any inconvenience.
Monitoring
Our Engineering team has successfully mitigated the issue and the remaining clusters have been remediated.
At this time, all Kubernetes Clusters which were affected in AMS3 are operational. Users should be able to access their Kubernetes Clusters without any problem.
We are monitoring the situation and will post an update once we confirm the issue is fully resolved.
At this time, all Kubernetes Clusters which were affected in AMS3 are operational. Users should be able to access their Kubernetes Clusters without any problem.
We are monitoring the situation and will post an update once we confirm the issue is fully resolved.
Update
As of 10:00 UTC January 18, our Engineering team has made progress toward internal recovery, although it might take some time for the Clusters to be fully operational and accessible. Some Kubernetes Clusters in AMS3 remain impacted and may be inaccessible. Our Engineering team is continuing to work to mitigate the issue as soon as possible.
We will post another update as soon as we have new developments.
We will post another update as soon as we have new developments.
Update
As of 09:00 UTC January 18, our Engineering team has made progress toward internal recovery, although it might take some time for the Clusters to be fully operational and accessible.
At this point, 7% out of the total 21% of Kubernetes Clusters have been remediated and approximately 14% of Kubernetes Clusters in AMS3 remain impacted and may be inaccessible. Our Engineering team is continuing to work to mitigate the issue as soon as possible.
We will post another update by 10:00 UTC.
At this point, 7% out of the total 21% of Kubernetes Clusters have been remediated and approximately 14% of Kubernetes Clusters in AMS3 remain impacted and may be inaccessible. Our Engineering team is continuing to work to mitigate the issue as soon as possible.
We will post another update by 10:00 UTC.
Update
Our Engineering team has continued work to mitigate this issue and additional clusters have been remediated.
The total percentage of Kubernetes Clusters impacted by this incident was determined to be approximately 21%.
From 11:41 - 18:00 UTC January 17, approximately 7% out of the total 21% of Kubernetes Clusters in AMS3 experienced issues and were remediated.
At this point, approximately 14% of Kubernetes Clusters in AMS3 remain impacted and may be inaccessible.
We will post another update by 09:00 UTC.
The total percentage of Kubernetes Clusters impacted by this incident was determined to be approximately 21%.
From 11:41 - 18:00 UTC January 17, approximately 7% out of the total 21% of Kubernetes Clusters in AMS3 experienced issues and were remediated.
At this point, approximately 14% of Kubernetes Clusters in AMS3 remain impacted and may be inaccessible.
We will post another update by 09:00 UTC.
Update
Our Engineering team is continuing to troubleshoot this issue with the main priority of getting Kubernetes clusters available for users. A fix has now been pushed for a subset of clusters. Unfortunately, there are still some clusters that remain impacted at this time and users could still be facing issues with loading the dashboard as well as cluster accessibility.
We will provide further updates as investigation and mitigation efforts continue. Thank you for your patience.
We will provide further updates as investigation and mitigation efforts continue. Thank you for your patience.
Investigating
Our Engineering team is investigating an issue with Kubernetes clusters in the AMS3 region. During this time, a subset of users may experience errors while communicating with their clusters in the AMS3 region. Specifically, as of 00:18 UTC, users may be unable to load the Kubernetes Dashboard in the Cloud Control Panel, see errors when accessing via kubectl, and/or see unauthorized errors when attempting to establish connectivity.
We apologize for the inconvenience and will share an update once we have more information.
We apologize for the inconvenience and will share an update once we have more information.
This incident affected: Regions (AMS3) and Services (Kubernetes).