Incident Summary
On January 23, 2024 at 12:30 PM UTC, we suffered an outage with an internal infrastructure cluster that powered a subset of DOKS clusters in the fra1 region. Specifically, an incorrect configuration was inadvertently applied to the infrastructure cluster during a maintenance operation, which then triggered a cascading event leading to the automatic deletion of the involved DOKS clusters. As a consequence, the control planes became inaccessible, and control plane data loss occurred.
Incident Details
- Root Cause: An incorrect configuration change to the production infrastructure during the maintenance effort.
- Impact: The configuration change triggered an automatic deletion process on the underlying compute resources hosting the control planes of the DOKS clusters. As a result, the impacted DOKS clusters lost their control planes.
- Response: All DOKS etcd clusters are backed up twice a day with a randomized schedule for each cluster. The recovery process involved restoring the etcd cluster from the latest available backup for the cluster. In the worst case, this caused Kubernetes control plane data to be outdated for up to 12h. Once etcd was recovered, the stateless control plane components of the cluster were also restored, thus completing the recovery of the control plane. Existing data plane workloads in the cluster continued to operate for the clusters until the recovery was complete.
Timeline of Events
Jan 23, 2024
12:21 - The incorrect configuration change is applied to the FRA1 internal cluster and triggers the automatic control plane deletion.
12:30 - The automatic deletion process removed all the control planes of the DOKS clusters from the specific infrastructure cluster.
12:36 - The incorrect configuration was identified as the root cause, preparations for recovery begin.
12:48 - Internal incident response kicks off to identify impacted clusters and track the recovery process.13:00 - Unexpected problems during the recovery process are encountered, delaying the full recovery and requiring additional patching of the recovery mechanism.
15:00 - The recovery problems are fully addressed.
20:31 - All impacted HA DOKS clusters are recovered.
21:21 - All impacted DOKS clusters are recovered.
Remediation Actions
As of this week, a fix to prevent the automatic deletion of infrastructure cluster resources under any circumstances in the production environment has been rolled out.
At this time, functionality to impacted clusters has been restored but customers may need to reconfigure some Kubernetes resources. Customer Support is contacting impacted customers directly with further instructions.
If you have any questions or concerns regarding this incident, please open a ticket with our support team.
We will post another update as soon as we have new developments.
Thank you for your patience and we apologize for any inconvenience.
Our Engineering team is engaged in remediating these clusters to restore accessibility. As soon as we are able to provide an estimated time to restore, we will provide an update.
We apologize for the inconvenience and will share an update once we have more information.