Networking and Downstream services in NYC1, NYC2, FRA1, TOR1
Incident Report for DigitalOcean
Resolved
Our Engineering team has resolved the issue with failure of multiple services in NYC1, NYC2, FRA1, TOR1. All the services and Networking should now be operating normally.

If you continue to experience problems, please open a ticket with our support team from within your Cloud Control Panel. Thank you for your patience and we apologize for any inconvenience.
Posted Jun 14, 2022 - 09:05 UTC
Monitoring
Our Engineering team has implemented a fix for the issue impacting multiple services in NYC1, NYC2, FRA1, TOR1. Our team completed the Networking maintenance in these regions and is currently monitoring the situation.

Thank you for your patience, and we will post an update as soon as the issue is fully resolved.
Posted Jun 14, 2022 - 08:30 UTC
Update
Our team is continuing work to complete maintenance in NYC2 and FRA1 . Maintenance in NYC1 and TOR1 is complete. We are not currently seeing any impact from the identified root cause, but there could be certain impact as mentioned in previous updates.

We are working as quickly as possible and will provide updates as necessary. Thank you.
Posted Jun 14, 2022 - 07:04 UTC
Update
Our team is continuing work to complete maintenance in TOR1 and NYC2. We are starting to work on FRA1. NYC1 is complete. We are not currently seeing any impact from the identified root cause, but there could be certain impact as mentioned in the previous update.

We will share an update as soon as we have more information.
Posted Jun 14, 2022 - 05:21 UTC
Update
Our team is continuing work to complete maintenance in TOR1. NYC1 is complete. We are not currently seeing any impact from the identified root cause, but until all regions have undergone emergency maintenance, recurrences of the networking and downstream service issue are possible in regions which have not had maintenance completed.

We are working as quickly as possible and will provide updates as necessary. Thank you.
Posted Jun 14, 2022 - 04:02 UTC
Update
Our team has completed maintenance in NYC1 at this time. We are seeing recurrence of the same issue in NYC2, TOR1, and FRA1, due to the same root cause and users may experience errors as noted in previous updates. We have confirmed that the completed maintenance in NYC1 mitigated the issue recurring there, so NYC1 is not impacted. Our team is now working on maintenance for TOR1.
Posted Jun 14, 2022 - 03:10 UTC
Update
Our Engineering team is now beginning emergency maintenance in NYC1, NYC2, FRA1, and TOR1 to implement a long-term fix for the multiple issues referenced in this incident. As this work progresses, we do not expect user impact to networking connectivity with regards to Droplet and Droplet-based products (such as Managed Kubernetes and Databases).

We do not have a firm ETA on completing maintenance but will post updates as we have more information.
Posted Jun 14, 2022 - 02:12 UTC
Update
As of 00:56 UTC, we saw a recurrence of this incident, impacting users similarly to our previous updates. We are now seeing recovery again for services and the root cause has been identified.

Our Engineering team is working to implement a long-term fix to ensure continued stability. Thank you for your patience and we will post another update as soon as we have further details.
Posted Jun 14, 2022 - 01:33 UTC
Update
Our Engineering team is currently investigating a recurrence of this issue and users may once again see issues with multiple products and services as noted in our last updates.

We apologize and will provide another update as soon as we have more information.
Posted Jun 14, 2022 - 01:07 UTC
Identified
Our Engineering team has identified the root cause of the multiple failures and a fix has been implemented. To add to our earlier update, we've confirmed this incident impacted Droplet networking as well and users may have experienced packet loss/latency and loss of connectivity with Droplets in NYC1, NYC2, FRA1, and TOR1, including Managed Kubernetes.

At this time, users should see error rates subsiding and normal connectivity returning.

Thank you for your patience.
Posted Jun 14, 2022 - 00:33 UTC
Investigating
As of approximately 23:40 UTC, our Engineering team is investigating multiple failures with DigitalOcean services, including our Cloud Control Panel, API, events such as creation of all resources, and WWW endpoints failing to resolve. We are starting to see recovery but are still investigating root cause and users may continue to experience elevated errors.

We will post another update as soon as further information is available.
Posted Jun 14, 2022 - 00:00 UTC
This incident affected: Regions (Global, FRA1, NYC1, NYC2, TOR1) and Services (API, App Platform, Cloud Control Panel, DNS, Droplets, Event Processing, Kubernetes, Managed Databases, Networking, WWW, Functions).