Subsea Fiber Faults in the APAC region
Incident
Report for DigitalOcean
Resolved
Our Engineering team has continued to actively monitor the situation resulting from multiple subsea fiber faults in the APAC region. Over the last few days, our team has continued to make routing changes where possible.
Crews have been able to complete some cable repairs and we expect to see our network routing stabilize once the repair of a few more cables are completed in the coming weeks. Until then, we expect to see intermittent periods of packet loss and latency on network routes between Singapore and New York City, as well as Singapore and Toronto. These are normally short-lived and happen during Singapore business hours when traffic is heavy.
Given the relative stability of routes, we will now close out this incident and provide any needed updates separately.
If users experience a disruption in service or have any questions, we invite them to submit a support ticket from within their account.
Thank you for your patience and understanding throughout this incident.
Crews have been able to complete some cable repairs and we expect to see our network routing stabilize once the repair of a few more cables are completed in the coming weeks. Until then, we expect to see intermittent periods of packet loss and latency on network routes between Singapore and New York City, as well as Singapore and Toronto. These are normally short-lived and happen during Singapore business hours when traffic is heavy.
Given the relative stability of routes, we will now close out this incident and provide any needed updates separately.
If users experience a disruption in service or have any questions, we invite them to submit a support ticket from within their account.
Thank you for your patience and understanding throughout this incident.
Update
Our Engineering team has detected large amounts of loss and latency on network routes between NYC/TOR and Singapore. Users may experience higher-than-normal latency or amounts of packet loss for traffic traversing those routes. The team is reviewing any possible traffic shifts to alleviate the situation.
If you have any questions or concerns, please open a support ticket from within your account.
If you have any questions or concerns, please open a support ticket from within your account.
Update
Our Engineering team detected large amounts of loss and latency on network routes between Singapore and Frankfurt, from 13:10 - 13:20 UTC, today. The issue self-recovered, likely due to upstream providers shuffling traffic.
Additionally, our team is seeing high levels of loss and latency on network routes between Bangalore and Sydney. Users may see packet loss or notice less performant services at this time. The Network Engineering team is currently exploring options to route around the issue or any other steps that can be taken to mitigate impact to users.
Additionally, our team is seeing high levels of loss and latency on network routes between Bangalore and Sydney. Users may see packet loss or notice less performant services at this time. The Network Engineering team is currently exploring options to route around the issue or any other steps that can be taken to mitigate impact to users.
Monitoring
Our Engineering team has not observed higher-than-normal periods of packet loss and latency over the last 12 hours and user reports of degraded performance have returned to pre-incident levels.
Until the cable cuts in the APAC region are completely resolved, we expect there to be intermittent periods of degraded performance, especially during APAC business hours.
We'll continue to monitor the situation through the next few days and provide any needed updates. If you experience any issues, feel free to open a support ticket and our team will be happy to assist.
Until the cable cuts in the APAC region are completely resolved, we expect there to be intermittent periods of degraded performance, especially during APAC business hours.
We'll continue to monitor the situation through the next few days and provide any needed updates. If you experience any issues, feel free to open a support ticket and our team will be happy to assist.
Identified
Due to the ongoing subsea fiber faults in the APAC region, our Engineering team is observing a recurrence of packet loss and increased latency between Singapore and Europe/North American regions, during peak hours in Asia.
We've received reports of users experiencing degraded performance for Droplets and Droplet-based services, as well as with pushes to Container Registry, and deployments of App Platform apps.
Our team is actively engaged in attempting to re-route traffic where possible to improve performance. Until the cable cuts are fully resolved, users may continue to experience intermittent periods of degraded service.
You can see our previous post on this incident here: https://status.digitalocean.com/incidents/4w1yx5p58p1t
We've received reports of users experiencing degraded performance for Droplets and Droplet-based services, as well as with pushes to Container Registry, and deployments of App Platform apps.
Our team is actively engaged in attempting to re-route traffic where possible to improve performance. Until the cable cuts are fully resolved, users may continue to experience intermittent periods of degraded service.
You can see our previous post on this incident here: https://status.digitalocean.com/incidents/4w1yx5p58p1t
This incident affected: Container Registry (SGP1), App Platform (Singapore), Droplets (SGP1), and Networking (SGP1).