Subsea Fiber Faults in the APAC region
Incident
Report for DigitalOcean
Identified
Our Engineering team has observed recurring incidents that are continuing to affect our customers' resources in the APAC region. As a result, some users may be experiencing packet loss and increased latency in this area. Our team is actively monitoring the situation and implementing traffic routing changes where applicable to alleviate the congestion.
We apologize for any inconvenience caused and will provide updates as the situation progresses.
We apologize for any inconvenience caused and will provide updates as the situation progresses.
Monitoring
The situation with network performance in the APAC region has continually improved in the past few weeks as repairs have progressed and traffic optimization to route around affected paths has continued. We have now been observing normal latency and zero packet loss for a few days, so we will move this incident to a Monitoring state for a few days. If the improvements we’ve seen continue, we’ll look to close this out and provide any further updates separately. Thank you for your patience as this situation unfolded and our Network Engineering team worked around it.
Update
Our Engineering team continues to monitor network performance in the APAC region and make routing changes as needed to alleviate performance concerns.
Customers are still experiencing periodic performance and connectivity issues and will continue to experience those issues until restoration efforts are completed by upstream providers for the undersea cable cuts (fiber faults) in the APAC region.
Full repair is expected to take multiple weeks. We will continue to communicate any updates we receive.
Customers are still experiencing periodic performance and connectivity issues and will continue to experience those issues until restoration efforts are completed by upstream providers for the undersea cable cuts (fiber faults) in the APAC region.
Full repair is expected to take multiple weeks. We will continue to communicate any updates we receive.
Update
Our Engineering team is continuing to take actions to mitigate customer impact and monitor the ongoing issues with network connectivity in the APAC region. Performance has been relatively stable as traffic is slowly increasing across the rerouted network. However, some customers may still experience periodic performance and connectivity issues until restoration efforts are completed by the upstream providers.
We have no ETA for the upstream issue being completely resolved, but we will continue to communicate any relevant information as soon as it is available to us. Thank you for your patience and we apologize for the inconvenience.
We have no ETA for the upstream issue being completely resolved, but we will continue to communicate any relevant information as soon as it is available to us. Thank you for your patience and we apologize for the inconvenience.
Update
Our Engineering team has made routing changes for some traffic to work around the congestion caused by multiple subsea fiber faults in the APAC region. We are slowly ramping up traffic on this new route and are carefully monitoring for the next 24 hours.
While this new route is expected to help ease some of the packet loss and latency from the APAC region fiber faults, we expect some users to still experience performance and connectivity drops until the faults are resolved.
We have no ETA for the upstream issues being restored but we will communicate any relevant information as we have it. We apologize for the inconvenience.
While this new route is expected to help ease some of the packet loss and latency from the APAC region fiber faults, we expect some users to still experience performance and connectivity drops until the faults are resolved.
We have no ETA for the upstream issues being restored but we will communicate any relevant information as we have it. We apologize for the inconvenience.
Update
Our Engineering team continues to monitor the ongoing issue with multiple subsea fiber faults in the APAC region. We have experienced multiple periods of packet loss and increased latency and expect to continue to see those until the issue is resolved upstream. Our team will continue observing performance and making traffic routing changes where applicable/available to work around the congestion.
While we have no ETA for the faults being restored, we will communicate updates as we have them. If you have questions or concerns about impacted services from this incident, we ask that you open a Support ticket from within your account. Thank you!
While we have no ETA for the faults being restored, we will communicate updates as we have them. If you have questions or concerns about impacted services from this incident, we ask that you open a Support ticket from within your account. Thank you!
Update
Our Engineering team is continuing to work with upstream providers to fix the network connectivity issues we are experiencing in the SGP1 region. The problems appear to be a direct result of multiple subsea fiber faults in the APAC region causing congestion resulting in packet loss and increased latency connecting to the SGP1 region.
At this time users will continue experiencing intermittent timeout errors with Droplet-based services in the SGP1 region. We do not have a firm ETA for resolution time but we will provide updates as needed. Thank you for your patience and we apologize for any inconvenience.
At this time users will continue experiencing intermittent timeout errors with Droplet-based services in the SGP1 region. We do not have a firm ETA for resolution time but we will provide updates as needed. Thank you for your patience and we apologize for any inconvenience.
Update
Our team is continuing to work on a fix for this issue. We will provide an update in the due course along with more information about this problem. We apologize for the inconvenience and thank you for your patience and continued support.
Identified
As of 12:00 UTC, our Engineering team has again identified an issue with an upstream provider causing packet loss and increased latency in the SGP1 region. Users may have been experiencing degraded performance including timeout errors with Droplet-based services in SGP1. Please note that the concerned team is actually working on a fix and we will share an update once we have further information.
This incident affects: Regions (SGP1) and Services (Networking).