Subsea Fiber Faults in the APAC region
Incident
Report for DigitalOcean
Resolved
Our Engineering team has continued to actively monitor the situation resulting from multiple subsea fiber faults in the APAC region. Over the last ten days, network connectivity on routes from Singapore to San Francisco and Singapore to Europe has not experienced any major packet loss or latency. Network routes to and from Bangalore also have not experienced any major packet loss or latency. Crews have been able to complete some cable repairs, leading to the stability for the routes mentioned.
Our team is still observing daily packet loss and latency on network routes between Singapore and New York City, as well as Singapore and Toronto. These are normally short-lived and happen during Singapore business hours when traffic is heavy.
Repair of all cables from crews is still pending and may continue for multiple weeks. Given the relative stability of routes, we will now close out this incident and provide any needed updates separately.
If users experience a disruption in service or have any questions, we invite them to submit a support ticket from within their account.
Thank you for your patience and understanding throughout this incident.
Resolved: May 01, 2023 - 15:58 UTC
-
Update
Around 07:00 UTC, we saw a small period of increased loss, due to increased traffic and congestion in the APAC region. This self-recovered and since then, we have not seen further periods of decreased performance. About two hours later, we experienced an unrelated issue in BLR1, details of which can be found here: https://status.digitalocean.com/incidents/6qr2q9yc9790. We originally believed the impact in BLR1 to be due to this incident, but confirmed it was a separate issue. We apologize for any confusion.
As a reminder, we expect periods of increased loss and latency to continue to occur, but they are normally short-lived, and most happen during APAC business hours due to increased traffic and congestion.
We continue to await the fiber fault repairs and make traffic routing changes when available for better performance. We expect the situation to continue to improve over the coming month as crews continue and complete work.
Mar 29, 2023 - 10:25 UTC
-
Update
Between 10:20 - 11:10 UTC, we experienced an uptick in loss and latency in the APAC region, due to a downstream effect of this incident. During this time, we saw a slight dip in public API availability, meaning a small subset of users may have experienced errors while sending requests. Users would have also seen loss and latency on Droplets and Droplet-based services in the APAC region.
Periods of increased loss and latency continue to occur, but are normally short-lived, and most happen during APAC business hours due to increased traffic and congestion.
We continue to await the fiber fault repairs and make traffic routing changes when available for better performance. We expect the situation to continue to improve over the coming month as crews continue and complete work.
Mar 28, 2023 - 17:54 UTC
-
Identified
Our Engineering team has observed recurring incidents that are continuing to affect our customers' resources in the APAC region. As a result, some users may be experiencing packet loss and increased latency in this area. Our team is actively monitoring the situation and implementing traffic routing changes where applicable to alleviate the congestion.
We apologize for any inconvenience caused and will provide updates as the situation progresses.
Mar 08, 2023 - 13:31 UTC
-
Monitoring
The situation with network performance in the APAC region has continually improved in the past few weeks as repairs have progressed and traffic optimization to route around affected paths has continued. We have now been observing normal latency and zero packet loss for a few days, so we will move this incident to a Monitoring state for a few days. If the improvements we’ve seen continue, we’ll look to close this out and provide any further updates separately. Thank you for your patience as this situation unfolded and our Network Engineering team worked around it.
Mar 07, 2023 - 20:56 UTC
-
Update
Our Engineering team continues to monitor network performance in the APAC region and make routing changes as needed to alleviate performance concerns.
Customers are still experiencing periodic performance and connectivity issues and will continue to experience those issues until restoration efforts are completed by upstream providers for the undersea cable cuts (fiber faults) in the APAC region.
Full repair is expected to take multiple weeks. We will continue to communicate any updates we receive.
Feb 27, 2023 - 16:34 UTC
-
Update
Our Engineering team is continuing to take actions to mitigate customer impact and monitor the ongoing issues with network connectivity in the APAC region. Performance has been relatively stable as traffic is slowly increasing across the rerouted network. However, some customers may still experience periodic performance and connectivity issues until restoration efforts are completed by the upstream providers.
We have no ETA for the upstream issue being completely resolved, but we will continue to communicate any relevant information as soon as it is available to us. Thank you for your patience and we apologize for the inconvenience.
Feb 04, 2023 - 18:33 UTC
-
Update
Our Engineering team has made routing changes for some traffic to work around the congestion caused by multiple subsea fiber faults in the APAC region. We are slowly ramping up traffic on this new route and are carefully monitoring for the next 24 hours.
While this new route is expected to help ease some of the packet loss and latency from the APAC region fiber faults, we expect some users to still experience performance and connectivity drops until the faults are resolved.
We have no ETA for the upstream issues being restored but we will communicate any relevant information as we have it. We apologize for the inconvenience.
Feb 03, 2023 - 17:58 UTC
-
Update
Our Engineering team continues to monitor the ongoing issue with multiple subsea fiber faults in the APAC region. We have experienced multiple periods of packet loss and increased latency and expect to continue to see those until the issue is resolved upstream. Our team will continue observing performance and making traffic routing changes where applicable/available to work around the congestion.
While we have no ETA for the faults being restored, we will communicate updates as we have them. If you have questions or concerns about impacted services from this incident, we ask that you open a Support ticket from within your account. Thank you!
Feb 01, 2023 - 23:55 UTC
-
Update
Our Engineering team is continuing to work with upstream providers to fix the network connectivity issues we are experiencing in the SGP1 region. The problems appear to be a direct result of multiple subsea fiber faults in the APAC region causing congestion resulting in packet loss and increased latency connecting to the SGP1 region.
At this time users will continue experiencing intermittent timeout errors with Droplet-based services in the SGP1 region. We do not have a firm ETA for resolution time but we will provide updates as needed. Thank you for your patience and we apologize for any inconvenience.
Jan 30, 2023 - 06:48 UTC
-
Update
Our team is continuing to work on a fix for this issue. We will provide an update in the due course along with more information about this problem. We apologize for the inconvenience and thank you for your patience and continued support.
Jan 29, 2023 - 20:47 UTC
-
Identified
As of 12:00 UTC, our Engineering team has again identified an issue with an upstream provider causing packet loss and increased latency in the SGP1 region. Users may have been experiencing degraded performance including timeout errors with Droplet-based services in SGP1. Please note that the concerned team is actually working on a fix and we will share an update once we have further information.
Jan 29, 2023 - 16:27 UTC
Our team is still observing daily packet loss and latency on network routes between Singapore and New York City, as well as Singapore and Toronto. These are normally short-lived and happen during Singapore business hours when traffic is heavy.
Repair of all cables from crews is still pending and may continue for multiple weeks. Given the relative stability of routes, we will now close out this incident and provide any needed updates separately.
If users experience a disruption in service or have any questions, we invite them to submit a support ticket from within their account.
Thank you for your patience and understanding throughout this incident.
Resolved: May 01, 2023 - 15:58 UTC
-
Update
Around 07:00 UTC, we saw a small period of increased loss, due to increased traffic and congestion in the APAC region. This self-recovered and since then, we have not seen further periods of decreased performance. About two hours later, we experienced an unrelated issue in BLR1, details of which can be found here: https://status.digitalocean.com/incidents/6qr2q9yc9790. We originally believed the impact in BLR1 to be due to this incident, but confirmed it was a separate issue. We apologize for any confusion.
As a reminder, we expect periods of increased loss and latency to continue to occur, but they are normally short-lived, and most happen during APAC business hours due to increased traffic and congestion.
We continue to await the fiber fault repairs and make traffic routing changes when available for better performance. We expect the situation to continue to improve over the coming month as crews continue and complete work.
Mar 29, 2023 - 10:25 UTC
-
Update
Between 10:20 - 11:10 UTC, we experienced an uptick in loss and latency in the APAC region, due to a downstream effect of this incident. During this time, we saw a slight dip in public API availability, meaning a small subset of users may have experienced errors while sending requests. Users would have also seen loss and latency on Droplets and Droplet-based services in the APAC region.
Periods of increased loss and latency continue to occur, but are normally short-lived, and most happen during APAC business hours due to increased traffic and congestion.
We continue to await the fiber fault repairs and make traffic routing changes when available for better performance. We expect the situation to continue to improve over the coming month as crews continue and complete work.
Mar 28, 2023 - 17:54 UTC
-
Identified
Our Engineering team has observed recurring incidents that are continuing to affect our customers' resources in the APAC region. As a result, some users may be experiencing packet loss and increased latency in this area. Our team is actively monitoring the situation and implementing traffic routing changes where applicable to alleviate the congestion.
We apologize for any inconvenience caused and will provide updates as the situation progresses.
Mar 08, 2023 - 13:31 UTC
-
Monitoring
The situation with network performance in the APAC region has continually improved in the past few weeks as repairs have progressed and traffic optimization to route around affected paths has continued. We have now been observing normal latency and zero packet loss for a few days, so we will move this incident to a Monitoring state for a few days. If the improvements we’ve seen continue, we’ll look to close this out and provide any further updates separately. Thank you for your patience as this situation unfolded and our Network Engineering team worked around it.
Mar 07, 2023 - 20:56 UTC
-
Update
Our Engineering team continues to monitor network performance in the APAC region and make routing changes as needed to alleviate performance concerns.
Customers are still experiencing periodic performance and connectivity issues and will continue to experience those issues until restoration efforts are completed by upstream providers for the undersea cable cuts (fiber faults) in the APAC region.
Full repair is expected to take multiple weeks. We will continue to communicate any updates we receive.
Feb 27, 2023 - 16:34 UTC
-
Update
Our Engineering team is continuing to take actions to mitigate customer impact and monitor the ongoing issues with network connectivity in the APAC region. Performance has been relatively stable as traffic is slowly increasing across the rerouted network. However, some customers may still experience periodic performance and connectivity issues until restoration efforts are completed by the upstream providers.
We have no ETA for the upstream issue being completely resolved, but we will continue to communicate any relevant information as soon as it is available to us. Thank you for your patience and we apologize for the inconvenience.
Feb 04, 2023 - 18:33 UTC
-
Update
Our Engineering team has made routing changes for some traffic to work around the congestion caused by multiple subsea fiber faults in the APAC region. We are slowly ramping up traffic on this new route and are carefully monitoring for the next 24 hours.
While this new route is expected to help ease some of the packet loss and latency from the APAC region fiber faults, we expect some users to still experience performance and connectivity drops until the faults are resolved.
We have no ETA for the upstream issues being restored but we will communicate any relevant information as we have it. We apologize for the inconvenience.
Feb 03, 2023 - 17:58 UTC
-
Update
Our Engineering team continues to monitor the ongoing issue with multiple subsea fiber faults in the APAC region. We have experienced multiple periods of packet loss and increased latency and expect to continue to see those until the issue is resolved upstream. Our team will continue observing performance and making traffic routing changes where applicable/available to work around the congestion.
While we have no ETA for the faults being restored, we will communicate updates as we have them. If you have questions or concerns about impacted services from this incident, we ask that you open a Support ticket from within your account. Thank you!
Feb 01, 2023 - 23:55 UTC
-
Update
Our Engineering team is continuing to work with upstream providers to fix the network connectivity issues we are experiencing in the SGP1 region. The problems appear to be a direct result of multiple subsea fiber faults in the APAC region causing congestion resulting in packet loss and increased latency connecting to the SGP1 region.
At this time users will continue experiencing intermittent timeout errors with Droplet-based services in the SGP1 region. We do not have a firm ETA for resolution time but we will provide updates as needed. Thank you for your patience and we apologize for any inconvenience.
Jan 30, 2023 - 06:48 UTC
-
Update
Our team is continuing to work on a fix for this issue. We will provide an update in the due course along with more information about this problem. We apologize for the inconvenience and thank you for your patience and continued support.
Jan 29, 2023 - 20:47 UTC
-
Identified
As of 12:00 UTC, our Engineering team has again identified an issue with an upstream provider causing packet loss and increased latency in the SGP1 region. Users may have been experiencing degraded performance including timeout errors with Droplet-based services in SGP1. Please note that the concerned team is actually working on a fix and we will share an update once we have further information.
Jan 29, 2023 - 16:27 UTC