Spaces CDN in Multiple Regions
Incident
Report for DigitalOcean
Resolved
Our Engineering team has confirmed the resolution of the issue that impacted the Spaces CDN. Objects should be accessible over the CDN endpoint without any issues. However, HTTP/2 is temporarily unavailable due to upstream issues, and due to this HTTP/2 requests should automatically be re-negotiated to 1.1, but in case you experience failures please open a ticket with our support team.
Thank you for your patience and we apologize for any inconvenience.
Thank you for your patience and we apologize for any inconvenience.
Update
After our upstream provider implemented a remediation step to resolve the issue with the Spaces CDN, the Spaces CDN is serving the objects stored in the Spaces bucket without any errors or performance issues. However, we are still monitoring the situation closely and we'll share more information as soon as it's available.
Update
We are continuing to monitor this at our end and will share further information as soon as it is available.
Update
The issue with the Spaces CDN with our upstream provider appears to have been corrected and we are currently monitoring the situation. At this time, users should see objects served over the Spaces CDN loading normally without any errors.
We will post an update as soon as we confirm the issue is fully resolved.
We will post an update as soon as we confirm the issue is fully resolved.
Update
Our Engineering team has an update from our upstream provider regarding the ongoing issue with the Spaces CDN. We have observed significant improvements in the CDN performance and the error rates have gone down as well with the recent efforts to mitigate the impact.
We are continuing to monitor this at our end and will share further information as soon as it is available.
We are continuing to monitor this at our end and will share further information as soon as it is available.
Update
Our Engineering team has observed recurrences of this incident and a subset of users might be experiencing timeouts or 5xx errors for objects served over the CDN especially in the SGP1 region.
We are closely monitoring the situation and will post another update in the due course. We appreciate your patience and understanding. If you have any questions or concerns in the meantime, please feel free to let us know.
We are closely monitoring the situation and will post another update in the due course. We appreciate your patience and understanding. If you have any questions or concerns in the meantime, please feel free to let us know.
Monitoring
As of 06:10 UTC, our Engineering team has been made aware that our upstream provider has implemented a remediation step for the Spaces CDN. Users should no longer experience 5xx or time-out errors.
During this time, users may still experience slow loading times for objects served over the CDN.
We will continue to closely monitor the issue from our side and will provide further updates as they become available to us.
During this time, users may still experience slow loading times for objects served over the CDN.
We will continue to closely monitor the issue from our side and will provide further updates as they become available to us.
Identified
Our Engineering team has been made aware of an issue with our upstream provider for the Spaces CDN. During this time, the provider is experiencing service degradation in certain point of presence regions, including: Sao Paolo, Sydney, Melbourne, Miami, and Madrid.
Customers may experience 5xx errors, time-outs, and slow loading times for objects served over the CDN.
We will provide updates as they become available to us.
Customers may experience 5xx errors, time-outs, and slow loading times for objects served over the CDN.
We will provide updates as they become available to us.
This incident affected: Spaces CDN (AMS3, FRA1, NYC3, SFO3, SGP1, SYD1).