Degraded Spaces Availability in NYC3
Incident Report for DigitalOcean
Resolved
Our engineering team has resolved the issue with Spaces in our NYC3 region. If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.
Posted 2 months ago. Jul 11, 2018 - 20:55 UTC
Update
While Spaces availability in our NYC3 region is stable for most users and currently averaging 99%, we are still experiencing issues with an affected cluster, resulting in periodic errors for some users. Overnight, we implemented a change suggested by the Ceph community, and performed emergency maintenance on the cluster. Around 7am UTC, the cluster had stabilized and we paused maintenance for a few hours. There continue to be instances of errors and slow requests and we have a temporary workaround in place to address these drops while we continue to work towards full resolution. Our continued maintenance is structured to keep clusters online and limit impact to our users. Again, we appreciate your patience as we work through this issue and will continue to post updates here.
Posted 2 months ago. Jul 11, 2018 - 15:00 UTC
Update
We continue to work to resolve the issues impacting Spaces availability in our NYC3 region, and have confirmed an issue in a recently upgraded Ceph cluster. Our engineering team is performing an emergency maintenance for the affected nodes in the cluster, which we expect will last over the next several hours. While some users may continue to experience errors during this time, we are performing this emergency maintenance in a way that enables us to keep the cluster online. In an effort to maintain availability, we have also put temporary workarounds in place; if you receive an error when attempting to access your Spaces, we suggest that you wait for a moment and then reattempt. We apologize for the inconvenience and will share an update in the morning, or as soon as the issue is fully resolved.
Posted 2 months ago. Jul 11, 2018 - 02:58 UTC
Update
While we continue to experience issues with Spaces availability for some users in our NYC3 region, our engineering team has implemented mitigations that have reduced the frequency of errors for users attempting to access their Spaces. We continue to work towards a complete resolution and will post updates here as they become available.
Posted 2 months ago. Jul 10, 2018 - 19:18 UTC
Update
Our engineering team has identified an issue impacting Ceph clusters in our NYC3 region. At this time, users with Spaces accounts hosted in this region may receive an error message when they try to access their Spaces. We are actively working on a fix and will post an update as soon as possible.
Posted 2 months ago. Jul 10, 2018 - 18:00 UTC
Identified
Our engineering team has identified the cause of the issue with Spaces availability in our NYC3 region and is actively working on a fix. We will post an update as soon as additional information is available.
Posted 2 months ago. Jul 10, 2018 - 17:02 UTC
Update
We are continuing to investigate this issue. You will likely continue to see degraded performance and request failures to Spaces in our NYC3 region at this time. We will share additional details when we have them.
Posted 2 months ago. Jul 10, 2018 - 16:01 UTC
Investigating
Our engineering team is investigating an issue with decreased Spaces availability in our NYC3 region. During this time you may experience intermittent request failures to Spaces in NYC3. You may also experience slower load times for objects in that region. We apologize for the inconvenience and will share an update once we have more information.
Posted 2 months ago. Jul 10, 2018 - 15:03 UTC
This incident affected: Regions (NYC3) and Services (Spaces).