Limited Availability of Spaces API in NYC3
Incident Report for DigitalOcean
Resolved
Our Engineering team has resolved the issues that have been impacting the NYC3 Spaces region. If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.
Posted 23 days ago. Nov 21, 2018 - 16:06 UTC
Monitoring
Our Engineering team has restored all systems affecting the availability of the NYC Spaces cluster. We will continue to monitor the situation to ensure ongoing stability, and we'll post an update when the situation is fully resolved.
Posted 24 days ago. Nov 21, 2018 - 02:25 UTC
Update
Our Engineering Team is continuing the effort to restore accessibility to all objects. Although we have succeeded in restoring accessibility to the majority of affected objects, we estimate a very small percentage of our customers are still affected. We plan to share our next update tomorrow. Thank you for your continued patience.
Posted 25 days ago. Nov 19, 2018 - 23:50 UTC
Update
Our Engineering Team is continuing the effort to restore accessibility to the 0.01% of objects that are currently unavailable. We anticipate this work will last through the evening, and we plan to share our next update tomorrow morning (ET). We appreciate your patience.
Posted 26 days ago. Nov 19, 2018 - 00:52 UTC
Update
We have restored availability to over 90% of the objects that were previously inaccessible. We estimate now that about 0.01% of the objects are inaccessible, and the team is continuing efforts to restore these objects. We appreciate your patience and we will provide another update here when we have more information.
Posted 26 days ago. Nov 18, 2018 - 12:55 UTC
Update
Our Engineering Team continues to work on the issue that has impacted Spaces in our NYC3 region since an emergency maintenance that was started and halted on November 12. Currently, .1% of the Objects stored in this datacenter are unavailable.

Recovery efforts are progressing, and the team is working to restore the cluster; we have made progress bringing up some disk systems. Resolving this issue remains our team's top priority, and we are working diligently to return access to the inaccessible Objects. Our recovery efforts are manual and intensive, and we still expect this effort to continue over the next several days. We apologize for the frustrations caused by this on-going issue and will share another update here tomorrow.
Posted 27 days ago. Nov 18, 2018 - 02:52 UTC
Update
Our Engineering Team continues to work on the issue that has impacted Spaces in our NYC3 region since an emergency maintenance that was started and halted on November 12. Currently, .1% of the Objects stored in this datacenter are unavailable.

Resolving this issue remains our team's top priority; and we are working diligently to return access to the inaccessible objects. Our recovery efforts are manual and intensive, and we still expect this effort to continue over the next several days. We apologize for the frustrations caused by this on-going issue and will share another update here tomorrow.
Posted 28 days ago. Nov 17, 2018 - 02:29 UTC
Update
Our Engineering Team continues to work on the issue that has impacted Spaces in our NYC3 region since an emergency maintenance that was started and halted on November 12. During the maintenance, we had completed a RAM upgrade on a data node, and following the restart several drives on the node failed to come back online. This impacted several Objects that had data on those drives. The issue was compounded when several other processes in the cluster encountered a secondary file system problem that manifested as drives locking up. Several data nodes were affected, and as a result a small percentage of Objects stored in our NYC3 cluster are currently unavailable.

Resolving this issue remains our team's top priority; however, we do expect that our efforts to return access to the inaccessible objects will last at least over the next couple of days. We plan to post our next update tomorrow, and will reach out to impacted users with more information, as it becomes available. We apologize for the frustrations caused by this on-going issue.
Posted 29 days ago. Nov 15, 2018 - 20:33 UTC
Update
Our Engineering Team continues to work on an issue impacting Spaces in our NYC3 region. During the emergency maintenance, we encountered a hardware problem and, as a result, a small percentage of objects are not currently accessible for some users.

Resolving this issue remains our team's top priority; however, we do expect that our efforts to return access to the inaccessible objects will last at least over the next couple of days. Updates about this incident will be posted here, and we will email all impacted users additional information.
Posted 30 days ago. Nov 14, 2018 - 20:35 UTC
Identified
Our Engineering Team is actively working on an issue impacting Spaces in our NYC3 region. Some users may be unable to access Spaces through the API.

Upon identifying this issue yesterday, our team began an emergency maintenance. Unfortunately, we ran into additional issues during that maintenance, thus temporarily halting that operation. We continue to work towards a fix for the issue affecting Spaces, and once we have resolved the situation, we will resume maintenance to improve overall performance and stability for Spaces in our NYC3 region.

While resolving this issue is one of our top priorities, we do expect that resolution to take some time. Updates about this incident will be posted here, and we will email all impacted users additional information about the maintenance once this issue is resolved and plans for the maintenance window are confirmed.
Posted about 1 month ago. Nov 13, 2018 - 21:18 UTC
This incident affected: Regions (NYC3) and Services (Spaces).