Update - We have restored availability to over 90% of the objects that were previously inaccessible. We estimate now that about 0.01% of the objects are inaccessible, and the team is continuing efforts to restore these objects. We appreciate your patience and we will provide another update here when we have more information.
Nov 18, 12:55 UTC
Nov 18, 12:55 UTC
Update - Our Engineering Team continues to work on the issue that has impacted Spaces in our NYC3 region since an emergency maintenance that was started and halted on November 12. Currently, .1% of the Objects stored in this datacenter are unavailable.
Recovery efforts are progressing, and the team is working to restore the cluster; we have made progress bringing up some disk systems. Resolving this issue remains our team's top priority, and we are working diligently to return access to the inaccessible Objects. Our recovery efforts are manual and intensive, and we still expect this effort to continue over the next several days. We apologize for the frustrations caused by this on-going issue and will share another update here tomorrow.
Nov 18, 02:52 UTC
Recovery efforts are progressing, and the team is working to restore the cluster; we have made progress bringing up some disk systems. Resolving this issue remains our team's top priority, and we are working diligently to return access to the inaccessible Objects. Our recovery efforts are manual and intensive, and we still expect this effort to continue over the next several days. We apologize for the frustrations caused by this on-going issue and will share another update here tomorrow.
Nov 18, 02:52 UTC
Update - Our Engineering Team continues to work on the issue that has impacted Spaces in our NYC3 region since an emergency maintenance that was started and halted on November 12. Currently, .1% of the Objects stored in this datacenter are unavailable.
Resolving this issue remains our team's top priority; and we are working diligently to return access to the inaccessible objects. Our recovery efforts are manual and intensive, and we still expect this effort to continue over the next several days. We apologize for the frustrations caused by this on-going issue and will share another update here tomorrow.
Nov 17, 02:29 UTC
Resolving this issue remains our team's top priority; and we are working diligently to return access to the inaccessible objects. Our recovery efforts are manual and intensive, and we still expect this effort to continue over the next several days. We apologize for the frustrations caused by this on-going issue and will share another update here tomorrow.
Nov 17, 02:29 UTC
Update - Our Engineering Team continues to work on the issue that has impacted Spaces in our NYC3 region since an emergency maintenance that was started and halted on November 12. During the maintenance, we had completed a RAM upgrade on a data node, and following the restart several drives on the node failed to come back online. This impacted several Objects that had data on those drives. The issue was compounded when several other processes in the cluster encountered a secondary file system problem that manifested as drives locking up. Several data nodes were affected, and as a result a small percentage of Objects stored in our NYC3 cluster are currently unavailable.
Resolving this issue remains our team's top priority; however, we do expect that our efforts to return access to the inaccessible objects will last at least over the next couple of days. We plan to post our next update tomorrow, and will reach out to impacted users with more information, as it becomes available. We apologize for the frustrations caused by this on-going issue.
Nov 15, 20:33 UTC
Resolving this issue remains our team's top priority; however, we do expect that our efforts to return access to the inaccessible objects will last at least over the next couple of days. We plan to post our next update tomorrow, and will reach out to impacted users with more information, as it becomes available. We apologize for the frustrations caused by this on-going issue.
Nov 15, 20:33 UTC
Update - Our Engineering Team continues to work on an issue impacting Spaces in our NYC3 region. During the emergency maintenance, we encountered a hardware problem and, as a result, a small percentage of objects are not currently accessible for some users.
Resolving this issue remains our team's top priority; however, we do expect that our efforts to return access to the inaccessible objects will last at least over the next couple of days. Updates about this incident will be posted here, and we will email all impacted users additional information.
Nov 14, 20:35 UTC
Resolving this issue remains our team's top priority; however, we do expect that our efforts to return access to the inaccessible objects will last at least over the next couple of days. Updates about this incident will be posted here, and we will email all impacted users additional information.
Nov 14, 20:35 UTC
Identified - Our Engineering Team is actively working on an issue impacting Spaces in our NYC3 region. Some users may be unable to access Spaces through the API.
Upon identifying this issue yesterday, our team began an emergency maintenance. Unfortunately, we ran into additional issues during that maintenance, thus temporarily halting that operation. We continue to work towards a fix for the issue affecting Spaces, and once we have resolved the situation, we will resume maintenance to improve overall performance and stability for Spaces in our NYC3 region.
While resolving this issue is one of our top priorities, we do expect that resolution to take some time. Updates about this incident will be posted here, and we will email all impacted users additional information about the maintenance once this issue is resolved and plans for the maintenance window are confirmed.
Nov 13, 21:18 UTC
Upon identifying this issue yesterday, our team began an emergency maintenance. Unfortunately, we ran into additional issues during that maintenance, thus temporarily halting that operation. We continue to work towards a fix for the issue affecting Spaces, and once we have resolved the situation, we will resume maintenance to improve overall performance and stability for Spaces in our NYC3 region.
While resolving this issue is one of our top priorities, we do expect that resolution to take some time. Updates about this incident will be posted here, and we will email all impacted users additional information about the maintenance once this issue is resolved and plans for the maintenance window are confirmed.
Nov 13, 21:18 UTC
Regions
Operational
Global
Operational
AMS2
Operational
AMS3
Operational
BLR1
Operational
FRA1
Operational
LON1
Operational
NYC1
Operational
NYC2
Operational
NYC3
Operational
SFO1
Operational
SFO2
Operational
SGP1
Operational
TOR1
Operational
Services
Degraded Performance
API
Operational
Block Storage
Operational
Cloud Control Panel
Operational
Cloud Firewall
Operational
Community
Operational
DNS
Operational
Droplets
Operational
Event Processing
Operational
Load Balancers
Operational
Monitoring
Operational
Networking
Operational
Spaces
Degraded Performance
Support Center
Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance