Event Processing Delays and Droplets Down in SFO2
Incident Report for DigitalOcean
Resolved
All networking, power, event processing, and Droplets are back online in SFO2. If you experience any further issues, please feel free to write in a support ticket.
Posted 6 months ago. Apr 11, 2017 - 22:57 UTC
Update
Power, networking, events, and Droplets are all still online in SFO2. We are continuing to monitor the situation to ensure there are no further problems.
Posted 6 months ago. Apr 11, 2017 - 22:30 UTC
Monitoring
We have restored power and networking, re-enabled event processing, and brought all affected Droplets back online for SFO2 at this time. We will continue to monitor the situation to ensure there are no further issues.
Posted 6 months ago. Apr 11, 2017 - 20:18 UTC
Update
Multiple redundant power sources in our SFO2 datacenter failed temporarily causing a large number of Droplets and core systems, including networking, to go down. We have restored power and networking, and have re-enabled event processing for the datacenter.

We have successfully powered on 50% of the Droplets which were powered off. We will continue to power on the remaining Droplets in a staggered fashion, which may cause some event processing delays. We will update status when that process is complete.
Posted 6 months ago. Apr 11, 2017 - 19:38 UTC
Update
Multiple redundant power sources in our SFO2 datacenter failed temporarily causing a large number of Droplets and core systems, including networking, to go down. We have restored power and networking, and have re-enabled event processing for the datacenter.

We are currently powering on all affected Droplets in a staggered fashion, which may cause some event processing delays. We will update status when that process is complete.
Posted 6 months ago. Apr 11, 2017 - 18:32 UTC
Update
Multiple redundant power sources in our SFO2 datacenter failed temporarily causing a large number of Droplets and core systems, including networking, to go down. We have restored power and networking, and have re-enabled event processing for the datacenter.

We will begin bringing Droplets back online shortly, but in the interim there may be delays in event processing. We will update the status page once all events are processing without delay and publish further details to our blog once we've conducted a full post-mortem.
Posted 6 months ago. Apr 11, 2017 - 17:44 UTC
Update
Multiple redundant power sources in our SFO2 datacenter failed temporarily causing a large number of Droplets and core systems, including networking, to go down. We have restored power and networking, and have re-enabled event processing for the datacenter.

We will begin bringing Droplets back online shortly, but in the interim Load Balancers and Floating IPs may experience intermittent issues and there may be delays in event processing. We will update the status page once all events are processing without delay and publish further details to our blog once we've conducted a full post mortem.
Posted 6 months ago. Apr 11, 2017 - 16:44 UTC
Update
Multiple redundant power sources in our SFO2 datacenter failed temporarily causing a large number of Droplets and core systems, including networking, to go down. We have restored power and networking, and are currently working to bring all affected Droplets safely online.

Following the full power recovery, we started to experience delays in event processing. Our engineering team was able to isolate the root cause. We will update the status page once all events are processing without delay and publish further details to our blog once we’ve conducted a full post mortem.
Posted 6 months ago. Apr 11, 2017 - 14:49 UTC
Issue
Our engineering team is continuing to work toward resolving connectivity issues in the SFO2 region. During this time you may continue to experience slow or no connection to your Droplets. We will provide another update as soon as possible.
Posted 6 months ago. Apr 11, 2017 - 12:59 UTC
Update
Our engineering team is continuing to work toward resolving connectivity issues in the SFO2 region. During this time you may continue to experience slow or no connection to your Droplets. We will provide another update as soon as possible.
Posted 6 months ago. Apr 11, 2017 - 10:23 UTC
Update
Our engineering team has identified the cause of this connectivity issue in the SFO2 region and is working with our facility vendor to have this corrected as soon as possible.
Posted 6 months ago. Apr 11, 2017 - 08:03 UTC
Update
Our engineering team is aware of connectivity issues in the SFO2 region, and we are actively working towards a resolution. During this time you may experience slow, or no connection to your droplets. We will provide an update soon.
Posted 6 months ago. Apr 11, 2017 - 07:45 UTC
Investigating
Our engineering team is investigating reports of network connectivity issues in the SFO2 region. During this time you may experience slow, or no connection to your droplets. We will provide an update soon.
Posted 6 months ago. Apr 11, 2017 - 07:00 UTC
This incident affected: Services (Droplets, Event Processing) and Regions (SFO2).