Multiple Services Impacted due to Missed Actions
Incident Report for DigitalOcean
Resolved
Our engineering team has resolved the aforementioned issues impacting the following services:

Support Center
Cloud Firewalls
Tags
Managed Databases
Load Balances
VPC
Custom Images
Projects
Referral Codes

There is no ongoing impact expected for these services listed above, and all actions should have been reconciled. Remaining areas of impact from this incident are outlined below:

User and Team Emails: Some email notifications directed towards users during the incident window on February 20 from 15:40 UTC until approximately 20:50 UTC were not sent. This would include email notifications related to Two-Factor Authentication, Secure Sign-In, Password Reset, New Droplet Creation/Password, Team invitation emails, and Monitoring alerts configured in the Cloud Control Panel. Due to the nature of this incident these emails are not able to be resent, and if you are missing any such notifications then you may attempt to re-run the initial action at this time(such as issuing a new password reset request).

Activity History: Users will not be able to see activity items on the Security page within our Cloud Control Panel for events during the affected window. Due to the nature of this incident, these actions cannot be backfilled.

If you continue to experience any problems related to these items, then please open a ticket with our support team. We apologize for any inconvenience, and we appreciate your patience throughout this process.
Posted Feb 21, 2020 - 20:52 UTC
Update
We've identified impact to multiple services, please refer to https://status.digitalocean.com/incidents/q5nr333lpzhs for the complete details.
Posted Feb 21, 2020 - 01:32 UTC
Identified
Our Engineering team has identified an issue impacting multiple services across our platform. Impact from this issue started occurring at 15:40 UTC. Around 20:50 UTC, new impact from the issue was stopped due to changes made by our Engineering Team. Our team is currently in the process of mitigating impact from this issue.

We are able to remediate some of the impacts from this issue, but users will need to retry some actions to ensure they are in place. While we continue to identify which services users need to take direction action on, we are providing a list of services impacted, along with a short description of impact to users below:

Support Center: Users who signed up for DigitalOcean within the timeframe noted above may experience an error when attempting to access the support portal or submit tickets at https://cloudsupport.digitalocean.com/s/. Existing users should not be affected, and may continue to use the support portal normally.

To work around any issues with the support portal, users may submit tickets via our contact form here: https://www.digitalocean.com/company/contact/#support


Cloud Firewalls: Users who made Cloud Firewall changes during the affected window may experience failures or delays. We recommend double-checking your Cloud Firewalls and re-applying rules as needed.

Tags: As per our previous post here: https://status.digitalocean.com/incidents/10drvkrcrhp0, Tags applied to Droplets at creation time were not applied during the affected window.

Monitoring: Monitoring alerts set up on user services may not have been sent, or may be delayed during the affected window.

Activity History: Users will not be able to see activity items on the Security page within our Cloud Control Panel for events during the affected window.


Managed Databases: Account or configuration changes for Managed Databases may have failed to be processed, or may be reverted to a prior state during the affected window.


Load Balancers: Automatic addition of tagged target droplets to a tagged Load Balancer will be delayed or fail.


Cloud Control Panel Notifications: Pop-up notifications for events like Droplet creation, deletion, etc., as well as notifications related to Projects, such as moving resources, were not displayed in our Cloud Control Panel during the affected window.

VPC: A small subset of VPC Limited Availability users may have a stale view of resources in their VPC when reading from the Cloud Control Panel or API during the affected window.

Custom Images: Users who created Custom Images during the affected window would have observed longer than expected delays before upload completion.

User Emails: Emails to users during the affected window related to Two-Factor Authentication and Secure Sign-In were not sent to users.

User Sign-Ins: Users that converted their personal DigitalOcean account to a DigitalOcean Team account during the affected window would not have their Two-Factor Authentication method moved to their new user, resulting in being unable to sign-in if the user logged out.

DigitalOcean Teams: Users may not have been able to join teams during the affected window, due to team invite emails not being sent. Users would have been able to join teams via the invite link in their Cloud Control Panel.


Projects: During the affected window, users may have experienced some resources not displaying in any projects. These resources would have still been available via other pages in the Cloud Control Panel. Project activity history during the affected window is also missing.

Referral Codes: Users who signed up for DigitalOcean during the affected window did not have their referral codes generated correctly.
Posted Feb 21, 2020 - 01:16 UTC
This incident affected: Services (API, Cloud Control Panel, Cloud Firewall, Droplets, Event Processing, Load Balancers, Managed Databases, Monitoring, Networking) and Regions (Global).