Our Engineering team has identified an issue impacting multiple services across our platform. Impact from this issue started occurring at 15:40 UTC. Around 20:50 UTC, new impact from the issue was stopped due to changes made by our Engineering Team. Our team is currently in the process of mitigating impact from this issue.
We are able to remediate some of the impacts from this issue, but users will need to retry some actions to ensure they are in place. While we continue to identify which services users need to take direction action on, we are providing a list of services impacted, along with a short description of impact to users below:
Support Center: Users who signed up for DigitalOcean within the timeframe noted above may experience an error when attempting to access the support portal or submit tickets at https://cloudsupport.digitalocean.com/s/
. Existing users should not be affected, and may continue to use the support portal normally.
To work around any issues with the support portal, users may submit tickets via our contact form here: https://www.digitalocean.com/company/contact/#support
Cloud Firewalls: Users who made Cloud Firewall changes during the affected window may experience failures or delays. We recommend double-checking your Cloud Firewalls and re-applying rules as needed.
Tags: As per our previous post here: https://status.digitalocean.com/incidents/10drvkrcrhp0
, Tags applied to Droplets at creation time were not applied during the affected window.
Monitoring: Monitoring alerts set up on user services may not have been sent, or may be delayed during the affected window.
Activity History: Users will not be able to see activity items on the Security page within our Cloud Control Panel for events during the affected window.
Managed Databases: Account or configuration changes for Managed Databases may have failed to be processed, or may be reverted to a prior state during the affected window.
Load Balancers: Automatic addition of tagged target droplets to a tagged Load Balancer will be delayed or fail.
Cloud Control Panel Notifications: Pop-up notifications for events like Droplet creation, deletion, etc., as well as notifications related to Projects, such as moving resources, were not displayed in our Cloud Control Panel during the affected window.
VPC: A small subset of VPC Limited Availability users may have a stale view of resources in their VPC when reading from the Cloud Control Panel or API during the affected window.
Custom Images: Users who created Custom Images during the affected window would have observed longer than expected delays before upload completion.
User Emails: Emails to users during the affected window related to Two-Factor Authentication and Secure Sign-In were not sent to users.
User Sign-Ins: Users that converted their personal DigitalOcean account to a DigitalOcean Team account during the affected window would not have their Two-Factor Authentication method moved to their new user, resulting in being unable to sign-in if the user logged out.
DigitalOcean Teams: Users may not have been able to join teams during the affected window, due to team invite emails not being sent. Users would have been able to join teams via the invite link in their Cloud Control Panel.
Projects: During the affected window, users may have experienced some resources not displaying in any projects. These resources would have still been available via other pages in the Cloud Control Panel. Project activity history during the affected window is also missing.
Referral Codes: Users who signed up for DigitalOcean during the affected window did not have their referral codes generated correctly.