Email Notifications
Incident Report for DigitalOcean
Resolved
Our Engineering team has completed work to identify remaining emails impacted by this incident. At this time, there remains a small subset of emails not being delivered to users, including some billing emails like finalized invoices and billing threshold alerts, App Platform alerts, and DBaaS alerts. Multiple teams are working together to ensure that these emails are able to be sent out correctly and that work will continue.

At this time, we are going to resolve the incident due to critical email functionality being restored. If you have any concerns about the remaining emails impacted, or any questions about this incident, please open a support ticket from within your account.

Thank you for your patience throughout this process and we apologize for the extended disruption.
Posted Aug 12, 2022 - 01:34 UTC
Update
Our Engineering team has deployed another fix for an additional subset of emails. Users are now able to receive the following in addition to our previous updates:

- Droplet root password resets
- Let's Encrypt certificate creations and renewals
- Monitoring alerts for resource utilization
- Payment and credit notifications
- New Droplet creation notifications
- Maintenance notices for Droplets and Droplet-based products
- Broader maintenance notices like region-wide networking maintenance

Our team continues working to restore full functionality. At this time, we're identifying additional emails that are not flowing correctly and working to restore other billing emails, like completed invoices. We'll continue to provide updates as this work progresses. Thank you.
Posted Aug 11, 2022 - 17:39 UTC
Update
Our Engineering team has deployed another fix for an additional subset of emails. Users are now able to receive the following in addition to our previous updates:

- Team owner changes/additions
- Team secure sign-in emails
- All 2FA emails (notifications of enabling, disabling, and backup code reminders)

Our team continues working to restore full functionality, including Droplet root password reset emails and billing notifications. We'll continue to provide updates as this work progresses. Thank you.
Posted Aug 10, 2022 - 15:44 UTC
Update
Our Engineering team has deployed and verified the fix. Users are now able to receive the following in addition to our previous update:

- Team member invitations
- New team creation notifications

Our team will continue work to restore full functionality of email notifications and we will provide an update before 16:00 UTC August 10.

If you submitted a support ticket to regain access to your account or help with inviting team members as a result of this incident and are now able to access your account/add team members, we ask that you utilize the "Close Ticket" functionality on https://cloudsupport.digitalocean.com/. Our Support team is working as quickly as possible to address all account access queries. Thank you.
Posted Aug 10, 2022 - 02:38 UTC
Update
Our Engineering team has deployed and verified the fix. Users are now able to receive:

- Account password resets
- One-time device verification codes
- New user verifications when signing up at cloud.digitalocean.com

Our team is now working to deploy a fix for team-related emails such as team member invitations. We will post an update once that fix has been deployed and verified.

If you submitted a support ticket to regain access to your account as a result of this incident and are now able to access your account, we ask that you utilize the "Close Ticket" functionality on https://cloudsupport.digitalocean.com/. Our Support team is working as quickly as possible to address all account access queries.
Posted Aug 10, 2022 - 02:11 UTC
Update
Our Engineering team has taken action to restore email functionality and is currently implementing a fix to allow a subset of email notifications to be sent correctly. This will include account password resets, one-time verification codes, and new user sign up confirmations to be sent.

We will post another update once the fix has been deployed and verified. Thank you for your patience thus far.
Posted Aug 10, 2022 - 00:38 UTC
Update
Our Engineering team is continuing work to restore email functionality, but we do not have any additional updates at this time. Users continue to be unable to receive a variety of email notifications.

As soon as we have further information, we will provide an update. Thank you.
Posted Aug 09, 2022 - 13:43 UTC
Identified
Our Engineering team has identified the cause of the issue and is working to restore full email functionality.

At this time, we have confirmed the following emails are impacted:
- Account password resets
- One-time device verification codes
- New user verifications (users may still sign up using Google or GitHub)
- Team invitations
- Droplet root password resets
- User credit grants
- Payment notifications
- Let's Encrypt certificate creations and renewals

We realize how impacting this is and our Support team is working to address login issues. If users need to reset Droplet passwords, we ask they do so via the Recovery Console. If you are having trouble accessing your cloud.digitalocean.com account, please open a ticket via https://www.digitalocean.com/support/contact

We expect to provide another update around 13:00 UTC. We apologize for the trouble and thank you for your patience.
Posted Aug 09, 2022 - 02:44 UTC
Investigating
Our Engineering team is investigating an issue related to email notifications for DigitalOcean accounts not being sent. At this time, users will not be able to receive emails for account password resets, team invites, one-time verification codes, and other transactional email.

Users will also be unable to sign up for new cloud.digitalocean.com accounts using an email address at this time, but can sign up using Google or GitHub authentication.

We are working to determine what other email notices are impacted and will post an update with further information as soon as possible.

If you are unable to sign into your DigitalOcean account and need urgent access, please open a ticket with our Support team to assist you in doing so.
Posted Aug 08, 2022 - 20:53 UTC
This incident affected: Services (Cloud Control Panel).