Every month, as part of our process for handling user suspension for nonpayment, we send up to three notices within a 15 day period to notify users that their account will be suspended if they don’t pay the past due balance. If no action is taken we reserve the right to delete the user’s suspended machines after the final termination notice (see item 5.7 in our Terms of Service Agreement). On 2017-06-21 during a routine update to our account suspension process, an incorrect query caused the deletion of all resources (Droplets, Load Balancers, etc) belonging to 1,913 users. This occurred sooner than we intended for it to happen.

We learned that the root cause was a legacy process that did not correctly delete the suspension time for these users. When we performed a routine update there was an inconsistency between the users’ status and the suspension time, which caused the deletion of the accounts and their data.

We conducted a recovery process for all users who had backups enabled on their Droplets as well as users who had taken existing snapshots. In total we were able to recover backups and snapshots for 522 Droplets for some 315 total users. We were unable to recover the resources of all other affected users. We are sorry for this incident and the impact it has had on you. Below we’ve outlined the measures we’re taking to prevent this in the future.

Future Measures
Now that we have identified the underlying cause of this incident, we have taken steps to ensure that user records will not be able to get into this state going forward. Additionally, we will be adding a ‘buffering’ mechanism between our suspension flow and resource deletion, to ensure that a more thorough recovery is possible in the future.

In Conclusion
We apologize to all of the affected users for this incident, and we are sorry that the result was a loss of your personal data. We take your data and trust very seriously and we are going through the process of crediting users that have reached out to us about the issue. If we are not already in direct contact and you did suffer a loss from this event, please open a ticket with our support team as soon as possible. While we encourage all users to take backups into their own hands, we fully take responsibility for this incident and apologize for the problems it may have caused you.