Support Center Unavailable
Incident Report for DigitalOcean

The Incident

On Monday July 31st 2017, we discovered a security bug that allowed one of our users to view other customer data in the newly deployed support portal. After validating and inspecting our logs, we have confirmed that only two records were exposed to one user, and the user who found this issue reported it to us immediately. It’s important to note that no customer billing data was compromised. We have personally notified the two customers whose accounts were impacted. We take your data privacy very seriously, and have mitigated the issue with the bugfix, and updated our internal change management policies to ensure this issue will not reoccur.

Timeline of Events

  • 10:29 UTC: A user reports an information disclosure in the new support portal

  • 12:24 UTC: DigitalOcean support discovers the report and escalates internally

  • 12:50 UTC: Incident response starts, our security team disables the endpoint that contains the primary information disclosure while further investigation is performed

  • 13:35 UTC: Full scope of information disclosure is now understood, support portal is taken offline

  • 14:12 UTC: Investigation of access logs shows scope of disclosure

  • 15:53 UTC: Final mitigations are in place for the specific incident

  • 16:28 UTC: Support portal brought back online and incident resolved

Future Measures

Our new support portal is built on top of a well-known third party SaaS platform. This sits outside the traditional operational model for DigitalOcean engineers. Our post-mortem investigation led to the discovery of a blind spot in our operational practices and playbooks for these kinds of systems. Over the coming months, we will review and write new standard organizational procedures for the development, operations, and security of this class of system.

Our standard procedures made it unclear as to what the scope of any necessary security assessment for this system was. After conducting our internal analysis we are reviewing our internal procedures and making a determination as to what further security assessments are necessary to prevent similar situations from occurring in the future.

In Conclusion

We take any form of negative impact on your service seriously. Our goal is to constantly improve and we hope that our transparency and the details that we have shared help show you that.

Posted about 2 months ago. Aug 02, 2017 - 17:17 UTC

Resolved
Access to the Support Center has been fully restored. Thank you for your patience, and we apologize for the inconvenience.
Posted about 2 months ago. Jul 31, 2017 - 17:02 UTC
Monitoring
We've restored access to the Support Center and we'll continue to monitor closely. We apologize for the inconvenience while it was unavailable. If you still experience any problems loading the Support Center, please open a ticket from your account or fill out the contact form at https://do.co/contact.
Posted about 2 months ago. Jul 31, 2017 - 16:18 UTC
Update
We're continuing internal testing to resolve the issue, and once we validate the fixes, they will be deployed and the Support Center will be available again. We appreciate your patience.
Posted about 2 months ago. Jul 31, 2017 - 15:48 UTC
Identified
We've mitigated the issue affecting the Support Center, and are continuing work on fully restoring connectivity. We'll share additional updates as we have them, and in the meantime users may submit tickets through our contact form at https://do.co/contact.
Posted about 2 months ago. Jul 31, 2017 - 14:35 UTC
Update
Our Support Center is currently unavailable and our engineers are continuing to investigate the issue. During this time, you can fill out the contact form at https://do.co/contact to submit a ticket, though responses from our Support team may still be delayed. We'll share further updates as we have more information.
Posted about 2 months ago. Jul 31, 2017 - 13:23 UTC
Investigating
Our engineers are investigating an issue with the Support Center. During this time, the Support Center may be unavailable for customers to log in, and replies from our Support team may be delayed.
Posted about 2 months ago. Jul 31, 2017 - 13:05 UTC
This incident affected: Services (Support Center).