At 08:57 UTC on October 13th, 2017, our Support team began seeing errors with starting the Droplet console, attaching/detaching volumes, and login device verification. Our teams eventually determined this was the result of a failing auditing subsystem. Our engineers worked to bring the subsystem back to a normal state with a patch to ensure that internal auditing failures would not cause the entire operation to be rejected. At 11:30 UTC, the fix was pushed to our production systems and the support team confirmed affected operations were then operational.
08:57 UTC - Support team detects operation failures
09:40 UTC - We identify that the internal auditing subsystem was failing
10:30 UTC - Implement patch to resolve the issue
11:30 UTC - Patch deployment is completed
11:33 UTC - Support team confirms affected actions are now operational
We are working on increasing the reliability of the aforementioned auditing subsystem. Meanwhile, we have developed a patch to ensure users’ requests will succeed regardless of the operational status of the auditing subsystem.
We’re disappointed we have let a non-essential sub-system affect user operations and we apologize for the inconveniences and frustrations it has caused.