DigitalOcean Status

Feb 27, 2020

No incidents reported today.

Feb 26, 2020

Elevated Error Rates for Domains in Cloud Control Panel and API

Resolved - Our engineering team has resolved the issue with Domains in our Cloud Control Panel and API, and these services should be operating normally at this time. If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.
Feb 26, 21:35 UTC

Monitoring - Our engineering team has implemented a fix to correct the errors with Domains in our Cloud Control Panel and API, and functionality should be fully restored at this time. Our team is currently monitoring the situation for any further errors, and we will share an update once we are certain that the matter is fully resolved.
Feb 26, 21:03 UTC

Investigating - Our engineering team is investigating an issue with elevated error rates for Domains in our Cloud Control Panel, as well as our API. During this time, users may experience errors accessing the Networking > Domains section in our Cloud Control Panel, or while working with domains via API. We apologize for the inconvenience and will share an update once we have more information.
Feb 26, 20:13 UTC

Issues with Newly Created NYC3 Spaces

Resolved - Our Engineering team has confirmed resolution of this issue. All systems should now be operating correctly. If you continue to face any issues, please open a ticket with our Support team.
Feb 26, 21:01 UTC

Monitoring - Our Engineering team was able to identify the cause of the issue, and has implemented a fix. Corrective actions have been taken on Spaces that were facing this issue, so users should now be able to view and manage Spaces correctly, as well as upload without error. Newly created Spaces in NYC3 going forward should not see any issues. We are monitoring the situation and will post an update as soon as we confirm the situation is fully resolved.
Feb 26, 20:24 UTC

Investigating - Our Engineering team is investigating an issue with Spaces created in our NYC3 region. At this time, users who have recently created a Space in our NYC3 region (from approximately February 25, 15:00 UTC) may be experiencing issues viewing or managing the Space within the Cloud Control Panel, as well as experience issues when attempting to upload to the Space via API/CLI. Our team is working to determine the root cause. We apologize for the inconvenience and will share an update once we have more information.
Feb 26, 19:45 UTC

AMS3 Spaces Connection Issues

Resolved - Our Engineering team has resolved the issue with increased errors and connection issues for Spaces in our AMS3 region. All services should now be operating normally. If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.
Feb 26, 19:50 UTC

Monitoring - Our Engineering team has implemented a fix to resolve the issue with increased errors and connection issues for Spaces in our AMS3 region and is monitoring the situation. We will post an update as soon as the issue is fully resolved.
Feb 26, 18:24 UTC

Identified - Our Engineering team has identified the cause of the issue with increased errors and connections issues for Spaces in our AMS3 region, and is actively working on a fix. We will post an update as soon as additional information is available.
Feb 26, 14:39 UTC

Investigating - Our Engineering team is investigating an issue with an increased rate of errors and connection issues to objects in Spaces in our AMS3 region. During this time users may experience issues accessing objects, or see errors being returned while trying to connect. We apologize for the inconvenience and will share an update once we have more information.
Feb 26, 14:06 UTC

Cloud Control Panel Connectivity

Resolved - Our Engineering team has resolved the issue with our Cloud Control Panel, and all systems should now be operating normally. If you continue to experience issues, please open a ticket with our support team. We apologize for any inconvenience.
Feb 26, 18:56 UTC

Monitoring - Our Engineering team has identified the issue and implemented a fix to resolve the issues with the Cloud Control Panel and API. Users should be able to navigate the Cloud Control Panel correctly at this time, and we are currently monitoring the situation. We will post an update as soon as the issue is fully resolved.
Feb 26, 18:29 UTC

Investigating - Our Engineering team is investigating an issue with our Cloud Control Panel. During this time, users may experience issues accessing and using components of the Cloud Control Panel. We apologize for the inconvenience and will share an update once we have more information.
Feb 26, 18:18 UTC

SGP1 Networking Maintenance

Completed - The scheduled maintenance has been completed.
Feb 26, 18:00 UTC

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Feb 26, 16:00 UTC

Scheduled - During the above window, the networking team will be upgrading core networking equipment to improve performance and reliability for our SGP1 location.

Expected Impact:

There will be several minutes of packet loss while we shift traffic between redundant devices. There may be additional event processing delays, as well as Cloud Panel and API interruptions during this period. We will endeavor to keep this to a minimum for the duration of the change.

If you have any questions or concerns, please reach out to us by opening up a ticket on your account.
Feb 26, 15:57 UTC

Issues with Kubernetes dashboard via Cloud Control Panel

Resolved - Our engineering team has confirmed resolution of the issue impacting the accessibility of the Kubernetes dashboard via Control Panel, and all services should be functioning normally at this time. If you notice any further problems, then please contact support for assistance. Thank you for your patience throughout this process, and we apologize for any inconvenience.
Feb 26, 13:32 UTC

Monitoring - Our engineering team has implemented a fix to resolve the issue with inaccessible of the Kubernetes dashboard via Control Panel and is monitoring the situation. We will post an update as soon as the issue is fully resolved.
Feb 26, 13:11 UTC

Identified - Our engineering team has identified the cause of the issue with the inaccessibility of the Kubernetes dashboard via Control Panel and is actively working on a fix. We will post an update as soon as additional information is available.
Feb 26, 12:57 UTC

Investigating - Our engineering team is investigating an issue with the inaccessibility of the Kubernetes dashboard. During this time, you may experience a problem while accessing the Kubernetes dashboard via our Cloud Control Panel, but doctl, kubectl, cluster creations and deletions succeed. We apologize for the inconvenience and will share an update once we have more information.
Feb 26, 11:18 UTC

NYC1 Region Network Maintenance

Completed - The scheduled maintenance has been completed.
Feb 26, 10:00 UTC

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Feb 26, 05:00 UTC

Scheduled - The Networking team will be making changes to our core networking equipment to improve performance and reliability for our NYC1 location.

Expected Impact:

There will be several minutes of packet loss while we shift traffic between redundant devices. There may be additional event processing delays, as well as Cloud Panel and API interruptions during this period. We will endeavor to keep this to a minimum for the duration of the change.

If you have any questions or concerns, please create a support ticket.
Feb 26, 04:10 UTC

Feb 25, 2020

Cloud Control Panel Not Loading

Resolved - Our engineering team has resolved the issue with with the Cloud control panel, and should now be operating normally. If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.
Feb 25, 21:31 UTC

Monitoring - Our engineering team is currently monitoring an issue impacting the Cloud control panel. During this time, users may experience errors or blank pages after logging in to the Cloud control panel. We apologize for the inconvenience and will share an update once we are certain that the issue is fully resolved.
Feb 25, 20:25 UTC

CDN Configuration Change Failures for Spaces

Resolved - Between 21:26 - 21:33 UTC, our Engineering team observed a spike in failures when applying CDN configuration changes to Spaces in all regions. Examples of user impact include enabling or disabling of the Spaces CDN feature, as well as applying changes to existing Spaces CDNs. Our systems have gracefully recovered from this impact, and we would advise users who applied CDN changes during the above time window to review and re-apply any desired changes to their CDN configuration.
Feb 25, 21:26 UTC

Networking issue impacting multiple services

Resolved - Our Engineering team has resolved the issue with networking. Services should now be operating normally. If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.
Feb 25, 11:14 UTC

Monitoring - Our Engineering team has implemented a fix to resolve the issue with networking and is monitoring the situation. We will post an update as soon as the issue is fully resolved.
Feb 25, 09:53 UTC

Investigating - Our Engineering team is investigating an issue with networking which is impacting multiple services. During this time, you may experience slow response from services. We apologize for the inconvenience and will share an update once we have more information.
Feb 25, 05:47 UTC

Feb 24, 2020

No incidents reported.

Feb 23, 2020

Products and Services in SGP1 can be unreachable

Resolved - Our Engineering team has resolved the issue with services not reachable in SGP1. All the services should now be operating normally. If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.
Feb 23, 07:45 UTC

Monitoring - Our Engineering team has implemented a fix to resolve the issue with services not reachable in SGP1 and is monitoring the situation. We will post an update as soon as the issue is fully resolved.
Feb 23, 07:09 UTC

Feb 22, 2020

SGP1 Networking Incident

Resolved - Our engineering team has confirmed resolution of the issue impacting network traffic in SGP1, and there should be no lingering or residual impact related to this incident. If you experience any problems going forward then please open a ticket with support. We apologize for any inconvenience, and thank you for your patience throughout this process.
Feb 22, 15:55 UTC

Monitoring - Our engineering team has identified an issue that impacted network traffic in SGP1 between 14:45 and 15:00 UTC today. During this window, users may have experienced a brief period of network slowness or packet loss for traffic coming in or out of the SGP1 region. We have applied a fix and are monitoring the situation closely at this time. We will share an update as soon as we believe the matter is fully resolved.
Feb 22, 15:20 UTC

Feb 21, 2020

Multiple Services Impacted due to Missed Actions

Resolved - Our engineering team has resolved the aforementioned issues impacting the following services:

Support Center
Cloud Firewalls
Tags
Managed Databases
Load Balances
VPC
Custom Images
Projects
Referral Codes

There is no ongoing impact expected for these services listed above, and all actions should have been reconciled. Remaining areas of impact from this incident are outlined below:

User and Team Emails: Some email notifications directed towards users during the incident window on February 20 from 15:40 UTC until approximately 20:50 UTC were not sent. This would include email notifications related to Two-Factor Authentication, Secure Sign-In, Password Reset, New Droplet Creation/Password, Team invitation emails, and Monitoring alerts configured in the Cloud Control Panel. Due to the nature of this incident these emails are not able to be resent, and if you are missing any such notifications then you may attempt to re-run the initial action at this time(such as issuing a new password reset request).

Activity History: Users will not be able to see activity items on the Security page within our Cloud Control Panel for events during the affected window. Due to the nature of this incident, these actions cannot be backfilled.

If you continue to experience any problems related to these items, then please open a ticket with our support team. We apologize for any inconvenience, and we appreciate your patience throughout this process.
Feb 21, 20:52 UTC

Update - We've identified impact to multiple services, please refer to https://status.digitalocean.com/incidents/q5nr333lpzhs for the complete details.
Feb 21, 01:32 UTC

Identified - Our Engineering team has identified an issue impacting multiple services across our platform. Impact from this issue started occurring at 15:40 UTC. Around 20:50 UTC, new impact from the issue was stopped due to changes made by our Engineering Team. Our team is currently in the process of mitigating impact from this issue.

We are able to remediate some of the impacts from this issue, but users will need to retry some actions to ensure they are in place. While we continue to identify which services users need to take direction action on, we are providing a list of services impacted, along with a short description of impact to users below:

Support Center: Users who signed up for DigitalOcean within the timeframe noted above may experience an error when attempting to access the support portal or submit tickets at https://cloudsupport.digitalocean.com/s/. Existing users should not be affected, and may continue to use the support portal normally.

To work around any issues with the support portal, users may submit tickets via our contact form here: https://www.digitalocean.com/company/contact/#support

Cloud Firewalls: Users who made Cloud Firewall changes during the affected window may experience failures or delays. We recommend double-checking your Cloud Firewalls and re-applying rules as needed.

Tags: As per our previous post here: https://status.digitalocean.com/incidents/10drvkrcrhp0, Tags applied to Droplets at creation time were not applied during the affected window.

Monitoring: Monitoring alerts set up on user services may not have been sent, or may be delayed during the affected window.

Activity History: Users will not be able to see activity items on the Security page within our Cloud Control Panel for events during the affected window.

Managed Databases: Account or configuration changes for Managed Databases may have failed to be processed, or may be reverted to a prior state during the affected window.

Load Balancers: Automatic addition of tagged target droplets to a tagged Load Balancer will be delayed or fail.

Cloud Control Panel Notifications: Pop-up notifications for events like Droplet creation, deletion, etc., as well as notifications related to Projects, such as moving resources, were not displayed in our Cloud Control Panel during the affected window.

VPC: A small subset of VPC Limited Availability users may have a stale view of resources in their VPC when reading from the Cloud Control Panel or API during the affected window.

Custom Images: Users who created Custom Images during the affected window would have observed longer than expected delays before upload completion.

User Emails: Emails to users during the affected window related to Two-Factor Authentication and Secure Sign-In were not sent to users.

User Sign-Ins: Users that converted their personal DigitalOcean account to a DigitalOcean Team account during the affected window would not have their Two-Factor Authentication method moved to their new user, resulting in being unable to sign-in if the user logged out.

DigitalOcean Teams: Users may not have been able to join teams during the affected window, due to team invite emails not being sent. Users would have been able to join teams via the invite link in their Cloud Control Panel.

Projects: During the affected window, users may have experienced some resources not displaying in any projects. These resources would have still been available via other pages in the Cloud Control Panel. Project activity history during the affected window is also missing.

Referral Codes: Users who signed up for DigitalOcean during the affected window did not have their referral codes generated correctly.
Feb 21, 01:16 UTC

Event Processing Errors & Delays in BLR1

Resolved - Our Engineering team has resolved the issue with event processing delays and timeouts in the BLR1 region. Event processing for this region is expected to be operating normally at this time. If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.
Feb 21, 00:31 UTC

Monitoring - Our Engineering team has implemented a fix to resolve the event processing delays and timeouts in the BLR1 region. Events have been enabled for this region and our team will continue to monitor the situation. We will post an update as soon as the issue is fully resolved.
Feb 20, 23:52 UTC

Update - Our Engineering team has identified the issue causing event processing delays and timeouts in the BLR1 region and is actively working on a fix. As part of remediation, we have temporarily disabled events in this region and will enable events and provide additional information as soon as possible.
Feb 20, 23:31 UTC

Update - Our Engineering team has further identified the scope of impact for this incident, as event processing delays and timeouts have been observed in the BLR1 region. We are actively working on a fix, and will post an update as soon as additional information is available.
Feb 20, 22:21 UTC

Identified - Our Engineering team has identified the cause of the intermittent issues with Snapshot transfers to and from our BLR1 region and is actively working on a fix. We will post an update as soon as additional information is available.
Feb 20, 21:50 UTC

Investigating - Our Engineering team is investigating intermittent issues with Snapshot transfers to and from our BLR1 region. During this time, users may experience errors or delays while transferring Snapshots to BLR1, or from BLR1 to other regions. We apologize for the inconvenience and will share an update once we have more information.
Feb 20, 20:59 UTC

Feb 20, 2020

Issues with Droplet Tagging

Resolved - Our Engineering team has resolved the issue with applying tags to Droplets at creation time. Droplets created with tags will now reflect those tags correctly. However, during the course of investigation, the impact of this issue was determined to be part of a broader incident, impacting multiple services. We will be posting a new status to provide information and updates encompassing the impact to those multiple services.
Feb 20, 22:41 UTC

Identified - Our Engineering team has identified the cause of the issue with applying tags to Droplets at creation times and is actively working on a fix. We will post an update as soon as additional information is available.
Feb 20, 20:28 UTC

Investigating - Our Engineering team is currently investigating an issue with tags not being applied to Droplets during creation in all regions. At this time, users may experience issues with selected tags not being applied during Droplet creation, via both our Cloud Control Panel and API. Existing tagged Droplets and adding additional tags to existing Droplets does not appear to be impacted at this time. We apologize for the inconvenience and will share an update once we have more information.
Feb 20, 19:17 UTC

Intermittent Event Processing Issue in SGP1

Resolved - Our Engineering team has confirmed resolution of the issue affecting event processing in the SGP1 region. All systems should be operating normally at this time. If you continue to experience any problems, then please open a ticket with our support team. We apologize for any inconvenience, and we appreciate your patience throughout this process.
Feb 20, 08:10 UTC

Monitoring - Our Engineering team has implemented a fix to resolve the issue impacting event processing in SGP1, and is monitoring the situation at this time. We will post an update when we are confident that the issue is fully resolved.
Feb 20, 07:06 UTC

Investigating - Our Engineering team is investigating an intermittent issue with event processing in the SGP1 region. During this time, users may experience errors or delays while working with resources in the SGP1 region, such as creating or destroying Droplets. We apologize for the inconvenience and will share an update once we have more information.
Feb 20, 06:51 UTC

Feb 19, 2020

Errors with Control Panel Billing Page

Resolved - Our engineering team has confirmed resolution of the issue impacting the Billing page in our Cloud control panel. All systems should be operating normally at this time. If you continue to experience any problems, then please open a ticket with our support team. We apologize for any inconvenience, and we appreciate your patience throughout this process.
Feb 19, 20:53 UTC

Monitoring - Our engineering team has implemented a fix for the issue impacting the Billing page in our Cloud control panel, and are currently monitoring the situation. We will post an update once we feel the situation is fully resolved.
Feb 19, 20:36 UTC

Investigating - Our engineering team is currently investigating an issue impacting the Billing page in our Cloud control panel. During this time, users may experience errors while accessing the Billing page of the control panel. We apologize for the inconvenience and will share an update once we have more information.
Feb 19, 20:23 UTC

Intermittent Event Processing Issue in SGP1

Resolved - Our engineering team has confirmed resolution of the issue affecting event processing in the SGP1 region. All systems should be operating normally at this time. If you continue to experience any problems, then please open a ticket with our support team. We apologize for any inconvenience, and we appreciate your patience throughout this process.
Feb 19, 20:40 UTC

Monitoring - Our engineering team has implemented a fix to resolve the issue impacting event processing in SGP1, and is monitoring the situation at this time. We will post an update when we are confident that the issue is fully resolved.
Feb 19, 18:37 UTC

Identified - Our engineering team has identified the cause of the intermittent issue affecting event processing in the SGP1 region, and is actively working on a fix. We will post an update as soon as more information is available.
Feb 19, 16:50 UTC

Investigating - Our engineering team is investigating an intermittent issue with event processing in the SGP1 region. During this time, users may experience errors or delays while working with resources in the SGP1 region, such as creating or destroying Droplets. We apologize for the inconvenience and will share an update once we have more information.
Feb 19, 16:03 UTC

Block Storage, Cloud Firewall, and Event Processing Issues

Resolved - Our Engineering team has resolved the issue with multiple services in our AMS2 and AMS3 regions, and all services should now be operating normally. If you continue to experience any issues, please open a ticket with our support team. We apologize for any inconvenience and appreciate your patience.
Feb 19, 20:32 UTC

Monitoring - Our Engineering team has implemented a fix to resolve the issues in our AMS2 and AMS3 regions, and is monitoring the situation at this time. Cloud Firewalls that previously did not apply are now being retried and applied correctly. We will post an update when we are confident that the issue is fully resolved.
Feb 19, 18:55 UTC

Identified - Our engineering team has identified an issue with Block Storage, Cloud Firewalls, and Event Processing in our AMS2 and AMS3 regions, and is working on a fix. During this time, users may experience errors interacting with or accessing Volumes (Block Storage) in AMS3. Cloud Firewalls are currently failing to apply to Droplets in AMS2 and AMS3. These firewalls will be retried, so users will see those applied once the issue has been resolved. Lastly, users may experience issues or delays while processing events in AMS3, such as creating or destroying Droplets. We apologize for the inconvenience and will share an update once we have more information.
Feb 19, 18:36 UTC

Load Balancers and Let's Encrypt Certificates Maintenance

Completed - The scheduled maintenance has now been completed, and all services should be operating normally. If you continue to experience any issues, please open a Support ticket from within your account.
Feb 19, 16:25 UTC

Update - Scheduled maintenance is still in progress, but we encountered a temporary issue that resulted in users being unable to load the Networking page in our Cloud Control Panel. Our Engineers were able to quickly resolve this, and will continue to monitor this throughout the maintenance. We apologize for any inconvenience caused.
Feb 19, 16:10 UTC

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Feb 19, 15:31 UTC

Scheduled - Our Engineering team will be performing scheduled maintenance on Load Balancers and Let's Encrypt Certificates offered within DigitalOcean.

During this time, users will not be able to create, update, or delete Load Balancers or Let's Encrypt Certificates through either the Cloud Control Panel, or the API. Existing Load Balancers and Certificates will continue to operate normally.
Feb 19, 14:28 UTC

Feb 18, 2020

No incidents reported.

Feb 17, 2020

No incidents reported.

Feb 16, 2020

No incidents reported.

Feb 15, 2020

Droplet Creation Delays.

Resolved - Our engineering team has resolved the issue with Droplet creation delays. If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.
Feb 15, 02:05 UTC

Monitoring - Our Engineering team has implemented a fix for the Droplet creation delays, and functionality should be fully restored at this time. Our team is now monitoring the situation for any further issues. We apologize for the inconvenience and will share an update once we have confirmed that the issue is resolved.
Feb 15, 01:18 UTC

Identified - Our engineering team has identified the cause of Droplet creation delays in all regions and is actively working on a fix. During this time, you may experience a delay when creating Droplets in the control panel or via the API. We apologize for the inconvenience and will share an update once we have more information.
Feb 14, 23:48 UTC

Feb 14, 2020

Slow Requests for Spaces in SGP1

Resolved - Our Engineering team has resolved the issue with increased error rates and latency with Spaces in the SGP1 region. Spaces should now be operating normally. If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience this may have caused.
Feb 14, 02:32 UTC

Monitoring - Our Engineering team has implemented a fix to resolve the increased error rates and latency for Spaces in the SGP1 region and is monitoring the situation. We will post an update as soon as the issue is fully resolved.
Feb 14, 01:59 UTC

Identified - Our Engineering team has identified the cause of the increased error rates and latency with Spaces in our SGP1 region. We will post an update as soon as additional information is available.
Feb 14, 01:36 UTC

Investigating - Our Engineering team is investigating an issue with increased error rates and latency with Spaces in the SGP1 region. We apologize for the inconvenience and will share an update once we have more information.
Feb 14, 01:16 UTC

Feb 13, 2020

PayPal payment processing failures

Resolved - Our Engineering team has resolved the issue with processing payments via PayPal on our platform. Services should now be operating normally. If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.
Feb 13, 17:01 UTC

Monitoring - Our Engineering team has identified the cause of the issue with processing payments via PayPal on our platform, and has implemented a fix. We are currently monitoring the situation and will post an update as soon as the issue is fully resolved.
Feb 13, 14:33 UTC

Investigating - Our Engineering team is investigating an issue with payments using PayPal. During this time, you may experience timeouts/failures when trying to use PayPal. We apologize for the inconvenience and will share an update once we have more information.
Feb 13, 10:51 UTC

Event Processing Delays/Errors via API or WebUI

Resolved - Our Engineering team has resolved the issue affecting delays and timeouts with event processing across all regions. Services should now be operating normally. If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.
Feb 13, 10:55 UTC

Monitoring - Our Engineering team has identified the cause of issues affecting delays and timeouts with event processing across all regions, and functionality should be fully restored at this time. Our team is now monitoring the situation for any further issues. We apologize for the inconvenience and will share an update once we have confirmed that the issue is resolved.
Feb 13, 10:02 UTC

Investigating - Our Engineering team is investigating an issue affecting delays and timeouts with event processing across all regions. During this time, users may experience intermittent errors or delays while working with resources, such as creating or destroying Droplets. We apologize for the inconvenience and will share an update once we have more information.
Feb 13, 09:26 UTC