Degraded Performance On Storage Services

Operational, Monitoring

Failure rates have returned to 0%. (15:31 UTC — Sep 9)

Operational, Monitoring

CPU usage on the affected server is back to normal levels. Failure rates of the services continue to decrease. Most users will have success using the platform. (15:07 UTC — Sep 9)

Degraded, Mitigating

Services have been restarted to clear the backlog of long-running requests. Failure rates are decreasing again. (14:53 UTC — Sep 9)

Degraded, Mitigating

The backlog of long-running requests has produced timeouts which caused the failure rates to increase again. (14:48 UTC — Sep 9)

Operational, Monitoring

Failure rates have decreased as a result of the mitigation actions taken. We continue to monitor the situation. (14:44 UTC — Sep 9)

Degraded, Mitigating

We are aware of an issue impacting several platform services. A routine process was started to reclaim storage space made available through user database archiving and deleting. This process caused an unexpected CPU spike on a system server which did not resolve quickly. The issue was identified quickly and the process was stopped. We are currently working to decrease the load on the affected system database. (14:19 UTC — Sep 9)