Degraded Performance, Identifying - We are aware that many are experiencing degraded service for job picking. This appears to be inconsistent as some jobs are still being picked in a timely manner. However, some jobs are remaining in a submitted state without being initialized. We are working to identify the issue and in the mean time are resubmitting jobs on behalf of users. All other non-job related systems are operating normally. (18:20 UTC — Oct 28)
Mitigated, Observing - We have identified a possible service connection bug that was affecting job message submissions. We have cleared the backlog of connections and submitted jobs. Testing is showing that jobs are being picked up in a timely manner. We will continue to monitor the system to ensure that jobs are being picked up as expected. (19:27 UTC — Oct 28)
Operational, Observing - All systems are operational and jobs are being picked up in a timely manner. We will continue to monitor the system to ensure continued performance. (19:33 UTC — Oct 28)
Last updated: October 28, 2025 at 1:39 PM UTC