Resolved -
CloudBees has resolved the issues affecting Workflow Execution and Feature Management flags. The incident was related to an automated process causing a resource bottleneck and preventing legitimate Git webhooks from being handled in a timely manner. The aforementioned automated process has been stopped and protections have been put in place to prevent it from being re-enabled. The backlog of workflows have processed successfully as of 22:15 UTC. All systems are now fully operational and working as expected.
Dec 8, 22:51 UTC
Identified -
CloudBees has identified the root cause of the delayed execution and is working to address it.
Dec 8, 21:58 UTC
Update -
As we continue to troubleshoot this issue, we have determined that creating and updating existing flags is delayed. The scope of impact has been isolated and CloudBees is attempting to execution some remediation steps to address the impact.
Dec 8, 19:36 UTC
Update -
CloudBees has received some reports indicating manually executed runs are also intermittently failing to start. Investigation in to root cause is still ongoing and additional information will be shared as it becomes available.
Dec 8, 17:32 UTC
Investigating -
CloudBees has discovered an issue where workflows initiated by Git commits are not currently executing as expected since ~15:10 UTC. Investigation in to the root cause is currently underway.
Dec 8, 16:12 UTC