Yesterday there was a mini mass-rebuild in Rawhide. ~500 packages were rebuild in just couple of hours, which generated 2000+ tasks in CI (rpminspect+rpmdeplint+installability+dist-git, and some compose-ci and eln tasks * 500).

This is just way too much for such a short period of time. As expected, all 100 executors submitted requests to Testing Farm and blocked. But because the load was too much, the times for delivering the test results went up dramatically (hours). Other tasks stayed queued in Jenkins and pilled up.

The long running tasks in combination with the slow storage killed Jenkins (known issue with logging output -- longer the task is running, more output it produces).

There are two problems here:

• waiting for Testing Farm and blocking executors
  • https://pagure.io/fedora-ci/general/issue/173
  • PR (in progress): https://github.com/fedora-ci/dist-git-pipeline/pull/29
• quality of service: single user can submit a ton of builds and then others would need to wait hours/days in queue before their requests would be processed; kinda unacceptable...

I think for the QoS part we should prioritize what will be tested and when, based on the number of builds that a particular user submitted in the last hour or so. Requests from users who only need a few builds tested would be placed at the beginning of the queue, and requests from users who do their own mini mass-rebuilds would go at the end of the queue.