Learn more about these different git repos.
Other Git URLs
Koji currently has the ability to use persistent/static builders provisioned by the admin or dynamic builders through libvirt using kojivmd. I propose that we add a third method: cloud-provisioned dynamic builders, similar to what Fedora COPR does now.
kojivmd
This would allow us to "burst up" capacity for things like mass builds, but more importantly, give us the ability to have more resources for specific build channels that need more powerful builders while not costing an arm and a leg. :sweat_smile:
The context for this request is that Fedora Asahi SIG and Fedora KDE SIG wind up having to work with packages that under the ordinary builders take whole days to build (kernel, firefox, chromium, webkitgtk, qt5-qtwebengine, qt6-qtwebengine, etc.). With this enhancement, we could avoid issues like fedora-infrastructure#10796.
kernel
firefox
chromium
webkitgtk
qt5-qtwebengine
qt6-qtwebengine
This would also be useful for cloud-based deployments of Koji by third-parties, as it's possible their build loads would be significantly lower than a distribution and would benefit from dynamically provisioning builders as needed.
There is an equivalent pair of tickets discussing this for COPR: gh#fedora-copr/copr#2241, gh#fedora-copr/copr#2925.
cc: @davdunc @dcavalca @marcan @catanzaro @jgrulich @praiskup @kevin
This would have also probably helped @adamwill with rhbz#2241690 too.
Does it make sense to solve it on koji level? There are few approaches which could be considered here: 1) Long-term solution is part of the solution. 2) External daemon monitoring queue and triggerring/monitoring/tearing down new builders on demand. There was and attempt to do this, but repo has almost zero code. Easiest way would be to have builders preestablished (created certs/krb credentials, having these builders in koji). If monitored queue is longer than allowed (e.g. based on channel/arch bins), some of these builders would be started. If queue is shorter, builders can be disabled + waited until all tasks ended there and then they can be stopped. The biggest issue (why I woulnd't do it in koji) are the backends. It would involve some (plugin-based) backends for AWS/openshift/openstack, ...) I believe that copr's resalloc could do most of this.
Metadata Update from @tkopecek: - Custom field Size adjusted to None
Koji currently has the ability to use persistent/static builders provisioned by the admin or dynamic builders through libvirt using kojivmd
Kojivmd uses libvirt, yes. However, this does not amount to a "dynamic builder". It is more of a higher-isolation build environment.
Adding new builders in Koji is relatively high impact. Each host has special privileges within the system, and filling the host list with a bunch of transient entries will cause problems.
This is one of the main reasons that COPR doesn't use Koji as a backend.
This is a laudable goal, but not an easy one. Doing it right within Koji would likely require a huge amount of invasive work. Also note this is not a new suggestion. If this weren't very difficult, it would have been done years ago.
That said, an external system like Tomas suggests could be feasible, though perhaps less than ideal.
Here's the thing: with the "powerful builders" enabled on the Fedora Asahi kernel COPR project, kernel builds now take around an hour to complete (an hour for aarch64, ~40 minutes for x86_64). That is a fraction of the time it takes on Fedora's Koji instance.
The amount of time burned on things like WebKitGTK and Chromium builds due to not having the ability to have enough good builders is nuts. :frowning:
Login to comment on this ticket.