Learn more about these different git repos.
Other Git URLs
== Request ==
Please create a new Koji channel for building Java packages which would exclude ARM builders, or implement an equivalent solution. My goal is to be able to build noarch Java packages skipping any ARM builders. Using custom koji command is not a problem for me.
== Justification ==
The great majority of Java packages are noarch. As such they are most often pisked up by ARM builders.
Java performance on ARM is very poor. The situation got even worse after JIT for ARM was disabled in JVM. ARM machines are often slower by one to three orders of magnitude than x86_64. Some tests which execute in a few seconds on x86_64 can take over half an hour on ARM.
Current state of Java on ARM combined with Koji configuration makes it difficult to maintain Java packages in Fedora. A substantial amount of human resources is wasted on waiting for builds to complete, debugging tests which fail only on ARM, keeping track on builds and resubmitting them. That resources could be utilised better and I think that justifies my request.
== Alternative solutions ==
I have tried differnt alternative solutions since ARM became a primary architecture and none of them is good enough.
One ad-hoc solution is submitting numerous dummy koji tasks for ARM builders until their capacity is exhausted, then submitting Java package build and canceling ARM tasks. This is obviously a hack and should be avoided.
If Koji configuration is not improved the only remaining solution will be converting Java packages from noarch to arch-specific and adding ExcludeArch. I still hope that ARM situation will eventually improve, so I would like to avoid doing drastic things like that, if possible.
I second this request as from Eclipse POV it's a nightmare to wait for 20 h when x86 needs 30 min. It's a clean showstopper for doing any serious work on Fedora if one starts build and has to wait for next day before seeing results. Even though Eclipse itself is an arch package some solution needs to be found as this is preventing any serious work on Fedora.
This isn't such a terrible idea, given that some ARM boxes are often much less powerful than our faster x86 builders. However, the ARM-32 JIT is (hopefully) about to get fixed, and ARM boxes are due to get quite a lot faster.
Having said that, this is really an infrastructure bug: the Fedora build system really should have some notion of priority when deciding where to place jobs
I'd be for a more intelligent build system that knows to allocate noarch builds to the fastest builders, rather than an arch-specific inclusion. That should really have been in place before making ARM32 a primary architecture. You'd have the same problem if some other archs were primary as well. Maybe look at how Debian deals with this?
the Fedora build system really should have some notion of priority when deciding where to place jobs
It does.
That's why it allocates noarch builds in priority to the least loaded machines, which turn out to be ARM most of the time as there are many more ARM builders.
Replying to [comment:5 ahughes]:
I'm not even dreaming about improving Koji, which would require some effort. All I ask for is a few lines of configuration to allow me to tell Koji to exclude ARM builders when I am building Java packages.
Debian developers can build binary packages on their own systems and upload them to FTP.
I'd suggest discussing options on devel and/or buildsys lists first, and once some consensus is reached, then work to implement it (and not before).
Could we please get some reply to this request?
there is no specific/actionable request here, just a criticism about the current setup being slower and not-ideal for java-related builds.
See comment 8 about constructive ways forward (e.g. discuss on mailing lists first).
Replying to [comment:11 rdieter]:
There is a clear request for creating a Koji channel, which is in domain of release engineering. Other packages (like kernel) have their channel and I don't see a reason a channel couldn't be provided for Java packages.
I would like to get an authoritative answer from a member of release engineering team before trying other means of resolving this issue.
+1 for this ticket.
A local build of wildfly application server takes a few minutes, whereas on koji ARM builder it takes about 3 hours. This makes it impossible to rapidly fix build issues.
wildfly
Replying to [comment:4 aph]:
I talked to jvanek, he thinks JIT could be turned on now for the next update, probably happening in the first half of May.
I believe the JIT will help '''somewhat'''. What I don't believe is that it will magically bridge the whole gap. PPC doesn't have JIT and it's still much much faster then ARM. For comparison, eclipse build times: * x86_64 (JIT): 30 minutes * ppc64le (no JIT): 3 hours * armv7hl (no JIT): 20 hours
So even if JIT sped it up tenfold, it would still be 2 hours vs 30 minutes. Sure, most packages don't take '''that''' much time, but try maintaining 200+ packages like this.
Another point is that quite a number of packages have testsuites that are timed. On arm hardware these tests often fail (sometimes due to race conditions in tests, sometimes because the code is really slow...). In either case we lose a lot of time debugging and fixing testsuites (which are not packaged or executed) instead of actually improving Fedora. I'll also note that a lot of testsuites are "write-once-read-never" deals and '''are''' hacky, but they are not meant to be perfect.
I don't see reason why we should inflict this on people working on Fedora when we clearly have a better way to deal with this.
I made a change to the capacites of the builders last night.all the x86 builders have higher capacities than the arm builders. what this should mean is that most of the time an x86 builder will take noarch only builds. but they will go to arm builders in times of high build loads.
:+1:
Could the koji balancer just parse for any "java" keywords in SRPM/spec file?
I am going to close this as fixed. seems the ssds speed things up enough
Login to comment on this ticket.