#2772 Change proposal: Drop i686 builds of jdk8,11,17 and latest (18) rpms from f37 onwards
Closed: Accepted 6 months ago by churchyard. Opened 8 months ago by bcotton.

java-1.8.0-openjdk, java-11-openjdk, java-17-openjdk and java-latest-openjdk packages will no longer build i686 subpackages


+1

-1 as-is. The change does not describe the impact on other packages and I feel like the change owner is unaware of the scope. This will potentially lead to an explosive recursive exponential ExcludeArch growth and we don't know if the impact-set won't leak into the multlib-set we actually still want to ship.

I imagine quite a lot of packages (see e.g. protobuf) will need to stop building their java subpackages on i686 which requires more complex changes to the spec file than adding one ExcludeArch line. The change owner does not appear to commit to fixing all this mess themselves. In fact, this kinda looks like a "drop it to rawhide it and go away" kind of change.

I am in favor of the general idea, I just don't feel it is well prepared. Let's:

  • determine the actual impact
  • verify the impact is manageable and commit to help mitigate it
  • remove the entire i686 Java tree in one atomic side tag update, or at least provide an actual contingency deadline and a way to revert this change if it needs to happen in the middle of the slow recursive process

Note: The other i686 change proposes to start removing the packages with the leaves. While there is a discussion on whether or not this will add unnecessary garbage to our spec files, it is safe -- distribution-wise. This change, however, proposes to rip off a non-leaf package and that can break the distro quite a lot (and what's worse, slowly -- we may be still dealing with the impact after 5 mass rebuilds, as the recursion level grows deeper and deeper).

-1, for now.

As far as I can tell, the change owners are underestimating the impact of this change. The page was updated to list all affected non-noarch Java packages, which would all need to be adapted manually. Additionally, all noarch Java packages will also need to be adapted (so, all Java packages).

Eek, I think I underestimated the impact here. I'll switch my vote to -1, given the above explanations.

Metadata Update from @sgallagh:
- Issue tagged with: meeting

8 months ago

AGREED: FESCo awaits more feedback from the Change owners (+7, 0, -0) (sgallagh, 18:21:15)

Metadata Update from @sgallagh:
- Issue untagged with: meeting

8 months ago

TY! Will bring it up in JDK team.

@jvanek: my comment from yesterday:
Querying x86_64 gives an overly pessimistic view, because there are less packages for i686. And recursive query is not useful, because we don't want to drop all packages that require java. For some packages, we'll just disable the feature that requires java for i686, and the propagation stops there. For other packages, we might want to completely disable the package/subpackage on i686, and in that case the check needs to be recursed. But this needs to be checked for each package individually, to see what makes the most sense.

$ parallel repoquery -q --qf='%{name}' --arch=i686 --whatrequires {} ::: java-headless java java-devel
csound-java
DecodeIR
link-grammar-java
octave
openmpi-java-devel
plplot-java
qdbm-java
R-java-devel
will-crash

$ repoquery -q --qf='%{name}' --arch=src --repo=fedora-source --whatrequires java-headless                                     
R
cocoalib
enjarify

$ repoquery -q --qf='%{name}' --arch=src --repo=fedora-source --whatrequires java         

$ repoquery -q --qf='%{name}' --arch=src --repo=fedora-source --whatrequires java-devel   
BareBonesBrowserLaunch
CardManager
OpenStego
Singular
ant
antlr
apron
arduino-listSerialPortsC
astyle
automake
beansbinding
bigloo
bolzplatz2006
brazil
brltty
cambozola
capstone
ceph
collectd
colossus
console-image-viewer
cortado
cryptlib
csound
cvc4
diffoscope
ditaa
dl_poly
erlang
erlang-corba
fernflower
filedrop
flute
freecol
freerouting
freewrl
frysk
gdal
gdl
genders
gnulib
graphviz
hdf
hdf5
healpix
hyperestraier
imagej
jFormatString
jargs
java-gnome
java-mersenne-twister
java-runtime-decompiler
javahelp2
jblas
jcip-annotations
jcommon-serializer
jcuber
jdepend
jericho-html
jgrapht
jigawatts
jmol
jorbis
jpanoramamaker
jpcap
jpype
jssc
kernel-tools
laf-plugin
libbase
libbluray
libfonts
libformula
libgda
libidn
liblayout
libloader
libphidget
libreoffice
librepository
libserializer
libsvm
libvirt-java
libwebp
link-grammar
mapserver
maven-verifier-plugin
mecab-java
miglayout
mp
mysql-connector-java
naga
octave
opencv
openjdk-asmtools
openmpi
openni
pcl
pentaho-libxml
pentaho-reporting-flow-engine
pl
plexus-component-api
plplot
portmidi
postgresql-jdbc
ppl
procyon
proguard
python-javabridge
python-jep
python-jnius
qdbm
rachota
rsyntaxtextarea
sblim-cim-client
sblim-cim-client2
scalpel
shaman
sphinx
swing-layout
systemtap
taggle
test-interface
tomcat
tomcat-native
tzdata
vecmath1.2
vtk
will-crash
writer2latex
xerces-j2
xjparse
xmltool
yecht
z3

That last list is the longest, but it needs to be filtered: we only care about packages which do archful builds and don't already exclude i686.

parallel repoquery -q --qf='%{name}' --arch=i686 --whatrequires {} ::: java-headless java java-devel

unrelated tip:

repoquery -q --qf='%{name}' --arch=i686 --whatrequires java-headless --whatrequires java --whatrequires java-devel

But what repo are you querying? Just the multilib stuff in x86_64? Try the koji i686 repo instead.

Thanx a lot!
Jsut note, that from those really only few build archfull (intersect with https://fedoraproject.org/wiki/Changes/Drop_i686_JDKs#list_of_non-noarch_java_packages.2C_transitive_to_chain_of_three )

On contrary, if they do they moreover 100% build also i686.

This is just 100 packages, where tehe safety bell is to drop i686. The transitive chain may be worse, but when I did that transitive check, it grown by 1 in the fourth level.

Thanx a lot. Will continue elaborating

One message from openjdk development: "java-11-openjdk & java-17-openjdk may still experience crashes, even as they stand now, and java-1.8.0-openjdk is still running as an
interpreter-only (Zero) build, with performance implications."

parallel repoquery -q --qf='%{name}' --arch=i686 --whatrequires {} ::: java-headless java java-devel

unrelated tip:

repoquery -q --qf='%{name}' --arch=i686 --whatrequires java-headless --whatrequires java --whatrequires java-devel

But what repo are you querying? Just the multilib stuff in x86_64? Try the koji i686 repo instead.

I think to make this work, you have to include --arch=src. So one have to run two queryes and interset... ut you guys know those tricks much better then me.

$ repoquery --repo=koji-{source,i386} --whatrequires java-headless --whatrequires java --whatrequires java-devel
...

$ repoquery -q --repo=koji-{source,i386} --whatrequires java-headless --whatrequires java --whatrequires java-devel | wc -l
849

$ repoquery -q --repo=koji-{source,i386} --whatrequires java-headless --whatrequires java --whatrequires java-devel | grep 'src$' | wc -l
140

$ repoquery -q --repo=koji-{source,i386} --whatrequires java-headless --whatrequires java --whatrequires java-devel | grep 'noarch$' | wc -l
665

$ repoquery -q --repo=koji-{source,i386} --whatrequires java-headless --whatrequires java --whatrequires java-devel | grep '86$' | wc -l
44

$ repoquery -q --repo=koji-{source,i386} --whatrequires java-headless --whatrequires java --whatrequires java-devel --recursive | wc -l
2133

Right, the recursive query is misleading, because it does not account for possibly dropped subpackges.

However, this also does not account for recursive build rerequires (a.src requires java, b.src requires a "binary" package built from a.src and so on).

Do we have sufficient feedback from the Change owners to begin the voting process?

Hello!
I'm afraid rcm is a bit reluctant to sign themselves to anything. It seems it is technically possible to stop building noarch pkgs on i686, but even if FESCO support that, they consider the solution as hacky. Also people around OpenJDK got little bit puzzled how first-glance simple change grown.
From what I see, except dropping the proposal, there are jsut two courses to do:
a) I will write a system wide change to stop building noarch on i686, but it is most likely bigger can of worms then "Drop i686 builds of jdk8,11,17 and latest (18) rpms from f37 onwards ", and I do not feel exactly enlightened enough in that topic keep discussion on reasonable levle
b) to adapt java-packages guidelines that i686 must be in ExcludeArches or workarounded to not use java/javac (there are such cases) and to all current java packages inject exclude arch i686 or contact maintainer to do so, otherwise he will face random failures (if theirs builder will be by bad luck i686)

For the (b), I'm not sleeping. I'm investigating what will go wrong if al java packages will suddenly case to exists on i686. It run over weekend, and I made two major mistakes: I forget to record the edges of the char, recorded just nodes, it ended in level4 of dependencies of dependencies of dependencies of dependencies various jdks and mavens... Second I forger to exlude noarchs from intermediate reuslts (as thy will not stop working at all, will jsut not build on i686) so there ended to much pkg (some 30000) from which about 4000 are built on i686. As there ended moreover whole perl stack , I need to rerun my queries, and track that chart better, who is the main culprit.

Any opinion on path a) xor b) is welcomed, any other possible ways to!
Note that there is also https://pagure.io/releng/issue/10686#comment-787222 now, but I do not want to bother RCM yet again.

Ok... my algorithm was right.. I can not exclude noarch from the parents set...

Hmm.. the culprint pulling whole system in is:

java-devel~is~req~by~automake

Will exclude this one

hm..another culprit is graphviz.. but will leave this round to finish already.

automake is noarch... that is baaad...

another noarch which is pulling half of the system is autoconf

Hello!
I'm afraid that automake and autoconf build-time depndece is a showstopepr to this. Any ideas how to continue?
if you wish, I can generate you the chart, but it is currently hige (14k nodes..s.. msotly caused by autoconf.. next generaraion would be without it, but not sre it is worthy.

Something like this is exactly what I was afraid of. I don't think it will be feasible to work around all (direct and indirect) Java dependencies in Fedora (which is also a much bigger scope than the original proposal) within the original timeframe. So what do you think about these steps instead:

  • keep building the JRE / JDK for i686 for now, even if it will run with very bad performance (which is basically irrelevant, since it should only ever be used during package builds)
  • start a broader effort for dropping the Java dependency on i686 from "important" packages like autoconf + automake (or at least, determine if that's even possible) - basically, by marking Java on i686 as "deprecated"
  • revisit dropping Java builds for i686 entirely when the impact will be smaller and more manageable

Yup. I moreover agree. I already contacted maintainers of autoconf and automake, but I do not belive there will be some immediate action possible, due to unforseen consequences.

I keep thinking. I had included also build-requires, which may be missleading.
If I exclude BR on noarch, then the set will be much smaller (will rerun).
Because if the BR of java is of is noarch pkg, it do not need to build on i686, and if there is no runtime dep (like autoconf/automake) on java then the whole sub-tree may disappear. wdyt?

Such noarch packages will still need to be modified to add ExclusiveArch: %{java_arches} so that they don't get built on i686 builders.

Well, yes... It is not an technical issue. But I must admit that I do not know where and how to let everybody know - some kind of packaging gudeline line I guess?

Such noarch packages will still need to be modified to add ExclusiveArch: %{java_arches} so that they don't get built on i686 builders.

Actually - ExcludeArch: %{non_java_arches} may be better, as any major arch shoudl be always supported by java.

I'm not sure why we keep updating this ticket with questions about the details of this Change. Shouldn't this happen on the mailing list instead?

Anyhoo, there are already Packaging Guidelines for this kind of scenario ("noarch packages with unported dependencies"), and applying them would look like this:

  • noarch package:
BuildArch: noarch
ExclusiveArch: %{java_arches} noarch
  • arch'd package:
ExclusiveArch: %{java_arches}

c.f. https://docs.fedoraproject.org/en-US/packaging-guidelines/#_noarch_with_unported_dependencies

Actually - ExcludeArch: %{non_java_arches} may be better, as any major arch shoudl be always supported by java.

If we ever add a new architecture, something like riscv64 maybe, will it support Java from the start? I doubt it. And this double-negative exclusion construct would also break for any third-party package builds for no-longer-supported-by-Fedora architectures.

I suggest changing the status of this change to InProgress and restarting the change process if/when a new proposal is ready.

Yes please, am changing to inProgress.

also sorry for misusing this thread, I di not know t is wrong.

Am still in work of best enumeration. Currently I think it is safe to exclude everyhitng what have BuildRequires(just BR!) on some java stack and builds as noarch. As such tool can happily ignore i686 iin built time, and still wil be availabel in runtime if needed. This is exactly the case autoconf and automake.

Not sure if it needs to be announced since the proposal was withdrawn, but tagging for that just in case. I'll open a new issue when/if the proposal is re-submitted.

Metadata Update from @bcotton:
- Issue tagged with: pending announcement

7 months ago

Hello! I had reworked the proposal's workflow and moved back to ChangeReadyForFesco

Metadata Update from @zbyszek:
- Issue untagged with: pending announcement

7 months ago

The new proposal is reasonable.

I think it'd make sense to start small with a subset of packages which have hundreds of dependencies. I.e. add the %java_arches macro, then handle graphviz, meson, and a few other important packages, and only then start filing bugs for what remains. Initially there'll be a lot a lot of packages in the list, but once those packages low in the stack are fixed, the number should go down significantly.

I'm sure that some details will need to be figured out, but I think the whole plan is worth attempting.

+1

--

BTW. I looked at graphviz: it requires java for graphviz-java subpackage. But nothing seems to Require or BuildRequire graphviz-java. So a simple archful exclude in graphviz should be enough in this case.

Can we/should we get the reworked proposal posted back to the devel list for input from the community?

Can we/should we get the reworked proposal posted back to the devel list for input from the community?

+1

I mailed devel on the old thread to relook at this revised change.

Hi! It seems there was no reply at all:(

What now? To get the trackig BZ, to commit new macro and to start spamming the main packages I need your consensus.

Tagged with meeting, so it won't get lost.

Metadata Update from @churchyard:
- Issue tagged with: meeting

6 months ago

+1 from me

But be prepared that the new plan will take time and might not make it in time for Fedora 37 release.

Yup. Am aware. Thanx a lot!

Looks good to me now, thank you for revising the proposal. +1

The only question I have is what

all noarch direct java, amven ant... dependencies will (todo, generate list as for java-17-openjdk as system jdk) get auto injected ExclusiveArch: %{java_arches}

is supposed to mean. Will there be an automated / scripted approach to file PRs to add ExclusiveArch: %{java_arches} to affected packages?

Yup, it will require some learning and investigations from me, but at the end there should be automation which will inject it, and I will use provenpackager powers to commit, push and build (so no PR planned).

Thanx a lot for review!

Metadata Update from @churchyard:
- Issue untagged with: meeting

6 months ago

Thanx! Can I get the tracking BZ please?

Wait a bit, I'm gonna count the votes tomorrow (a week after your comment).

https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/UI4SLUMPSWU2YEMRS4VGF7DMDV4TFJQC/

Closing this ticket.

@jvanek The change s now approved and you are free to proceed, @bcotton will open a tracking bug shortly.

Metadata Update from @churchyard:
- Issue close_status updated to: Accepted
- Issue status updated to: Closed (was: Open)

6 months ago

@churchyard Why was this change approved even though the corresponding releng ticket was resolved as Can't Fix? Thanks.

Sorry, pasted the wrong link. It should have been this:

Well, they asked releng for something that's not possible to do in koji. The final solution was to add ExclusiveArch: %{java_arches} noarch to all noarch java packages.

The Java packaging guidelines will be updated to reflect the fact that this ExclusiveArch will become mandatory for Java packages.

@decathorpe What about other noarch packages that depend on java-devel, for example for building a subpackage or running the test suite? What should those packages do? Could you add guidelines as well?

@fweimer what do you mean?

  • packages that are all-noarch add ExclusiveArch: %{java_arches} noarch in addition to the already existing BuildArch: noarch
  • packages that are "archful" add ExclusiveArch: %{java_arches}

And the case where you're handling possibly overlapping sets of supported architectures, you'd have to deal with those yourself (for example, determine the intersection between %{node_arches} and %{java_arches} for a fictional package that requires both the NodeJS runtime and a JRE, which might or might not be the same on all Fedora branches). So, same as always when dealing with two (or more) hopefully-overlapping sets of supported architectures.

Any packages that need to do more complicated things than that (i.e. only build certain sub-packages on certain architectures), you've always been on your own. The Java Packaging Guidelines aren't the place to document advanced RPM packaging topics like that :)

@decathorpe

  • packages that are all-noarch add ExclusiveArch: %{java_arches} noarch in addition to the already existing BuildArch: noarch

What is the impact on the presence of the package in the repositories with this ?
Specifically in this case, the resulting package will be excluded from i686 repositories, right ?

No, "noarch packages with unported dependencies" (that's how they're called in the Packaging Guidelines) are copied into all arch-specific repositories. This is - apparently - a shortcoming of how RPM metadata works, as pungi doesn't have the information that's necessary to exclude those noarch packages from being added to repos for certain architectures.

My last koji bug report about this problem was closed here:
https://pagure.io/koji/issue/1843

And according to this closed issue, this is not solvable programmatically, because RPMs don't contain Exclusive/ExcludeArch metadata, only SRPMs do - and pungi operates on RPMs:
https://pagure.io/pungi/issue/1518

This leads to packages with broken dependencies (and broken packages peroid) being included in repositories for architectures where they should be excluded. meh

Just to note: we no longer produce i686 repos, only a koji buildroot.

Right, but it also affects the koji buildroot, and other packages on architectures for which we do publish repos.

At least RPM got an explicit fix to make the "ExclusiveArch: noarch" idiom work:
https://bugzilla.redhat.com/show_bug.cgi?id=1298668

Just to note: we no longer produce i686 repos, only a koji buildroot.

This sounds like a safety belt to me, as even in case of multilib system, the i686 pkgs are (copied) in the 86_64 repo

Login to comment on this ticket.

Metadata