I've commented on the releng ticket to not merge the side tag until this ticket is resolved:
https://pagure.io/releng/issue/9574#comment-665868

I share some of these concerns. However, there's another problem: Delaying to merge the side tag into rawhide will lead to more and more merge conflicts, because packages will start to get updated and submitted for rawhide in the meantime.

So ... should we delay merging the side tag until the IPA stack is fixed and risk growing divergence between rawhide and f33-java11 and more and more merge conflicts, or should we scrap the side tag and run another clean mass rebuild later? (I don't think this would be too bad, it's about 600 packages that are mostly noarch.)

Yes, I think the mass rebuild could have been handled better. The test COPR wasn't set up correctly / with latest patches for weeks (which is why I set up another one in the Java SIG namespace). Miro and I offered support with both the COPR rebuilds and the mass rebuild, but at least I wasn't asked for help, and was kinda surprised when the MR started, and that jvanek wants to wait for another two weeks (!) before merging it back into rawhide? That just sounds like asking for trouble to me.

(The build failure pasted above looks like something was compiled with -target 11 when it shouldn't have been, probably some ant project that doesn't set those properties. It should be easily resolvable.)

I would like to as FESCo to delay the merge of the f33-java11 side tag for at least a couple of days until the Dogtag engineering team had a chance to investigate and respond to the issue.

The mass rebuild happens on 2020-07-22. You have my +1 to delay the merge until then. Delaying the mere further than that would be problematic (but I am not strictly opposed to doing that or even restarting the Java rebuild after the mass rebuild).

In my opinion JDK should not be updated when important components like Dogtag FTBFS

+1, that was my intention whey I proposed that builds are done in side tag and the side tag is not merged when critical stuff doesn't work.

The mass rebuild happens on 2020-07-22. You have my +1 to delay the merge until then. Delaying the mere further than that would be problematic (but I am not strictly opposed to that).

Oh wow, I didn't realize this is going to happen so soon. Wanting to wait another two weeks (see bugzilla comment) with merging the side tag is completely unrealistic in this case.

I'll talk to @cipherboy. He is one of the lead engineers from the Dogtag team. Alex is not available today but should be able to respond on Monday.

FWIW, I am +1 to delay merging until worst issues are resolved, so long as it happens before the mass rebuild starts.

cc @mbooth I think this class file version mismatch is what you mentioned on the devel list. Looks like this is the culprit? https://src.fedoraproject.org/rpms/ldapjdk/blob/master/f/ldapjdk.spec#_108

FreeIPA server functionality is release blocking, so if we merge Java 11 with it broken we'll have a known release-blocking bug until it's fixed. We also won't be able to catch anything else going wrong with FreeIPA if it's broken because of this, which can be a problem. My personal opinion would be that we shouldn't merge Java 11 till this is fixed, exactly how we work the technical details of that I don't reallyt mind.

I think we're starting to be between a rock and a hard place with this :frowning:

• if we merge the side tag now, dogtag-pki / FreeIPA is broken
• if we delay merging the side tag until after the mass rebuild, things will be super duper broken
• if we don't merge the side tag, we need to revert Java 11 by default for the f33 mass rebuild next week
• if we scrap the side, tag we need to do another mass rebuild of ~600 packages in a fresh Java 11 side tag after the f33 mass rebuild is finished

Alex reached out to me with initial information to get started (thanks!). From my initial investigation:

From @cheimes scratch build log:

-- Found Sphinx: /usr/bin/sphinx-build  
-- Java_VERSION_STRING = '1.8.0_262'
-- Javac_VERSION_OUTPUT = 'javac 1.8.0_262'
-- Javadoc_VERSION_MAJOR = '1'
-- Javadoc_VERSION_MINOR = '8'
-- Configuring done

Even though PKI is building on a system with jdk11, cmake is forced to use jdk8. So, the primary problem is in pki-core. It is force using java8: https://src.fedoraproject.org/rpms/pki-core/blob/master/f/pki-core.spec#_51

Alex earlier had submitted an upstream patch to fix this: https://github.com/dogtagpki/pki/pull/399

I did a COPR build (after adding f33-java11 repo as external repo) with Alex's patch. It seems to fix the issue that @cheimes is seeing:
https://copr.fedorainfracloud.org/coprs/dmoluguw/f33-java11-pki/build/1557212/
(I don't know why there is core dump in armhfp)

According to Alex, if we merge his upstream patch, we won't be able to make usual builds we do today. But, I'll let Alex expand on it :smile:

While trying to solve the chicken-egg problem, I tried to rebuild LDAP with javac target=1.8. Technically, this should fix what Christian was observing but no promises that pki-core's build will successfully complete :confused:

if we merge the side tag now, dogtag-pki / FreeIPA is broken

We definitely shouldn't do that.

if we delay merging the side tag until after the mass rebuild, things will be super duper broken

Why so? Because of the patches in master that break 8.x support and make 11.x by default?

if we don't merge the side tag, we need to revert Java 11 by default for the f33 mass rebuild next week

We should not have in rawhide anything that is making Java 11 by default, no?

if we scrap the side, tag we need to do another mass rebuild of ~600 packages in a fresh Java 11 side tag after the f33 mass rebuild is finished

Yes, I think if we can't fix PKI thing before that, we should consider doing this or delay mass rebuild.

I'll dedicate time to fix build issues and fallout from Java 11 this weekend. After all, I've been doubly or triply CCd on all Java11FTBFS bugs :sob: Let's hope it helps.

Alright, I've fixed over 20 Java 11 FTBFS issues so far, and I think I've got a patch for ldapjdk that should fix the pki-core build issue linked by OP:

https://src.fedoraproject.org/rpms/ldapjdk/pull-request/2

So, with fixed ldapjdk and jakarta-commons-httpclient, I'm closer to fixing pki-core, but tomcatjss seems to be a major roadblock now.

tomcatjss was compiled with Java 11 in the f33-java11 side tag, but without modifying ant, which produced classfiles with incompatible version (Java 8 needs 52, Java 11 produces 55) - pki-core hard-codes java 8 (https://src.fedoraproject.org/rpms/pki-core/blob/master/f/pki-core.spec#_51). In the meantime, some other change in the buildroot (looks like jss or tomcat update) broke tomcatjss builds, so I can't fix that:

error: package org.apache.tomcat.util.net.jsse does not exist
    [javac] import org.apache.tomcat.util.net.jsse.JSSESupport

Seems like the PKI team needs to look at this one on Monday.

if we delay merging the side tag until after the mass rebuild, things will be super duper broken

Why so? Because of the patches in master that break 8.x support and make 11.x by default?

The change of the default was done in git. If it is not reverted, the mass rebuild would make it happen in rawhide.

Side note: dogtag-pki and pki-core are already broken in rawhide - possibly due to the CMake out-of-tree-build Change, and the latest tomcat update seems to cause problems as well (particularly with tomcatjss). So switching to Java 11 isn't going to break them significantly more than they already are.

I spent the weekend to fix about 40 FTBFS issues with Java 11, and tried to fix any packages that were built with too new classfile versions I came across. Other than the tomcatjss issue it looks pretty good now.

My preference would be to allow Sidetag to merge and then we'll fix Dogtag after. Dogtag can only build with one JDK at a time -- JDK 8 or JDK 11, and we need to know which we're building with at RPM build time. IIRC there's no easy way to support all three of:

1. Java 8 on F33 outside of side tag,
2. Java 11 on F33 in side tag,
3. Java 11 on F33 once side tag merges

If someone has a javapackage-tools macro that expands to JRE_HOME (or knows how to detect Java version at build time from the java-devel RPM), we can avoid this. But I wasn't able to find one easily.

So we either merge upstream Dogtag change into rawhide now, preventing us from doing any other rawhide builds until the side-tag merges, or we merge side-tag sooner, and fix Dogtag after.

My preference is on the latter, not the former. IIRC Debian is already building with JDK11, so it is just RPM packaging stuff to fix at this point. Don't hold up side tag merge on Dogtag.

Alex asked me to close the ticket. @jvanek and @cipherboy are going to handle the issue.

Everybody, thank you for your cooperation and hard work.