#9122 [Tracker] Nightly failure (rawhide) installing the CA
Closed: fixed 2 years ago by frenaud. Opened 2 years ago by frenaud.

Issue

The nightly tests running on rawhide are failing in the server installation, in the step configuring the CA server. See PR #1556 with for instance the test test_installation_TestInstallMaster: Details, report.

RUN ['ipa-server-install', '-n', 'ipa.test', '-r', 'IPA.TEST', '-p', 'Secret.123', '-a', 'Secret.123', '--domain-level=1', '--dirsrv-config-file', '/ipatests/ipatests_dse.ldif', '-U']
[...]
Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes
  [1/29]: configuring certificate server instance
Failed to configure CA instance
See the installation logs and the following files/directories for more information:
  /var/log/pki/pki-tomcat
  [error] RuntimeError: CA configuration failed.
CA configuration failed.
The ipa-server-install command failed. See /var/log/ipaserver-install.log for more information
Exit code: 1

The log file pki-ca-spawn shows that the pkispawn command fails waiting for PKI server to start:

2022-03-05 11:44:23 DEBUG: Command: systemctl start pki-tomcatd@pki-tomcat.service
2022-03-05 11:44:27 INFO: Waiting for PKI server to start
...
2022-03-05 11:46:26 INFO: Waiting for PKI server to start (119s)

And the journal shows the following:

Mar 05 11:44:27 master.ipa.test audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=pki-tomcatd@pki-tomcat comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Mar 05 11:44:27 master.ipa.test server[18886]: Java virtual machine used: /usr/lib/jvm/jre-17-openjdk/bin/java
Mar 05 11:44:27 master.ipa.test server[18886]: classpath used: /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:
Mar 05 11:44:27 master.ipa.test server[18886]: main class used: org.apache.catalina.startup.Bootstrap
Mar 05 11:44:27 master.ipa.test server[18886]: flags used: -Dcom.redhat.fips=false
Mar 05 11:44:27 master.ipa.test server[18886]: options used: -Dcatalina.base=/var/lib/pki/pki-tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp -Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager     -Djava.security.manager     -Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy
Mar 05 11:44:27 master.ipa.test server[18886]: arguments used: start
Mar 05 11:44:27 master.ipa.test server[18886]: WARNING: A command line option has enabled the Security Manager
Mar 05 11:44:27 master.ipa.test server[18886]: WARNING: The Security Manager is deprecated and will be removed in a future release
Mar 05 11:44:32 master.ipa.test server[18886]: WARNING: Some of the specified [protocols] are not supported by the SSL engine and have been skipped: [[TLSv1, TLSv1.1]]
Mar 05 11:44:46 master.ipa.test server[18886]: WARNING: The SHA1 algorithm used in com.netscape.ca.serviceCheckChallenge::<init>:1631 is deprecated. Use a more secure algorithm.
Mar 05 11:44:46 master.ipa.test server[18886]: WARNING: The SHA1 algorithm used in com.netscape.cmscore.authentication.ChallengePhraseAuthentication::init:116 is deprecated. Use a more secure algorithm.
Mar 05 11:44:47 master.ipa.test server[18886]: SEVERE: The required Server component failed to start so Tomcat is unable to start.
Mar 05 11:44:47 master.ipa.test server[18886]: org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina]]
Mar 05 11:44:47 master.ipa.test server[18886]:         at org.apache.catalina.util.LifecycleBase.handleSubClassException(LifecycleBase.java:440)
Mar 05 11:44:47 master.ipa.test server[18886]:         at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:198)
Mar 05 11:44:47 master.ipa.test server[18886]:         at org.apache.catalina.core.StandardService.startInternal(StandardService.java:432)
Mar 05 11:44:47 master.ipa.test server[18886]:         at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
Mar 05 11:44:47 master.ipa.test server[18886]:         at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:927)
Mar 05 11:44:47 master.ipa.test server[18886]:         at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
Mar 05 11:44:47 master.ipa.test server[18886]:         at org.apache.catalina.startup.Catalina.start(Catalina.java:772)
Mar 05 11:44:47 master.ipa.test server[18886]:         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Mar 05 11:44:47 master.ipa.test server[18886]:         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
Mar 05 11:44:47 master.ipa.test server[18886]:         at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Mar 05 11:44:47 master.ipa.test server[18886]:         at java.base/java.lang.reflect.Method.invoke(Method.java:568)
Mar 05 11:44:47 master.ipa.test server[18886]:         at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:345)
Mar 05 11:44:47 master.ipa.test server[18886]:         at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:476)
Mar 05 11:44:47 master.ipa.test server[18886]: Caused by: java.lang.ExceptionInInitializerError
Mar 05 11:44:47 master.ipa.test server[18886]:         at org.apache.tomcat.util.threads.TaskThreadFactory.newThread(TaskThreadFactory.java:63)
Mar 05 11:44:47 master.ipa.test server[18886]:         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPoolExecutor.java:630)
Mar 05 11:44:47 master.ipa.test server[18886]:         at java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:920)
Mar 05 11:44:47 master.ipa.test server[18886]:         at java.base/java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1593)
Mar 05 11:44:47 master.ipa.test server[18886]:         at java.base/java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:346)
Mar 05 11:44:47 master.ipa.test server[18886]:         at java.base/java.util.concurrent.ScheduledThreadPoolExecutor.scheduleWithFixedDelay(ScheduledThreadPoolExecutor.java:680)
Mar 05 11:44:47 master.ipa.test server[18886]:         at org.apache.tomcat.util.threads.ScheduledThreadPoolExecutor.scheduleWithFixedDelay(ScheduledThreadPoolExecutor.java:138)
Mar 05 11:44:47 master.ipa.test server[18886]:         at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:951)
Mar 05 11:44:47 master.ipa.test server[18886]:         at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:263)
Mar 05 11:44:47 master.ipa.test server[18886]:         at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
Mar 05 11:44:47 master.ipa.test server[18886]:         ... 11 more
Mar 05 11:44:47 master.ipa.test server[18886]: Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make field private java.security.AccessControlContext java.lang.Thread.inheritedAccessControlContext accessible: module java.base does not "opens java.lang" to unnamed module @6615435c
Mar 05 11:44:47 master.ipa.test server[18886]:         at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)
Mar 05 11:44:47 master.ipa.test server[18886]:         at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
Mar 05 11:44:47 master.ipa.test server[18886]:         at java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:178)
Mar 05 11:44:47 master.ipa.test server[18886]:         at java.base/java.lang.reflect.Field.setAccessible(Field.java:172)
Mar 05 11:44:47 master.ipa.test server[18886]:         at org.apache.tomcat.util.security.PrivilegedSetAccessControlContext.<clinit>(PrivilegedSetAccessControlContext.java:41)
Mar 05 11:44:47 master.ipa.test server[18886]:         ... 21 more
Mar 05 11:44:47 master.ipa.test server[18886]: WARNING: When running on Java 9 or later you need to add "--add-opens=java.base/java.io=null" to the JVM command line arguments to enable ObjectStream cache memory leak protection. Alternatively, you can suppress this warning by disabling ObjectStream class cache memory leak protection.
Mar 05 11:44:47 master.ipa.test server[18886]: WARNING: When running on Java 9 or later you need to add "--add-opens=java.base/java.lang=null" to the JVM command line arguments to enable ThreadLocal memory leak detection. Alternatively, you can suppress this warning by disabling ThreadLocal memory leak detection.
Mar 05 11:44:47 master.ipa.test server[18886]: WARNING: When running on Java 9 or later you need to add "--add-opens=java.rmi/sun.rmi.transport=null" to the JVM command line arguments to enable RMI Target memory leak detection. Alternatively, you can suppress this warning by disabling RMI Target memory leak detection.
Mar 05 11:44:47 master.ipa.test server[18886]: WARNING: When running on Java 9 or later you need to add "--add-opens=java.base/java.io=null" to the JVM command line arguments to enable ObjectStream cache memory leak protection. Alternatively, you can suppress this warning by disabling ObjectStream class cache memory leak protection.
Mar 05 11:44:47 master.ipa.test server[18886]: WARNING: When running on Java 9 or later you need to add "--add-opens=java.base/java.lang=null" to the JVM command line arguments to enable ThreadLocal memory leak detection. Alternatively, you can suppress this warning by disabling ThreadLocal memory leak detection.
Mar 05 11:44:47 master.ipa.test server[18886]: WARNING: When running on Java 9 or later you need to add "--add-opens=java.rmi/sun.rmi.transport=null" to the JVM command line arguments to enable RMI Target memory leak detection. Alternatively, you can suppress this warning by disabling RMI Target memory leak detection.
Mar 05 11:44:48 master.ipa.test server[18886]: WARNING: The web application [ca] appears to have started a thread named [AsyncLoader watchdog] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
Mar 05 11:44:48 master.ipa.test server[18886]:  java.base@17.0.2/java.lang.Object.wait(Native Method)
Mar 05 11:44:48 master.ipa.test server[18886]:  java.base@17.0.2/java.lang.Object.wait(Object.java:338)
Mar 05 11:44:48 master.ipa.test server[18886]:  java.base@17.0.2/java.util.TimerThread.mainLoop(Timer.java:537)
Mar 05 11:44:48 master.ipa.test server[18886]:  java.base@17.0.2/java.util.TimerThread.run(Timer.java:516)
Mar 05 11:44:49 master.ipa.test server[18886]: WARNING: When running on Java 9 or later you need to add "--add-opens=java.base/java.io=null" to the JVM command line arguments to enable ObjectStream cache memory leak protection. Alternatively, you can suppress this warning by disabling ObjectStream class cache memory leak protection.
Mar 05 11:44:49 master.ipa.test server[18886]: WARNING: When running on Java 9 or later you need to add "--add-opens=java.base/java.lang=null" to the JVM command line arguments to enable ThreadLocal memory leak detection. Alternatively, you can suppress this warning by disabling ThreadLocal memory leak detection.
Mar 05 11:44:49 master.ipa.test server[18886]: WARNING: When running on Java 9 or later you need to add "--add-opens=java.rmi/sun.rmi.transport=null" to the JVM command line arguments to enable RMI Target memory leak detection. Alternatively, you can suppress this warning by disabling RMI Target memory leak detection.

Installed packages (full list available here):

389-ds-base-2.1.0-1.fc36.x86_64
dogtag-pki-server-11.1.0-1.fc37.noarch
dogtag-pki-symkey-11.0.0-1.fc36.x86_64
tomcat-9.0.59-1.fc37.noarch
tomcatjss-8.1.0-1.fc37.noarch
java-11-openjdk-headless-11.0.14.1.1-5.fc37.x86_64
java-17-openjdk-headless-17.0.2.0.8-7.fc37.x86_64


The issue seems related to tomcat update to tomcat-9.0.59-1.fc37.noarch. Downgrading tomcat to 9.0.56 solves the issue.

The issue is tracked on tomcat side at https://bugzilla.redhat.com/show_bug.cgi?id=2061424 and has been solved with tomcat-9.0.59-2.fc37.
Successful run in PR#1571, test fedora-rawhide/test_installation_TestInstallWithoutNamed with the following logs and report.

This issue can be closed.

Metadata Update from @frenaud:
- Issue close_status updated to: fixed
- Issue status updated to: Closed (was: Open)

2 years ago

Login to comment on this ticket.

Metadata