Ticket was cloned from Red Hat Bugzilla (product Red Hat Enterprise Linux 7): Bug 1469246
Please note that this Bug is private and may not be accessible as it contains confidential Red Hat customer information.
+++ This bug was initially created as a clone of Bug #1467675 +++ Description of problem: Login fails at WebUi for replica server setup using ipa-server-docker image Version-Release number of selected component (if applicable): ipa-server-4.5.0-20.el7.x86_64 ipa-server image: ipa-server-docker-4.5.0-5 How reproducible: Always Steps to Reproduce: 1. Setup ipa master using ipa-docker image. # atomic install --name ipa-server-container rhel7/ipa-server net-host --hostname=`hostname` --setup-dns --ip-address=<ip address> --forwarder=1x.x.x.-r TESTRELm.TEST -a Secret123 -p Secret123 --no-ntp -U 2. Configure ipa-replica using ipa-docker image #atomic install --name ipa-replica-container rhel7/ipa-server net-host ipa-replica-install --setup-dns --setup-ca --server=ipa-server.testrelm.test --domain testrelm.test --forwarder=1x.x.x.x --admin-password Secret123 --principal admin -U 3. Start the ipa-replica container configurred. 4. Now try accessing the WebUi for IPA-master. 5. Now try accessing the WebUi for IPA-replica. Actual results: 1. After step4, login to ipa-master is successful. 2. After step5, login to ipa-replica fails with error "Login failed due to unknown reason" Expected results: The login should be successful for ipa-replica configured using ipa-docker image. Additional info: The login for ipa-replica configured on RHEL system is successful. --- Additional comment from Red Hat Bugzilla Rules Engine on 2017-07-04 10:14:48 EDT --- Since this bug report was entered in Red Hat Bugzilla, the release flag has been set to ? to ensure that it is properly evaluated for this release. --- Additional comment from Nikhil Dehadrai on 2017-07-04 10:15 EDT --- --- Additional comment from Nikhil Dehadrai on 2017-07-04 10:20:12 EDT --- Additional information for ipa-replica login failure: -bash-4.2# atomic run ipa-replica-container rpm -q ipa-server ipa-server-4.5.0-20.el7.x86_64 -bash-4.2# hostname auto-hv-01-guest07.testrelm.test -bash-4.2# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 10.19.34.106 auto-hv-01-guest07.testrelm.test -bash-4.2# atomic run ipa-replica-container /bin/bash [root@auto-hv-01-guest07 /]# tail -f /var/log/httpd/error_log [Tue Jul 04 14:18:18.518849 2017] [:error] [pid 1721] SSL Library Error: -12195 Peer does not recognize and trust the CA that issued your certificate [Tue Jul 04 14:18:34.708515 2017] [auth_gssapi:error] [pid 1722] [client 10.67.116.101:43554] NO AUTH DATA Client did not send any authentication headers, referer: https://auto-hv-01-guest07.testrelm.test/ipa/ui/ [Tue Jul 04 14:18:35.049957 2017] [auth_gssapi:error] [pid 1720] [client 10.67.116.101:43556] NO AUTH DATA Client did not send any authentication headers, referer: https://auto-hv-01-guest07.testrelm.test/ipa/ui/ [Tue Jul 04 14:18:41.642115 2017] [:error] [pid 1609] [remote 10.67.116.101:200] mod_wsgi (pid=1609): Exception occurred processing WSGI script '/usr/share/ipa/wsgi.py'. [Tue Jul 04 14:18:41.642243 2017] [:error] [pid 1609] [remote 10.67.116.101:200] Traceback (most recent call last): [Tue Jul 04 14:18:41.642296 2017] [:error] [pid 1609] [remote 10.67.116.101:200] File "/usr/share/ipa/wsgi.py", line 51, in application [Tue Jul 04 14:18:41.642392 2017] [:error] [pid 1609] [remote 10.67.116.101:200] return api.Backend.wsgi_dispatch(environ, start_response) [Tue Jul 04 14:18:41.642408 2017] [:error] [pid 1609] [remote 10.67.116.101:200] File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 262, in __call__ [Tue Jul 04 14:18:41.642437 2017] [:error] [pid 1609] [remote 10.67.116.101:200] return self.route(environ, start_response) [Tue Jul 04 14:18:41.642445 2017] [:error] [pid 1609] [remote 10.67.116.101:200] File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 274, in route [Tue Jul 04 14:18:41.642465 2017] [:error] [pid 1609] [remote 10.67.116.101:200] return app(environ, start_response) [Tue Jul 04 14:18:41.642490 2017] [:error] [pid 1609] [remote 10.67.116.101:200] File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 929, in __call__ [Tue Jul 04 14:18:41.642509 2017] [:error] [pid 1609] [remote 10.67.116.101:200] self.kinit(user_principal, password, ipa_ccache_name) [Tue Jul 04 14:18:41.642516 2017] [:error] [pid 1609] [remote 10.67.116.101:200] File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 965, in kinit [Tue Jul 04 14:18:41.642532 2017] [:error] [pid 1609] [remote 10.67.116.101:200] pkinit_anchors=[paths.KDC_CERT, paths.KDC_CA_BUNDLE_PEM], [Tue Jul 04 14:18:41.642549 2017] [:error] [pid 1609] [remote 10.67.116.101:200] File "/usr/lib/python2.7/site-packages/ipalib/install/kinit.py", line 125, in kinit_armor [Tue Jul 04 14:18:41.642577 2017] [:error] [pid 1609] [remote 10.67.116.101:200] run(args, env=env, raiseonerr=True, capture_error=True) [Tue Jul 04 14:18:41.642588 2017] [:error] [pid 1609] [remote 10.67.116.101:200] File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 511, in run [Tue Jul 04 14:18:41.642614 2017] [:error] [pid 1609] [remote 10.67.116.101:200] raise CalledProcessError(p.returncode, arg_string, str(output)) [Tue Jul 04 14:18:41.642681 2017] [:error] [pid 1609] [remote 10.67.116.101:200] CalledProcessError: Command '/usr/bin/kinit -n -c /var/run/ipa/ccaches/armor_1609 -X X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt -X X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem' returned non-zero exit status 1 --- Additional comment from Red Hat Bugzilla Rules Engine on 2017-07-04 10:26:51 EDT --- This bug report has Keywords: Regression or TestBlocker. Since no regressions or test blockers are allowed between releases, it is also being [proposed|marked] as a blocker for this release. Please resolve ASAP. --- Additional comment from Nikhil Dehadrai on 2017-07-06 04:04:52 EDT --- -bash-4.2# atomic host status State: idle Deployments: ? atomic-host:rhel-atomic-host/7/x86_64/standard Version: 7.4.0 (2017-06-30 18:37:40) Commit: 8018f95c2f2f38a79e68f174dd5888b53769c0e4adcd89c87a802219091c9d0e -bash-4.2# rpm -q selinux-policy selinux-policy-3.13.1-165.el7.noarch -bash-4.2# atomic run ipa-replica-container rpm -qa selinux-policy selinux-policy-3.13.1-165.el7.noarch -bash-4.2# docker images REPOSITORY TAG IMAGE ID CREATED SIZE mbasti/ipa-server-docker extras-rhel-7.4-docker-candidate-25601-20170704070600 75d4c389bfaf 2 days ago 699.1 MB rhel7/ipa-server latest 75d4c389bfaf 2 days ago 699.1 MB -bash-4.2# atomic run ipa-replica-container kinit admin Password for admin@TESTRELM.TEST: -bash-4.2# atomic run ipa-replica-container rpm -q ipa-server ipa-server-4.5.0-20.el7.x86_64 -bash-4.2# atomic run ipa-replica-container ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING named Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: STOPPED pki-tomcatd Service: RUNNING ipa-otpd Service: RUNNING ipa-dnskeysyncd Service: RUNNING ipa: INFO: The ipactl command was successful -bash-4.2# atomic run ipa-replica-container ls -l /var/log/audit/audit.log ls: cannot access /var/log/audit/audit.log: No such file or directory -bash-4.2# atomic run ipa-replica-container ausearch -m avc rpc error: code = 13 desc = invalid header field value "oci runtime error: exec failed: container_linux.go:247: starting container process caused \"exec: \\\"ausearch\\\": executable file not found in $PATH\"\n" -bash-4.2# ausearch -m avc bash: ausearch: command not found -bash-4.2# ls -l /var/log/audit/audit.log ls: cannot access /var/log/audit/audit.log: No such file or directory -bash-4.2# sestatus SELinux status: enabled SELinuxfs mount: /sys/fs/selinux SELinux root directory: /etc/selinux Loaded policy name: targeted Current mode: enforcing Mode from config file: enforcing Policy MLS status: enabled Policy deny_unknown status: allowed Max kernel policy version: 28 -bash-4.2# atomic run ipa-replica-container tail -50 /var/log/httpd/error_log [Thu Jul 06 07:35:19.083102 2017] [mpm_prefork:notice] [pid 3262] AH00170: caught SIGWINCH, shutting down gracefully [Thu Jul 06 07:35:22.334192 2017] [suexec:notice] [pid 3610] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Thu Jul 06 07:35:22.334391 2017] [:warn] [pid 3610] NSSSessionCacheTimeout is deprecated. Ignoring. [Thu Jul 06 07:35:22.708180 2017] [auth_digest:notice] [pid 3610] AH01757: generating secret for digest authentication ... [Thu Jul 06 07:35:22.709496 2017] [lbmethod_heartbeat:notice] [pid 3610] AH02282: No slotmem from mod_heartmonitor [Thu Jul 06 07:35:22.709545 2017] [:warn] [pid 3610] NSSSessionCacheTimeout is deprecated. Ignoring. [Thu Jul 06 07:35:22.762494 2017] [mpm_prefork:notice] [pid 3610] AH00163: Apache/2.4.6 (Red Hat Enterprise Linux) mod_auth_gssapi/1.5.1 mod_nss/1.0.14 NSS/3.28.4 mod_wsgi/3.4 Python/2.7.5 configured -- resuming normal operations [Thu Jul 06 07:35:22.762605 2017] [core:notice] [pid 3610] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND' [Thu Jul 06 07:35:51.711033 2017] [:error] [pid 3615] ipa: INFO: *** PROCESS START *** [Thu Jul 06 07:35:53.521263 2017] [:error] [pid 3614] ipa: INFO: *** PROCESS START *** [Thu Jul 06 07:36:05.507084 2017] [mpm_prefork:notice] [pid 3610] AH00170: caught SIGWINCH, shutting down gracefully [Thu Jul 06 07:39:30.205283 2017] [suexec:notice] [pid 402] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Thu Jul 06 07:39:30.205511 2017] [:warn] [pid 402] NSSSessionCacheTimeout is deprecated. Ignoring. [Thu Jul 06 07:39:30.788134 2017] [auth_digest:notice] [pid 402] AH01757: generating secret for digest authentication ... [Thu Jul 06 07:39:30.790782 2017] [lbmethod_heartbeat:notice] [pid 402] AH02282: No slotmem from mod_heartmonitor [Thu Jul 06 07:39:30.790840 2017] [:warn] [pid 402] NSSSessionCacheTimeout is deprecated. Ignoring. [Thu Jul 06 07:39:30.860616 2017] [mpm_prefork:notice] [pid 402] AH00163: Apache/2.4.6 (Red Hat Enterprise Linux) mod_auth_gssapi/1.5.1 mod_nss/1.0.14 NSS/3.28.4 mod_wsgi/3.4 Python/2.7.5 configured -- resuming normal operations [Thu Jul 06 07:39:30.860721 2017] [core:notice] [pid 402] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND' [Thu Jul 06 07:39:54.873755 2017] [:error] [pid 408] ipa: INFO: *** PROCESS START *** [Thu Jul 06 07:39:55.378169 2017] [:error] [pid 407] ipa: INFO: *** PROCESS START *** [Thu Jul 06 07:42:07.786211 2017] [mpm_prefork:notice] [pid 402] AH00170: caught SIGWINCH, shutting down gracefully [Thu Jul 06 07:42:11.043276 2017] [suexec:notice] [pid 992] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Thu Jul 06 07:42:11.043480 2017] [:warn] [pid 992] NSSSessionCacheTimeout is deprecated. Ignoring. [Thu Jul 06 07:42:11.414785 2017] [auth_digest:notice] [pid 992] AH01757: generating secret for digest authentication ... [Thu Jul 06 07:42:11.416089 2017] [lbmethod_heartbeat:notice] [pid 992] AH02282: No slotmem from mod_heartmonitor [Thu Jul 06 07:42:11.416137 2017] [:warn] [pid 992] NSSSessionCacheTimeout is deprecated. Ignoring. [Thu Jul 06 07:42:11.509966 2017] [mpm_prefork:notice] [pid 992] AH00163: Apache/2.4.6 (Red Hat Enterprise Linux) mod_auth_gssapi/1.5.1 mod_nss/1.0.14 NSS/3.28.4 mod_wsgi/3.4 Python/2.7.5 configured -- resuming normal operations [Thu Jul 06 07:42:11.510073 2017] [core:notice] [pid 992] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND' [Thu Jul 06 07:42:35.553640 2017] [:error] [pid 997] ipa: INFO: *** PROCESS START *** [Thu Jul 06 07:42:37.883217 2017] [:error] [pid 996] ipa: INFO: *** PROCESS START *** [Thu Jul 06 07:44:02.049575 2017] [:error] [pid 1002] SSL Library Error: -12195 Peer does not recognize and trust the CA that issued your certificate [Thu Jul 06 07:44:18.281631 2017] [auth_gssapi:error] [pid 1003] [client 10.67.116.82:56910] NO AUTH DATA Client did not send any authentication headers, referer: https://cypher.testrelm.test/ipa/ui/ [Thu Jul 06 07:44:18.667492 2017] [auth_gssapi:error] [pid 999] [client 10.67.116.82:56912] NO AUTH DATA Client did not send any authentication headers, referer: https://cypher.testrelm.test/ipa/ui/ [Thu Jul 06 07:44:24.245616 2017] [:error] [pid 996] [remote 10.67.116.82:64] mod_wsgi (pid=996): Exception occurred processing WSGI script '/usr/share/ipa/wsgi.py'. [Thu Jul 06 07:44:24.245771 2017] [:error] [pid 996] [remote 10.67.116.82:64] Traceback (most recent call last): [Thu Jul 06 07:44:24.245838 2017] [:error] [pid 996] [remote 10.67.116.82:64] File "/usr/share/ipa/wsgi.py", line 51, in application [Thu Jul 06 07:44:24.246084 2017] [:error] [pid 996] [remote 10.67.116.82:64] return api.Backend.wsgi_dispatch(environ, start_response) [Thu Jul 06 07:44:24.246112 2017] [:error] [pid 996] [remote 10.67.116.82:64] File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 262, in __call__ [Thu Jul 06 07:44:24.256853 2017] [:error] [pid 996] [remote 10.67.116.82:64] return self.route(environ, start_response) [Thu Jul 06 07:44:24.256896 2017] [:error] [pid 996] [remote 10.67.116.82:64] File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 274, in route [Thu Jul 06 07:44:24.256945 2017] [:error] [pid 996] [remote 10.67.116.82:64] return app(environ, start_response) [Thu Jul 06 07:44:24.256971 2017] [:error] [pid 996] [remote 10.67.116.82:64] File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 929, in __call__ [Thu Jul 06 07:44:24.257007 2017] [:error] [pid 996] [remote 10.67.116.82:64] self.kinit(user_principal, password, ipa_ccache_name) [Thu Jul 06 07:44:24.257030 2017] [:error] [pid 996] [remote 10.67.116.82:64] File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 965, in kinit [Thu Jul 06 07:44:24.257066 2017] [:error] [pid 996] [remote 10.67.116.82:64] pkinit_anchors=[paths.KDC_CERT, paths.KDC_CA_BUNDLE_PEM], [Thu Jul 06 07:44:24.257091 2017] [:error] [pid 996] [remote 10.67.116.82:64] File "/usr/lib/python2.7/site-packages/ipalib/install/kinit.py", line 125, in kinit_armor [Thu Jul 06 07:44:24.272025 2017] [:error] [pid 996] [remote 10.67.116.82:64] run(args, env=env, raiseonerr=True, capture_error=True) [Thu Jul 06 07:44:24.272061 2017] [:error] [pid 996] [remote 10.67.116.82:64] File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 511, in run [Thu Jul 06 07:44:24.286285 2017] [:error] [pid 996] [remote 10.67.116.82:64] raise CalledProcessError(p.returncode, arg_string, str(output)) [Thu Jul 06 07:44:24.286408 2017] [:error] [pid 996] [remote 10.67.116.82:64] CalledProcessError: Command '/usr/bin/kinit -n -c /var/run/ipa/ccaches/armor_996 -X X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt -X X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem' returned non-zero exit status 1 -bash-4.2# --- Additional comment from Nikhil Dehadrai on 2017-07-07 08:24:08 EDT --- On another similar setup with same replica login issues, checked the following for permissions: X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem IPA-MASTER: ============= -bash-4.2# atomic run ipa-server-container /bin/bash [root@auto-hv-01-guest09 /]# ls -l /var/kerberos/krb5kdc/kdc.crt -rw-r--r--. 1 root root 1675 Jul 7 06:45 /var/kerberos/krb5kdc/kdc.crt [root@auto-hv-01-guest09 /]# ls -l /var/lib/ipa-client/pki/kdc-ca-bundle.pem -rw-r--r--. 1 root root 1317 Jul 7 06:47 /var/lib/ipa-client/pki/kdc-ca-bundle.pem [root@auto-hv-01-guest09 /]# IPA_REPLICA: ============= -bash-4.2# atomic run ipa-replica-container /bin/bash [root@qe-blade-12 /]# cd /var/kerberos/krb5kdc/kdc. kdc.conf kdc.crt kdc.key [root@qe-blade-12 /]# cd /var/kerberos/krb5kdc/kdc.crt bash: cd: /var/kerberos/krb5kdc/kdc.crt: Not a directory [root@qe-blade-12 /]# ls -l /var/kerberos/krb5kdc/kdc.crt -rw-r--r--. 1 root root 1671 Jul 7 07:02 /var/kerberos/krb5kdc/kdc.crt [root@qe-blade-12 /]# ls -l /var/lib/ipa-client/pki/kdc-ca-bundle.pem -rw-r--r--. 1 root root 1317 Jul 7 06:58 /var/lib/ipa-client/pki/kdc-ca-bundle.pem --- Additional comment from Martin Babinsky on 2017-07-10 13:19:16 EDT --- The root cause seems to be that on replica container the /var/run/ipa/ccaches directory is not created which makes it impossible for Kerberos library to store both armor and user ccaches:
args=/usr/bin/kinit -n -c /var/run/ipa/ccaches/armo r_580 -X X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt -X X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle. pem [Mon Jul 10 16:45:58.869812 2017] [:error] [pid 580] ipa: DEBUG: Process finished, return code=1 [Mon Jul 10 16:45:58.869897 2017] [:error] [pid 580] ipa: DEBUG: stdout= [Mon Jul 10 16:45:58.869950 2017] [:error] [pid 580] ipa: DEBUG: stderr=kinit: Failed to store credentials: No crede ntials cache found (filename: /var/run/ipa/ccaches/armor_580) while getting initial credentials
This means that for some reason the directory either is not created (and fails silently) at the beginning of replica installation, or it is being removed in some subsequent step. Incidentally, while /etc/tmpfiles.d/ipa.conf exists on master container, it is absent on the replica. I need more time to investigate why this happens, I currently have no idea why replica behaves differently from master. --- Additional comment from Martin Babinsky on 2017-07-10 13:29:05 EDT --- After poking around the code I have found out that the issue is indeed in IPA code. For some reason the tmpfiles.d configuration is modified at runtime during server/replica install but the configuration on replica side is incomplete and does not work. I will clone this BZ to ipa-server.
Metadata Update from @mbabinsk: - Custom field rhbz adjusted to https://bugzilla.redhat.com/show_bug.cgi?id=1469246
ipa-4-5:
Metadata Update from @tkrizek: - Issue priority set to: blocker - Issue set to the milestone: FreeIPA 4.6
master:
Metadata Update from @stlaz: - Issue close_status updated to: fixed - Issue status updated to: Closed (was: Open)
Log in to comment on this ticket.