Issue #7336: CA cert name detection - freeipa - Pagure.io

freeipa

#7336 CA cert name detection

Opened 6 years ago by volga629. Modified 5 years ago

Issue

Upgrade failing from f26 to f27

Steps to Reproduce

Run upgrade from f26 to f27
Run ipa-server-upgrade

Actual behavior

Update complete
Upgrading IPA services
Upgrading the configuration of the IPA services
[Verifying that root certificate is published]
[Migrate CRL publish directory]
CRL tree already moved
[Verifying that CA proxy configuration is correct]
IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
CA did not start in 300.0s

Expected behavior

Upgrade completed successfully

Version/Release/Distribution

$ rpm -q freeipa-server freeipa-client ipa-server ipa-client 389-ds-base pki-ca krb5-server

[root@caipa00 ~]# rpm -q freeipa-server freeipa-client ipa-server ipa-client 389-ds-base pki-ca krb5-server
freeipa-server-4.6.1-3.fc27.x86_64
freeipa-client-4.6.1-3.fc27.x86_64
package ipa-server is not installed
package ipa-client is not installed
389-ds-base-1.3.7.8-1.fc27.x86_64
pki-ca-10.5.1-1.fc27.noarch
krb5-server-1.15.2-4.fc27.x86_64

Additional info:

In debug mode found that CA certificate name parser failing in certdb.py

ipapython.ipautil: DEBUG: stderr=
ipaserver.install.ldapupdate: DEBUG: Executing upgrade plugin: update_ra_cert_store
ipalib.frontend: DEBUG: raw: update_ra_cert_store
ipalib.frontend: DEBUG: raw: ca_is_enabled(version='2.229')
ipalib.frontend: DEBUG: ca_is_enabled(version='2.229')
ipapython.ipautil: DEBUG: Starting external process
ipapython.ipautil: DEBUG: args=/usr/bin/certutil -d /etc/httpd/alias -L -n ipaCert -a -f /etc/httpd/alias/pwdfile.txt
ipapython.ipautil: DEBUG: Process finished, return code=255
ipapython.ipautil: DEBUG: stdout=
ipapython.ipautil: DEBUG: stderr=certutil: Could not find cert: ipaCert
: PR_FILE_NOT_FOUND_ERROR: File not found

Also in NSS database certificate nicknames should not contain any spaces that cause parser to fail.
Current certs names

[root@caipa00 ~]# /usr/bin/certutil -d /etc/httpd/alias -L 

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

Signing-Cert                                                 u,u,u
DOMAIN.TLD IPA CA                                      CT,C,C
Server-Cert                                                  u,u,u

because file detection should based on nss CA attribute and not file name or regex.
Simple shell command will do:

 /usr/bin/certutil -d /etc/httpd/alias -L /etc/httpd/alias/pwdfile.txt | grep 'CT,C,C' | awk -F '  ' '{print $1}'

volga629 commented 6 years ago

Upgrade log.

volga629 commented 6 years ago

Upgade log is too big
https://www.dropbox.com/s/t1o61qaa6dicta5/ipaupgrade.log?dl=0

frenaud commented 6 years ago

The root cause of the issue is not the update_ra_cert_store message. This routine takes the ipaCert from /etc/httpd/alias and migrates the cert to /var/lib/ipa/ra-agent.key and /var/lib/ipa/ra-agent.pem.
When ipa upgrade is performed multiple times, it is expected that ipaCert is not found any more in /etc/httpd/alias and the log is only informative.

The ipaupgrade.log shows that Dogtag failed to start during the upgrade, and this needs to be investigated. Can you share dogtag's logs from /var/log/pki/pki-tomcat/ca/debug? Common causes include IPv6 configuration or the expiration of subsystemCert cert-pki-ca stored in /etc/pki/pki-tomcat/alias.

volga629 commented 6 years ago

Debug log shows authentication problem. I assume connection to ldap server

[24/Dec/2017:09:22:32][localhost-startStop-1]: CMS.start(): shutdown server
[24/Dec/2017:09:22:32][localhost-startStop-1]: CMSEngine.shutdown()
[24/Dec/2017:12:49:47][localhost-startStop-1]: ============================================
[24/Dec/2017:12:49:47][localhost-startStop-1]: =====  DEBUG SUBSYSTEM INITIALIZED   =======
[24/Dec/2017:12:49:47][localhost-startStop-1]: ============================================
[24/Dec/2017:12:49:47][localhost-startStop-1]: CMSEngine: restart at autoShutdown? false
[24/Dec/2017:12:49:47][localhost-startStop-1]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb
[24/Dec/2017:12:49:47][localhost-startStop-1]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca
[24/Dec/2017:12:49:47][localhost-startStop-1]: CMSEngine: found cert:auditSigningCert cert-pki-ca
[24/Dec/2017:12:49:47][localhost-startStop-1]: CMSEngine: done init id=debug
[24/Dec/2017:12:49:47][localhost-startStop-1]: CMSEngine: initialized debug
[24/Dec/2017:12:49:47][localhost-startStop-1]: CMSEngine: initSubsystem id=log
[24/Dec/2017:12:49:47][localhost-startStop-1]: CMSEngine: ready to init id=log
[24/Dec/2017:12:49:47][localhost-startStop-1]: Event filters:
[24/Dec/2017:12:49:47][localhost-startStop-1]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/signedAudit/ca_audit)
[24/Dec/2017:12:49:47][localhost-startStop-1]: Event filters:
[24/Dec/2017:12:49:47][localhost-startStop-1]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/system)
[24/Dec/2017:12:49:48][localhost-startStop-1]: Event filters:
[24/Dec/2017:12:49:48][localhost-startStop-1]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/transactions)
[24/Dec/2017:12:49:48][localhost-startStop-1]: CMSEngine: restart at autoShutdown? false
[24/Dec/2017:12:49:48][localhost-startStop-1]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb
[24/Dec/2017:12:49:48][localhost-startStop-1]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca
[24/Dec/2017:12:49:48][localhost-startStop-1]: CMSEngine: found cert:auditSigningCert cert-pki-ca
[24/Dec/2017:12:49:48][localhost-startStop-1]: CMSEngine: done init id=log
[24/Dec/2017:12:49:48][localhost-startStop-1]: CMSEngine: initialized log
[24/Dec/2017:12:49:48][localhost-startStop-1]: CMSEngine: initSubsystem id=jss
[24/Dec/2017:12:49:48][localhost-startStop-1]: CMSEngine: ready to init id=jss
[24/Dec/2017:12:49:48][localhost-startStop-1]: JssSubsystem: initializing JSS subsystem
[24/Dec/2017:12:49:48][localhost-startStop-1]: JssSubsystem: enabled: true
[24/Dec/2017:12:49:48][localhost-startStop-1]: JssSubsystem: NSS database: /var/lib/pki/pki-tomcat/alias/
[24/Dec/2017:12:49:48][localhost-startStop-1]: JssSubsystem: initializing CryptoManager
[24/Dec/2017:12:49:48][localhost-startStop-1]: JssSubsystem: initializing SSL
[24/Dec/2017:12:49:48][localhost-startStop-1]: JssSubsystem: random:
[24/Dec/2017:12:49:48][localhost-startStop-1]: JssSubsystem: - algorithm: pkcs11prng
[24/Dec/2017:12:49:48][localhost-startStop-1]: JssSubsystem: - provider: Mozilla-JSS
[24/Dec/2017:12:49:48][localhost-startStop-1]: JssSubsystem: initialization complete
[24/Dec/2017:12:49:48][localhost-startStop-1]: CMSEngine: restart at autoShutdown? false
[24/Dec/2017:12:49:48][localhost-startStop-1]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb
[24/Dec/2017:12:49:48][localhost-startStop-1]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca
[24/Dec/2017:12:49:48][localhost-startStop-1]: CMSEngine: found cert:auditSigningCert cert-pki-ca
[24/Dec/2017:12:49:48][localhost-startStop-1]: CMSEngine: done init id=jss
[24/Dec/2017:12:49:48][localhost-startStop-1]: CMSEngine: initialized jss
[24/Dec/2017:12:49:48][localhost-startStop-1]: CMSEngine: initSubsystem id=dbs
[24/Dec/2017:12:49:48][localhost-startStop-1]: CMSEngine: ready to init id=dbs
[24/Dec/2017:12:49:48][localhost-startStop-1]: DBSubsystem: init()  mEnableSerialMgmt=true
[24/Dec/2017:12:49:48][localhost-startStop-1]: Creating LdapBoundConnFactor(DBSubsystem)
[24/Dec/2017:12:49:48][localhost-startStop-1]: LdapBoundConnFactory: init 
[24/Dec/2017:12:49:48][localhost-startStop-1]: LdapBoundConnFactory:doCloning true
[24/Dec/2017:12:49:48][localhost-startStop-1]: LdapAuthInfo: init()
[24/Dec/2017:12:49:48][localhost-startStop-1]: LdapAuthInfo: init begins
[24/Dec/2017:12:49:48][localhost-startStop-1]: LdapAuthInfo: init ends
[24/Dec/2017:12:49:48][localhost-startStop-1]: init: before makeConnection errorIfDown is true
[24/Dec/2017:12:49:48][localhost-startStop-1]: makeConnection: errorIfDown true
[24/Dec/2017:12:49:48][localhost-startStop-1]: TCP Keep-Alive: true
[24/Dec/2017:12:49:48][localhost-startStop-1]: SSLClientCertificateSelectionCB: Setting desired cert nickname to: subsystemCert cert-pki-ca
[24/Dec/2017:12:49:48][localhost-startStop-1]: LdapJssSSLSocket: set client auth cert nickname subsystemCert cert-pki-ca
[24/Dec/2017:12:49:48][localhost-startStop-1]: SSLClientCertificatSelectionCB: Entering!
[24/Dec/2017:12:49:48][localhost-startStop-1]: Candidate cert: subsystemCert cert-pki-ca
[24/Dec/2017:12:49:48][localhost-startStop-1]: SSLClientCertificateSelectionCB: desired cert found in list: subsystemCert cert-pki-ca
[24/Dec/2017:12:49:48][localhost-startStop-1]: SSLClientCertificateSelectionCB: returning: subsystemCert cert-pki-ca
[24/Dec/2017:12:49:48][localhost-startStop-1]: SSL handshake happened
Could not connect to LDAP server host caipa00.domain.prod port 636 Error netscape.ldap.LDAPException: Authentication failed (49)
    at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205)
    at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:166)
    at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:130)
    at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:667)
    at com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1175)
    at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1081)
    at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:571)
    at com.netscape.certsrv.apps.CMS.init(CMS.java:189)
    at com.netscape.certsrv.apps.CMS.start(CMS.java:1620)
    at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117)
    at javax.servlet.GenericServlet.init(GenericServlet.java:158)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:293)
    at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:290)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
    at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:325)
    at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:176)
    at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124)
    at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1215)
    at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1140)
    at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1027)
    at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5038)
    at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5348)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
    at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:753)
    at org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:131)
    at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:153)
    at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:143)
    at java.security.AccessController.doPrivileged(Native Method)
    at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:727)
    at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:717)
    at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:621)
    at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1835)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Internal Database Error encountered: Could not connect to LDAP server host caipa00.domain.prod port 636 Error netscape.ldap.LDAPException: Authentication failed (49)
    at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:689)
    at com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1175)
    at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1081)
    at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:571)
    at com.netscape.certsrv.apps.CMS.init(CMS.java:189)
    at com.netscape.certsrv.apps.CMS.start(CMS.java:1620)
    at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117)
    at javax.servlet.GenericServlet.init(GenericServlet.java:158)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:293)
    at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:290)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
    at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:325)
    at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:176)
    at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124)
    at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1215)
    at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1140)
    at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1027)
    at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5038)
    at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5348)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
    at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:753)
    at org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:131)
    at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:153)
    at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:143)
    at java.security.AccessController.doPrivileged(Native Method)
    at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:727)
    at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:717)
    at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:621)
    at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1835)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
[24/Dec/2017:12:49:48][localhost-startStop-1]: CMS.start(): shutdown server
[24/Dec/2017:12:49:48][localhost-startStop-1]: CMSEngine.shutdown()

tbordaz commented 6 years ago

@volga629, just to eliminate the possibility of wrong nss version, would you check dirsrv error logs to be sure you are not hitting https://pagure.io/389-ds-base/issue/49498.

volga629 commented 6 years ago

I don't see any specific errors in dirsrv logs.

[24/Dec/2017:12:49:31.324355093 -0500] - ERR - schema-compat-plugin - scheduled schema-compat-plugin tree scan in about 5 seconds after the server startup!
[24/Dec/2017:12:49:31.335240924 -0500] - ERR - NSACLPlugin - acl_parse - The ACL target cn=groups,cn=compat,dc=domain,dc=prod does not exist
[24/Dec/2017:12:49:31.336379147 -0500] - ERR - NSACLPlugin - acl_parse - The ACL target cn=computers,cn=compat,dc=domain,dc=prod does not exist
[24/Dec/2017:12:49:31.337180907 -0500] - ERR - NSACLPlugin - acl_parse - The ACL target cn=ng,cn=compat,dc=domain,dc=prod does not exist
[24/Dec/2017:12:49:31.337909216 -0500] - ERR - NSACLPlugin - acl_parse - The ACL target ou=sudoers,dc=domain,dc=prod does not exist
[24/Dec/2017:12:49:31.338698808 -0500] - ERR - NSACLPlugin - acl_parse - The ACL target cn=users,cn=compat,dc=domain,dc=prod does not exist
[24/Dec/2017:12:49:31.339655742 -0500] - ERR - NSACLPlugin - acl_parse - The ACL target cn=vaults,cn=kra,dc=domain,dc=prod does not exist
[24/Dec/2017:12:49:31.340468400 -0500] - ERR - NSACLPlugin - acl_parse - The ACL target cn=vaults,cn=kra,dc=domain,dc=prod does not exist
[24/Dec/2017:12:49:31.341205459 -0500] - ERR - NSACLPlugin - acl_parse - The ACL target cn=vaults,cn=kra,dc=domain,dc=prod does not exist
[24/Dec/2017:12:49:31.341955296 -0500] - ERR - NSACLPlugin - acl_parse - The ACL target cn=vaults,cn=kra,dc=domain,dc=prod does not exist
[24/Dec/2017:12:49:31.342690195 -0500] - ERR - NSACLPlugin - acl_parse - The ACL target cn=vaults,cn=kra,dc=domain,dc=prod does not exist
[24/Dec/2017:12:49:31.343396990 -0500] - ERR - NSACLPlugin - acl_parse - The ACL target cn=vaults,cn=kra,dc=domain,dc=prod does not exist
[24/Dec/2017:12:49:31.344113590 -0500] - ERR - NSACLPlugin - acl_parse - The ACL target cn=vaults,cn=kra,dc=domain,dc=prod does not exist
[24/Dec/2017:12:49:31.344896928 -0500] - ERR - NSACLPlugin - acl_parse - The ACL target cn=vaults,cn=kra,dc=domain,dc=prod does not exist
[24/Dec/2017:12:49:31.345749910 -0500] - ERR - NSACLPlugin - acl_parse - The ACL target cn=vaults,cn=kra,dc=domain,dc=prod does not exist
[24/Dec/2017:12:49:31.346478224 -0500] - ERR - NSACLPlugin - acl_parse - The ACL target cn=vaults,cn=kra,dc=domain,dc=prod does not exist
[24/Dec/2017:12:49:31.347204217 -0500] - ERR - NSACLPlugin - acl_parse - The ACL target cn=vaults,cn=kra,dc=domain,dc=prod does not exist
[24/Dec/2017:12:49:31.351811305 -0500] - ERR - NSACLPlugin - acl_parse - The ACL target cn=ad,cn=etc,dc=domain,dc=prod does not exist
[24/Dec/2017:12:49:31.353917366 -0500] - ERR - NSACLPlugin - acl_parse - The ACL target cn=casigningcert cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=domain,dc=prod does not exist
[24/Dec/2017:12:49:31.354691057 -0500] - ERR - NSACLPlugin - acl_parse - The ACL target cn=casigningcert cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=domain,dc=prod does not exist
[24/Dec/2017:12:49:31.439005530 -0500] - ERR - NSACLPlugin - acl_parse - The ACL target cn=automember rebuild membership,cn=tasks,cn=config does not exist
[24/Dec/2017:12:49:31.441377023 -0500] - ERR - auto-membership-plugin - automember_parse_regex_rule - Unable to parse regex rule (invalid regex).  Error "nothing to repeat".
[24/Dec/2017:12:49:31.443775775 -0500] - ERR - cos-plugin - cos_dn_defs_cb - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=domain,dc=prod--no CoS Templates found, which should be added before the CoS Definition.
[24/Dec/2017:12:49:31.465311235 -0500] - ERR - set_krb5_creds - Could not get initial credentials for principal [ldap/caipa00.domain.prod@domain.prod] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm)
[24/Dec/2017:12:49:31.481211771 -0500] - INFO - slapd_daemon - slapd started.  Listening on All Interfaces port 389 for LDAP requests
[24/Dec/2017:12:49:31.482073897 -0500] - INFO - slapd_daemon - Listening on All Interfaces port 636 for LDAPS requests
[24/Dec/2017:12:49:31.482714388 -0500] - INFO - slapd_daemon - Listening on /var/run/slapd-domain-PROD.socket for LDAPI requests
[24/Dec/2017:12:49:31.591660866 -0500] - ERR - schema-compat-plugin - schema-compat-plugin tree scan will start in about 5 seconds!
[24/Dec/2017:12:49:36.644466655 -0500] - ERR - schema-compat-plugin - warning: no entries set up under cn=computers, cn=compat,dc=domain,dc=prod
[24/Dec/2017:12:49:36.647256243 -0500] - ERR - schema-compat-plugin - Finished plugin initialization.

tbordaz commented 6 years ago

That is correct, the current issue is not related to https://pagure.io/389-ds-base/issue/49498.

frenaud commented 6 years ago

The error with Could not connect to LDAP server host caipa00.domain.prod port 636 Error netscape.ldap.LDAPException: Authentication failed (49) is usually linked to an expired subsystemCert cert-pki-ca, or a failure when automatic renewal happened.

You can find troubleshooting tips in this blog. Start by checking the expiration date of the cert:
$ sudo certutil -L -d /etc/pki/pki-tomcat/alias -n "subsystemCert cert-pki-ca"| grep "Not After"

volga629 commented 6 years ago

Here output. Command return valid date

[root@caipa00 ~]# certutil -L -d /etc/pki/pki-tomcat/alias -n "subsystemCert cert-pki-ca"| grep "Not After"
            Not After : Thu Oct 03 01:06:55 2019
[root@caipa00 ~]#

volga629 commented 6 years ago

Based on troubleshooting guide key validation fail.

[root@caipa00 ~]# grep internal /var/lib/pki/pki-tomcat/conf/password.conf | cut -d= -f2 > /tmp/pwdfile.txt
[root@caipa00 ~]# certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt -n 'subsystemCert cert-pki-ca'
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services"
certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier.
[root@caipa00 ~]#

volga629 commented 6 years ago

Also CA certificates are different. How to update ldap with right cert

ftweedal commented 6 years ago

Add all relevant CA certificates to the FreeIPA certificate trust store by using
ipa-cacert-manage install. Then run ipa-certupdate an all FreeIPA masters
to ensure that each master has the required CA certificates in all of the relevant
places (system trust store, http/ldap/Dogtag NSSDBs, etc).

volga629 commented 6 years ago

Which cert is need to use for command ipa-cacert-manage install ?

ftweedal commented 6 years ago

Whatever external CA cert(s) are used in your infrastructure, including intermediate CAs.

volga629 commented 6 years ago

We was using generated by freeipa self sign

frenaud commented 6 years ago

What is the output of ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description seeAlso and sudo certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a?

If the certificates differ, you will need to manually modify the certificate in LDAP using ldapmodify (update the field "userCertificate" with the value found using certutil, and also update the field description with 2;<serial>;<issuer>;<subject> - serial issuer and subject also extracted from certutil output).

volga629 commented 6 years ago

See attached file

frenaud commented 6 years ago

The 2 certificates differ. In order to fix the issue, you need to run the following command:

$ ldapmodify -h master.domain.com -p 389 -D "cn=directory manager" -w password
dn: uid=pkidbuser,ou=people,o=ipaca
changetype: modify
replace: description
description: 2;19;CN=Certificate Authority,O=DOMAIN.PROD;CN=CA Subsystem,O=DOMAIN.PROD
-
add: usercertificate
usercertificate:: <here paste the content obtained from certutil, in a single line, without the header -----BEGIN CERTIFICATE----- and without the footer -----END CERTIFICATE-----

The description attribute needs to contain 2;19;... because the cert in /etc/pki/pki-tomcat/alias has a serial number 19 (can be seen using

$ sudo certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep "Serial Number"

), and the new certificate needs to be uploaded into ldap.
After that, pki-tomcat should be able to restart and you will be able to re-launch ipa-server-upgrade.

The certificate was renewed Oct 13 01:06:55 2017, you may find more information in the journal explaining why the renewal was not able to proceed till the end:

$ sudo journalctl -u certmonger

rcritten commented 6 years ago

Is this still a problem?

volga629 commented 6 years ago

Yes, I still need propagate CA cert. I will do it today just was pulled to other task.

volga629 commented 6 years ago

In store multiply certs

auditSigningCert cert-pki-ca u,u,Pu
ocspSigningCert cert-pki-ca u,u,u
caSigningCert cert-pki-ca CTu,Cu,Cu
Server-Cert cert-pki-ca u,u,u
subsystemCert cert-pki-ca u,u,u

Which one need update in ldap ?

rcritten commented 6 years ago

Sorry for the delay. You'd need to check on each cert individually to see if the latest is already in LDAP. The caSigningCert should be good for another 19+ years so I wouldn't worry about that, it gets stored separately anyway.

rcritten commented 6 years ago

We are closing this bug because we have not received sufficient information to make progress. Please feel free to open this bug again when you are able to provide the required information we requested.

Metadata Update from @rcritten:
- Issue close_status updated to: insufficientinfo
- Issue status updated to: Closed (was: Open)

6 years ago

volga629 commented 6 years ago

I am still unable upload cert to ldap

rcritten commented 6 years ago

Can you clarify what the current status is? I'm not sure what it is you are trying to do, what messages you are seeing, etc.

volga629 commented 6 years ago

I am trying update ldap with certificate from cert sudo certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca'
Because it miss match between store and ldap so will be possible complete upgrade

rcritten commented 6 years ago

Ok and flo provided instructions on how to do that. Are they not working?

volga629 commented 6 years ago

I can' understand exactly which cert it should be

volga629 commented 6 years ago

Yes, if I understand correctly it should one from certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca'

Metadata Update from @volga629:
- Issue status updated to: Open (was: Closed)

6 years ago

volga629 commented 5 years ago

I tried update cert.

ldapmodify: invalid format (line 7) entry: "uid=pkidbuser,ou=people,o=ipaca"

[root@caipa01 ~]# ldapmodify -h caipa01.networklab.prod -p 389 -D "cn=directory manager" -W
Enter LDAP Password:
dn: uid=pkidbuser,ou=people,o=ipaca
changetype: modify
replace: description
description: 2;19;CN=Certificate Authority,O=DOMAIN.PROD;CN=CA Subsystem,O=DOMAIN.PROD
-
add: usercertificate
usercertificate::
ldapmodify: invalid format (line 7) entry: "uid=pkidbuser,ou=people,o=ipaca"

volga629 commented 5 years ago

line 7 is
add: usercertificate
usercertificate::

volga629 commented 5 years ago

I updated cert with ldif , but tomcat still having issue

failed to map client certificate to LDAP DN (Could not matching certificate in User's LDAP entry)

volga629 commented 5 years ago

Do I need delete all other certs ? I see in list right now 3 certs

volga629 commented 5 years ago

Ok I got ldap cert updated. And ran upgrade ok, but right now apache complain about expire cert

[:error] [pid 1131:tid 140309205127168] Server certificate is expired: 'Server-Cert'

rcritten commented 5 years ago

Sorry for the delay. Were you able to complete the upgrade?

Login to comment on this ticket.

Metadata

Assignee

None

Tags

None

Blocking

None

Depending on

None

Priority

None

Milestone

None

cc

None

affects_doc

None

source

None

knownissue

None

type

None

blockedby

None

test_case

None

component

None

blocking

None

on_review

None

keywords

None

test_coverage

None

reviewer

None

external_tracker

None

rhbz

None

tester

None

changelog

None

design

None

Attachments 1

ipa-cert-diff.txt

Attached 6 years ago View Comment

Powered by Pagure 5.13.3

Documentation • File an Issue • About • SSH Hostkey/Fingerprint

© Red Hat, Inc. and others.