If the first certificate pulled is expired, this causes unwanted behavior in:
- ldapsearch/ldapmodify/etc
- ipa-client-install
- ipa-replica-install
and even ipa-certupdate.

If expired certs are an issue for clients, we probably also want to sort them so that certs (for same subject) with later notAfter date appear first. That way, if there are two currently-valid certs (a common situation because you hopefully installed the replacement cert before the older one expires!), the newer cert is found first.

I've just tested swapping the two certs in the file, having the newer first and the expired last. This makes openssl x509 display the "right" notAfter date, fixes ldapsearch again and might fix all the others too.

openssl x509 always prints the first cert and only the first cert in a bundle file.

I agree this is expected. However all the other tools listed above seem to behave the same way.

The current PR uses a quick way to fix the issue. I agree with @ftweedal in https://pagure.io/freeipa/issue/8223#comment-632448 - a better ordering should be used, I'll revisit ASAP.

Actually, not.
If there is an expired CA certificate present in cn=certificates,cn=ipa,cn=etc,$BASEDN, chances are older clients (~4.6.5) will construct a ca.crt with an expired CA certificate on top as they will still have an unordered version of get_ca_certs(). Sample run:

# ipa-client-install 
WARNING: ntpd time&date synchronization service will not be configured as
conflicting service (chronyd) is enabled
Use --force-ntpd option to disable it and force configuration of ntpd

Discovery was successful!
Client hostname: rhel7-ipa2.myinfra.test
Realm: MYINFRA.TEST
DNS Domain: myinfra.test
IPA Server: rhel7-ipa1.myinfra.test
BaseDN: dc=myinfra,dc=test

Continue to configure the system with these values? [no]: yes
Skipping synchronizing time with NTP server.
User authorized to enroll computers: admin
Password for admin@MYINFRA.TEST: 
Successfully retrieved CA cert
    Subject:     CN=Certificate Authority,O=MYINFRA.TEST
    Issuer:      CN=Certificate Authority,O=MYINFRA.TEST
    Valid From:  2011-12-31 23:05:19
    Valid Until: 2019-12-31 23:05:19

    Subject:     CN=Certificate Authority,O=MYINFRA.TEST
    Issuer:      CN=Certificate Authority,O=MYINFRA.TEST
    Valid From:  2019-11-02 03:26:14
    Valid Until: 2039-11-02 03:26:14

Joining realm failed: Unable to initialize STARTTLS session
Failed to bind to server!
Retrying with pre-4.0 keytab retrieval method...
Unable to initialize STARTTLS session
Failed to bind to server!
Failed to get keytab
child exited with 9

Installation failed. Rolling back changes.

Furthermore, LDAP cannot guarantee us the order in which the cACertificate attributes are listed.
Ergo, expired CA certificates must be removed or moved to an attic place in the LDAP tree at ipa-cacert-manage time.
@ftweedal @rcritten can you double-check the above please? I

@fcami you are right, the ipa-certupdate reads the LDAP objects directly so an old client could install an expired cert ahead of currently-valid certs.

Deploying an updated ipa-certupdate only on servers is not enough. There are a few options (not all mutually exclusive).

1. Backport our intended ipa-certupdate changes to earlier releases and affected customers would have to update the clients.

2. As you suggest, move expired CA certs to an "attic" when we run ipa-cacert-manage. If we do this, a "housekeeping" subcommand is warranted, e.g. to move aside expired certs (or if the customer is doing time-travel, to bring valid certs out of the attic).

3. Implement "get trusted certs" as an IPA-API command (with accompanying change to ipa-certupdate, but probably not one to backport). That way, if we ever need to change the ordering heuristics again, only the servers need to be updated (assuming client is at a verison that uses the API instead of pulling certs directly from LDAP).

4. ipa-cacert-manage learns how to remove specific certificates only (not all certs for a given subject). I think I prefer the attic idea though.

We are actually hitting this problem on our production servers (FreeIPA 4.6.5, CentOS 7.7). We had to renew our CA cert (ipa-cacert-manage renew). After that, some services didn't pick up the right certificate and continued using the old one. We applied the patch in the PR but it seems that some services are still confused. I am not sure if this is by the expired certificate or are not able to cope with multiple certificates.

One service that has the problem seems to be ldapsearch. Should ipa-certupdate also update /etc/openldap/certs/ ?

What's the best way forward? Is it okay to archive the old certificates in CA,cn=certificates,cn=ipa,cn=etc, $BASEDN

The fastest path forward would be to remove the expired CA from LDAP. It should be safe to do so but saving a copy of it is recommended, just in case.

Essentially that's option #2: move expired CA certs to an "attic" when we run ipa-cacert-manage. If we do this, a "housekeeping" subcommand is warranted, e.g. to move aside expired certs (or if the customer is doing time-travel, to bring valid certs out of the attic).
We plan to have that in LDAP, but you can remove your expired certs from LDAP as long as, as @rcritten mentioned, you backup them.

@ftweedal
- I'd like to avoid changing the client code.
- I'd be tempted to do #4 quickly, outputting the cert to disk somewhere so that it can be backuped.
- #2 (LDAP attic) is nice too obviously.

@bram could you please report if removing the expired cert works for you?

Issue linked to Bugzilla: Bug 1813385

@fcami in the end we didn't remove the certificate. The patch you provided worked fine. I don't know why it didn't work with the first run (might be related to some other issue with expired certificates).

FreeIPA was included in RHEL6.1/6.2 approx 8 years ago. For our installation it generated a CA with a lifetime of 8 years. Probably more installations will end up in a situation with expired certificates and multiple CA certificates soon. We had to go back a few times with snapshots before we managed to renew all the certificates succesfully. We learned a lot about FreeIPA but all the certificates remain a bit of black magic for us.

hi @bram did you apply the patch to clients as well?
I'm afraid ipa-4-6 era clients will need the patch to be able to enroll successfully.

@ftweedal I'm investigating doing #2 (the attic), do you have a preference or advice on how this should look like in LDAP?

@fcami certstore.get_ca_certs() uses SUBTREE search, so nested container is no good. It will have to be a new container object, e.g. cn=certificates-old,cn=ipa,cn=etc,{basedn}.

Note to self: check if older clients (e.g. RHEL5/6) fetch the CA cert at this place or somewhere else.
In short, the fix will only work if it works for all old IPA versions.