As a system, I want the freeipa-client install to place certificates properly into /etc/ssl/certs/ca-certificates.crt on Ubuntu systems and allow them to be use through p11-trust-kit.so so that system wide certificates will load properly instead of getting a parsing error.
The freeIPA client install seems to install extra data into the ca-certificates.crt file that p11-kit-trust.so can't parse on Ubuntu 18.04 systems.
(p11-kit:10001) loader_load_file: failed to parse: /etc/ssl/certs/ca-certificates.crt
loader_load_file to load the /etc/ssl/certs/ca-certificates.crt without issue
$ rpm -q freeipa-server freeipa-client ipa-server ipa-client 389-ds-base pki-ca krb5-server freeipa-client/bionic,now 4.7.0~pre`+git20180411-2ubuntu2 amd64 freeipa-common/bionic,bionic,now 4.7.0~pre1+git20180411-2ubuntu2 all python-ipaclient/bionic,bionic,now 4.7.0~pre1+git20180411-2ubuntu2 all p11-kit/bionic,now 0.23.9-2 p11-kit-modules/bionic,now 0.23.9-2
I ran
P11_KIT_DEBUG=all firefox
With that log output I finally found something to point me in the correct direction. Based on this log it seems like p11-kit is having parsing the ca-certificates.crt file.
$ P11_KIT_DEBUG=all firefox (p11-kit:10001) p11_library_init_impl: initializing library (p11-kit:10001) uninit_common: uninitializing library (p11-kit:10057) p11_library_init_impl: initializing library (p11-kit:10057) uninit_common: uninitializing library (p11-kit:10001) p11_library_init_impl: initializing library (p11-kit:10001) sys_C_Initialize: in (p11-kit:10001) sys_C_Initialize: doing initialization (p11-kit:10001) create_tokens_inlock: using paths: /etc/ssl/certs/ca-certificates.crt (p11-kit:10001) p11_token_new: token: System Trust: /etc/ssl/certs/ca-certificates.crt (p11-kit:10001) sys_C_Initialize: out: 0x0 (p11-kit:10001) sys_C_GetInfo: in (p11-kit:10001) sys_C_GetInfo: out: 0x0 (p11-kit:10001) sys_C_GetSlotList: in (p11-kit:10001) sys_C_GetSlotList: out: 0x0 (p11-kit:10001) sys_C_GetSlotList: in (p11-kit:10001) sys_C_GetSlotList: out: 0x0 (p11-kit:10001) sys_C_GetSlotInfo: in (p11-kit:10001) sys_C_GetSlotInfo: out: 0x0 (p11-kit:10001) sys_C_GetTokenInfo: in (p11-kit:10001) sys_C_GetTokenInfo: out: 0x0 (p11-kit:10001) sys_C_GetMechanismList: in (p11-kit:10001) sys_C_GetMechanismList: out: 0x0 (p11-kit:10001) sys_C_GetMechanismList: in (p11-kit:10001) sys_C_GetMechanismList: out: 0x0 (p11-kit:10001) sys_C_OpenSession: in (p11-kit:10001) sys_C_OpenSession: session: 17 (p11-kit:10001) sys_C_OpenSession: out: 0x0 (p11-kit:10001) sys_C_FindObjectsInit: in: 17, (1) [ { CKA_CLASS = CKO_NSS_BUILTIN_ROOT_LIST } ] (p11-kit:10001) message: ca-certificates.crt: BEGIN ...: pem block before p11-kit section header (p11-kit:10001) loader_load_file: failed to parse: /etc/ssl/certs/ca-certificates.crt (p11-kit:10001) sys_C_FindObjectsInit: out: 0x0 (p11-kit:10001) sys_C_FindObjects: in: 17, 1 (p11-kit:10001) sys_C_FindObjects: out: 0x11, 1 (p11-kit:10001) sys_C_FindObjectsFinal: in (p11-kit:10001) sys_C_FindObjectsFinal: out: 0x0
I looked at the ca-certificates.crt file
Nothing looked abnormal until I saw this…
----previous part of ca-certificates.crt----
This file was created by IPA. Do not edit.
[p11-kit-object-v1] class: certificate certificate-type: x-509 certificate-category: authority label: <removed> subject: ": <removed>" issuer: ": <removed>" serial-number: “<removed>" x-public-key-info: ": <removed>" trusted: true ------BEGIN CERTIFICATE------ ….. ----rest of ca-certificates.crt ----
Once I removed the section above the “…BEGIN CERTIFICATE…” and after the prior “----END CERTIFICATE----“ everything started working properly. I put it back and things broke.
So this indicates that p11-kit-trust.so isn’t parsing the ca-certificate.crt file due to the information that the FreeIPA client put into the file.
I am using the latest version that comes with Ubuntu 18.04 of p11-kit-trust (0.23).
I posted this question on the FreeIPA mailing list and Alexander suggested I open this ticket.
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org/thread/IUS4XEYA4HYEBSTF6OWPXB3LQVCH2OPX/
Log file locations: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Linux_Domain_Identity_Authentication_and_Policy_Guide/config-files-logs.html Troubleshooting guide: https://www.freeipa.org/page/Troubleshooting
I'm not sure it's a freeipa bug. I recommend getting an opinion from @ueno.
Everything that's not between ------BEGIN CERTIFICATE------ and -----END CERTIFICATE----- is a comment, so I would expect it to just be ignored by p11-kit.
But from the error message "BEGIN ...: pem block before p11-kit section header" I wonder if it's possible that p11-kit is using the comments in this file for some purpose and expects them to be under its control?
I'm not sure it's a freeipa bug. I recommend getting an opinion from @ueno. Everything that's not between ------BEGIN CERTIFICATE------ and -----END CERTIFICATE----- is a comment, so I would expect it to just be ignored by p11-kit. But from the error message "BEGIN ...: pem block before p11-kit section header" I wonder if it's possible that p11-kit is using the comments in this file for some purpose and expects them to be under its control?
At first glance I would say its a p11-kit bug, but I don't know what standard the file should abide by. If the ca-certificates.ca file should only have "----BEGIN CERTIFICATE----" <key information> "----END CERTIFICATE----", then I would say its a bug in the freeipa client install scripts. If you should be able to insert other content into the file and the only thing that the system should parse is the stuff between "----BEGIN CERTIFICATE----" <key information> "----END CERTIFICATE----", then I would say its a p11-kit bug.
I just didn't know where to put it. I asked on the mailing list and thats what Alexander suggested.
As a workaround I just did the follow
sudo cp /etc/ipa/ca.crt /usr/local/share/ca-certificates/ipa-ca.pem sudo update-ca-certificates
sudo cp /etc/ipa/ca.crt /usr/local/share/ca-certificates/ipa-ca.pem
sudo update-ca-certificates
the ipa-ca.pem file has the erroneous data, the /etc/ipa/ca.crt file does not.
@catanzaro ca-certificates(8) on Debian doesn't run p11-kit at all, so there is nothing parsing /etc/pki/ca-trust/source/ipa.p11-kit which we write and its content copied as it is.
/etc/pki/ca-trust/source/ipa.p11-kit
well, you could argue that the way this was added to IPA (by me) in the initial commit to add ipaplatform/debian wasn't really handled properly.. kinda surprising that no-one complained before now :)
Sounds like debian needs it's own insert_ca_certs_into_systemwide_ca_store(), and maybe add both, an p11-kit and openssl compatible cert file during client install.
Just to be clear, this is not a p11-kit bug. The p11-kit native format (block starting with "[p11-kit-object-v1]") has a higher precedence over PEM, and it can be followed by a PEM block.
Metadata Update from @abbra: - Issue assigned to tjaalton
@tjaalton so I'm assigning it to you ;)
Downstream bug: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=924590
In my opinion:
... on reflection I feel that even though the exact specification of what is valid in /usr/local/share/ca-certificates is not written down anywhere, freeipa-client should not violate the following rules: 1 - .crt files in that directory should only contain root CA certificates
... on reflection I feel that even though the exact specification of what is valid in /usr/local/share/ca-certificates is not written down anywhere, freeipa-client should not violate the following rules:
1 - .crt files in that directory should only contain root CA certificates
This is because I've noticed that if FreeIPA's certificate is signed by another root CA certificate, both the root CA certificate and the FreeIPA certificate end up in the file. This is probably harmless.
If this were to be fixed, it would be nice to have it fixed in a way that still provides the intermediate CA certificate to clients who need it (e.g., for ipa-getcert to be able to provide both a CertificateFile and a CertificateChainFile for mod_ssl).
2 - .crt files in that directory should not contain comments or any non-certificate data
Or ca-certificate should strip any extra data that it finds.
3 - .crt files in that directory should contain only one certificate
Some tools, like 'openssl x509' will only parse the first certificate in a file. If you're not aware of this then you might be mislead into trusting more certificates than you meant to!
(Yes, it would be good if ca-certificates could clarify the requirements for certificate files provided by other packages).
I've been revisiting this while trying to get some Debian machines to play nicely with our FreeIPA installation.
One of the thing that update-ca-certificates does is run 'openssl rehash' on /etc/ssl/certs to make the symlinks that OpenSSL uses to look up CA certificates without having to parse a huge CA certificate list file.
If ipa-ca.crt, which is symlinked into /etc/ssl/certs, contains multiple certificates then problems arise:
# openssl rehash /etc/ssl/certs rehash: warning: skipping ipa-ca.pem,it does not contain exactly one certificate or CRL
One way to fix this would be to regard ipa-ca.crt as a file that's specific only to freeipa, and install it to a private directory such as /var/lib/freeipa. Then, for each certificate found within, create a file within /usr/local/share/ca-certificates/ipa; update-ca-certificates iterates over this directory recursively so each certificate within it will be symlinked into /etc/ssl/certs and be processed correctly by 'openssl rehash'.
We really need to make Debian-specific method for updating system wide store. No need to create these workarounds instead, they are fragile and don't give the same experience as we have on other platforms.
Well, I'm trying to propose such a method...
ca-certificates has 2 possible integration points. Either you drop .crt files into /usr/local/share/ca-certificates (one CA per file), which causes update-ca-certificates to trust them unconditionally; or you drop them into a subdirectory of /usr/share/ca-certificates (e.g., /usr/share/ca-certificates/freeipa, in which case they will be trusted unless the user disabled them in /etc/ca-certificates.conf.
.crt
/usr/local/share/ca-certificates
update-ca-certificates
/usr/share/ca-certificates
/usr/share/ca-certificates/freeipa
/etc/ca-certificates.conf
How ipa-certupdate & friends decide to fulfil that contract is up to you. I am happy to send a proposed patch along the lines above if you don't want to discount it out of hand.
ipa-certupdate
BTW, there is one extra case to think of. My installation at work has two certificates for the same CA (i.e., same subject DN) in the FreeIPA trust store. This is because the CA certificate was renewed, and when I ran ipa-cacert-manage on the new certificate, the old one was not removed from the directory (cn=certificates,cn=ipa,cn=etc,$prefix).
ipa-cacert-manage
So when ipa-client-install and ipa-cacert-update want to write out the trusted certificates for consumption by update-ca-certificates, simply writing out one file for each certificate in the directory won't work: the old and the new certificates will have the same subject DN and hence openssl rehash will calculate the same hash.
ipa-client-install
ipa-cacert-update
openssl rehash
Either ipa-cacert-manage needs to replace the original certificate in such a scenario (and something has to go through the trust store on upgrade and remove the older certificate with the same subject DN); or ipa-certupdate has to deduplicate what's in the directory per subject DN (e.g., pick the certificate with a notBefore date in the past && the notAfter date the furthest in the future).
FYI I filed https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=945274 against ca-certificates to discover whose job it is to split multiple CA certificates out into separate files -- ca-certificates or integrators. :)
Thank you, @yrro. I think we should be not producing p11-kit config for Debian platform but I agree it would be nice to fix ca-certificate to support modern ways of handling certificates.
On the other hand, your issue with not removed certificate sounds like a separate bug against FreeIPA.
I'm thinking debian should produce both, p11-kit config and a stripped-down version for ca-certificates. Just haven't made it happen yet.
Cool - I have filed #8124 re: the duplicate CA entry in the directory.
master:
ipa-4-8:
Metadata Update from @abbra: - Issue close_status updated to: fixed - Issue status updated to: Closed (was: Open)
Metadata Update from @abbra: - Custom field changelog adjusted to On Debian-based platforms update-ca-certificates does not support multiple certificates in a single file. IPA installers now write individual files per each certificate for Debian-based platforms.
Log in to comment on this ticket.