#8106 ca-certificate file not being parsed correctly on Ubuntu with p11-kit-trust.so due to data inserted by FreeIPA Client install
Closed: fixed 3 months ago by abbra. Opened 9 months ago by kvasko.

Request for enhancement

As a system, I want the freeipa-client install to place certificates properly into /etc/ssl/certs/ca-certificates.crt on Ubuntu systems and allow them to be use through p11-trust-kit.so so that system wide certificates will load properly instead of getting a parsing error.

Issue

The freeIPA client install seems to install extra data into the ca-certificates.crt file that p11-kit-trust.so can't parse on Ubuntu 18.04 systems.

Steps to Reproduce

  1. Install freeIPA client on Ubuntu 18.04
  2. link p11-kit-trust.so with
    sudo apt-get update && sudo apt-get install -y p11-kit libnss3
    sudo mv /usr/lib/x86_64-linux-gnu/nss/libnssckbi.so /usr/lib/x86_64-linux-gnu/nss/libnssckbi.so.bak
    sudo ln -s /usr/lib/x86_64-linux-gnu/pkcs11/p11-kit-trust.so /usr/lib/x86_64-linux-gnu/nss/libnssckbi.so
  3. run $ P11_KIT_DEBUG=all firefox

Actual behavior

(p11-kit:10001) loader_load_file: failed to parse: /etc/ssl/certs/ca-certificates.crt

Expected behavior

loader_load_file to load the /etc/ssl/certs/ca-certificates.crt without issue

Version/Release/Distribution

$ rpm -q freeipa-server freeipa-client ipa-server ipa-client 389-ds-base pki-ca krb5-server
freeipa-client/bionic,now 4.7.0~pre`+git20180411-2ubuntu2 amd64
freeipa-common/bionic,bionic,now 4.7.0~pre1+git20180411-2ubuntu2 all
python-ipaclient/bionic,bionic,now 4.7.0~pre1+git20180411-2ubuntu2 all
p11-kit/bionic,now 0.23.9-2
p11-kit-modules/bionic,now 0.23.9-2

Additional info:

I ran

P11_KIT_DEBUG=all firefox

With that log output I finally found something to point me in the correct direction. Based on this log it seems like p11-kit is having parsing the ca-certificates.crt file.

$ P11_KIT_DEBUG=all firefox
(p11-kit:10001) p11_library_init_impl: initializing library
(p11-kit:10001) uninit_common: uninitializing library
(p11-kit:10057) p11_library_init_impl: initializing library
(p11-kit:10057) uninit_common: uninitializing library
(p11-kit:10001) p11_library_init_impl: initializing library
(p11-kit:10001) sys_C_Initialize: in
(p11-kit:10001) sys_C_Initialize: doing initialization
(p11-kit:10001) create_tokens_inlock: using paths: /etc/ssl/certs/ca-certificates.crt
(p11-kit:10001) p11_token_new: token: System Trust: /etc/ssl/certs/ca-certificates.crt
(p11-kit:10001) sys_C_Initialize: out: 0x0
(p11-kit:10001) sys_C_GetInfo: in
(p11-kit:10001) sys_C_GetInfo: out: 0x0
(p11-kit:10001) sys_C_GetSlotList: in
(p11-kit:10001) sys_C_GetSlotList: out: 0x0
(p11-kit:10001) sys_C_GetSlotList: in
(p11-kit:10001) sys_C_GetSlotList: out: 0x0
(p11-kit:10001) sys_C_GetSlotInfo: in
(p11-kit:10001) sys_C_GetSlotInfo: out: 0x0
(p11-kit:10001) sys_C_GetTokenInfo: in
(p11-kit:10001) sys_C_GetTokenInfo: out: 0x0
(p11-kit:10001) sys_C_GetMechanismList: in
(p11-kit:10001) sys_C_GetMechanismList: out: 0x0
(p11-kit:10001) sys_C_GetMechanismList: in
(p11-kit:10001) sys_C_GetMechanismList: out: 0x0
(p11-kit:10001) sys_C_OpenSession: in
(p11-kit:10001) sys_C_OpenSession: session: 17
(p11-kit:10001) sys_C_OpenSession: out: 0x0
(p11-kit:10001) sys_C_FindObjectsInit: in: 17, (1) [ { CKA_CLASS = CKO_NSS_BUILTIN_ROOT_LIST } ]
(p11-kit:10001) message: ca-certificates.crt: BEGIN ...: pem block before p11-kit section header
(p11-kit:10001) loader_load_file: failed to parse: /etc/ssl/certs/ca-certificates.crt
(p11-kit:10001) sys_C_FindObjectsInit: out: 0x0
(p11-kit:10001) sys_C_FindObjects: in: 17, 1
(p11-kit:10001) sys_C_FindObjects: out: 0x11, 1
(p11-kit:10001) sys_C_FindObjectsFinal: in
(p11-kit:10001) sys_C_FindObjectsFinal: out: 0x0

I looked at the ca-certificates.crt file

Nothing looked abnormal until I saw this…

----previous part of ca-certificates.crt----

This file was created by IPA. Do not edit.

[p11-kit-object-v1]
class: certificate
certificate-type: x-509
certificate-category: authority
label: <removed>
subject: ": <removed>"
issuer: ": <removed>"
serial-number: “<removed>"
x-public-key-info: ": <removed>"
trusted: true
------BEGIN CERTIFICATE------
…..
----rest of ca-certificates.crt ----

Once I removed the section above the “…BEGIN CERTIFICATE…” and after the prior “----END CERTIFICATE----“ everything started working properly. I put it back and things broke.

So this indicates that p11-kit-trust.so isn’t parsing the ca-certificate.crt file due to the information that the FreeIPA client put into the file.

I am using the latest version that comes with Ubuntu 18.04 of p11-kit-trust (0.23).

I posted this question on the FreeIPA mailing list and Alexander suggested I open this ticket.

https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org/thread/IUS4XEYA4HYEBSTF6OWPXB3LQVCH2OPX/

Log file locations: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Linux_Domain_Identity_Authentication_and_Policy_Guide/config-files-logs.html
Troubleshooting guide: https://www.freeipa.org/page/Troubleshooting


I'm not sure it's a freeipa bug. I recommend getting an opinion from @ueno.

Everything that's not between ------BEGIN CERTIFICATE------ and -----END CERTIFICATE----- is a comment, so I would expect it to just be ignored by p11-kit.

But from the error message "BEGIN ...: pem block before p11-kit section header" I wonder if it's possible that p11-kit is using the comments in this file for some purpose and expects them to be under its control?

I'm not sure it's a freeipa bug. I recommend getting an opinion from @ueno.
Everything that's not between ------BEGIN CERTIFICATE------ and -----END CERTIFICATE----- is a comment, so I would expect it to just be ignored by p11-kit.
But from the error message "BEGIN ...: pem block before p11-kit section header" I wonder if it's possible that p11-kit is using the comments in this file for some purpose and expects them to be under its control?

At first glance I would say its a p11-kit bug, but I don't know what standard the file should abide by. If the ca-certificates.ca file should only have "----BEGIN CERTIFICATE----" <key information=""> "----END CERTIFICATE----", then I would say its a bug in the freeipa client install scripts. If you should be able to insert other content into the file and the only thing that the system should parse is the stuff between "----BEGIN CERTIFICATE----" <key information=""> "----END CERTIFICATE----", then I would say its a p11-kit bug.

I just didn't know where to put it. I asked on the mailing list and thats what Alexander suggested.

As a workaround I just did the follow

sudo cp /etc/ipa/ca.crt /usr/local/share/ca-certificates/ipa-ca.pem
sudo update-ca-certificates

the ipa-ca.pem file has the erroneous data, the /etc/ipa/ca.crt file does not.

@catanzaro ca-certificates(8) on Debian doesn't run p11-kit at all, so there is nothing parsing /etc/pki/ca-trust/source/ipa.p11-kit which we write and its content copied as it is.

well, you could argue that the way this was added to IPA (by me) in the initial commit to add ipaplatform/debian wasn't really handled properly.. kinda surprising that no-one complained before now :)

Sounds like debian needs it's own insert_ca_certs_into_systemwide_ca_store(), and maybe add both, an p11-kit and openssl compatible cert file during client install.

Just to be clear, this is not a p11-kit bug. The p11-kit native format (block starting with "[p11-kit-object-v1]") has a higher precedence over PEM, and it can be followed by a PEM block.

Metadata Update from @abbra:
- Issue assigned to tjaalton

8 months ago

@tjaalton so I'm assigning it to you ;)

Downstream bug: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=924590

In my opinion:

... on reflection I feel that even though the exact specification of what is valid in /usr/local/share/ca-certificates is not written down anywhere, freeipa-client should not violate the following rules:

1 - .crt files in that directory should only contain root CA certificates

This is because I've noticed that if FreeIPA's certificate is signed by another root CA certificate, both the root CA certificate and the FreeIPA certificate end up in the file. This is probably harmless.

If this were to be fixed, it would be nice to have it fixed in a way that still provides the intermediate CA certificate to clients who need it (e.g., for ipa-getcert to be able to provide both a CertificateFile and a CertificateChainFile for mod_ssl).

2 - .crt files in that directory should not contain comments or any non-certificate data

Or ca-certificate should strip any extra data that it finds.

3 - .crt files in that directory should contain only one certificate

Some tools, like 'openssl x509' will only parse the first certificate in a file. If you're not aware of this then you might be mislead into trusting more certificates than you meant to!

(Yes, it would be good if ca-certificates could clarify the requirements for certificate files provided by other packages).

I've been revisiting this while trying to get some Debian machines to play nicely with our FreeIPA installation.

One of the thing that update-ca-certificates does is run 'openssl rehash' on /etc/ssl/certs to make the symlinks that OpenSSL uses to look up CA certificates without having to parse a huge CA certificate list file.

If ipa-ca.crt, which is symlinked into /etc/ssl/certs, contains multiple certificates then problems arise:

# openssl rehash /etc/ssl/certs
rehash: warning: skipping ipa-ca.pem,it does not contain exactly one certificate or CRL

One way to fix this would be to regard ipa-ca.crt as a file that's specific only to freeipa, and install it to a private directory such as /var/lib/freeipa. Then, for each certificate found within, create a file within /usr/local/share/ca-certificates/ipa; update-ca-certificates iterates over this directory recursively so each certificate within it will be symlinked into /etc/ssl/certs and be processed correctly by 'openssl rehash'.

We really need to make Debian-specific method for updating system wide store. No need to create these workarounds instead, they are fragile and don't give the same experience as we have on other platforms.

Well, I'm trying to propose such a method...

ca-certificates has 2 possible integration points. Either you drop .crt files into /usr/local/share/ca-certificates (one CA per file), which causes update-ca-certificates to trust them unconditionally; or you drop them into a subdirectory of /usr/share/ca-certificates (e.g., /usr/share/ca-certificates/freeipa, in which case they will be trusted unless the user disabled them in /etc/ca-certificates.conf.

How ipa-certupdate & friends decide to fulfil that contract is up to you. I am happy to send a proposed patch along the lines above if you don't want to discount it out of hand.

BTW, there is one extra case to think of. My installation at work has two certificates for the same CA (i.e., same subject DN) in the FreeIPA trust store. This is because the CA certificate was renewed, and when I ran ipa-cacert-manage on the new certificate, the old one was not removed from the directory (cn=certificates,cn=ipa,cn=etc,$prefix).

So when ipa-client-install and ipa-cacert-update want to write out the trusted certificates for consumption by update-ca-certificates, simply writing out one file for each certificate in the directory won't work: the old and the new certificates will have the same subject DN and hence openssl rehash will calculate the same hash.

Either ipa-cacert-manage needs to replace the original certificate in such a scenario (and something has to go through the trust store on upgrade and remove the older certificate with the same subject DN); or ipa-certupdate has to deduplicate what's in the directory per subject DN (e.g., pick the certificate with a notBefore date in the past && the notAfter date the furthest in the future).

FYI I filed https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=945274 against ca-certificates to discover whose job it is to split multiple CA certificates out into separate files -- ca-certificates or integrators. :)

Thank you, @yrro. I think we should be not producing p11-kit config for Debian platform but I agree it would be nice to fix ca-certificate to support modern ways of handling certificates.

On the other hand, your issue with not removed certificate sounds like a separate bug against FreeIPA.

I'm thinking debian should produce both, p11-kit config and a stripped-down version for ca-certificates. Just haven't made it happen yet.

Cool - I have filed #8124 re: the duplicate CA entry in the directory.

master:

  • 3985183 Debian: write out only one CA certificate per file

ipa-4-8:

  • 6fcc78b Debian: write out only one CA certificate per file

Metadata Update from @abbra:
- Issue close_status updated to: fixed
- Issue status updated to: Closed (was: Open)

3 months ago

Metadata Update from @abbra:
- Custom field changelog adjusted to On Debian-based platforms update-ca-certificates does not support multiple certificates in a single file. IPA installers now write individual files per each certificate for Debian-based platforms.

a month ago

Login to comment on this ticket.

Metadata