#90 ipa-server-install breaks if subject base RDN has an escaped comma
Closed: fixed 4 months ago by rcritten. Opened a year ago by rcritten.

Originally filed as https://pagure.io/freeipa/issue/7347

Request for enhancement

ipa-server-install should work when a custom RDN is used, when the RDN contains escaped commas. Alternatively, if this is not possible due to interactions with Dogtag code, the documentation should expressly indicate that such escaped values cannot be used.


When running ipa-server-install with the --subject-base option to override the default realm name as the organization, a comma that is properly escaped will cause an authentication failure later in the setup script.

One legitimate reason that the RDN would include a comma is a corporation's legal name — for example, "O=Acme\, Inc.,ST=Massachusetts,C=US". According to RFC 4514, commas can be escaped with a prefixed backslash (section 2.4; examples in section 4).

Steps to Reproduce

  1. On a clean minimal CentOS installation, install the FreeIPA packages with yum install ipa-server from the default repositories. Ensure that the hostname for the local system has been added to /etc/hosts.
  2. Invoke ipa-server-install with a --subject-base option whose value contains an escaped comma.
  3. Accept default values for other installation prompts.

Actual behavior

The installation proceeds, but halts with an error at step 25 of the CA setup with the following message:

Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes
  [1/29]: configuring certificate server instance
  [2/29]: exporting Dogtag certificate store pin
  [3/29]: stopping certificate server instance to update CS.cfg
  [2/29]: exporting Dogtag certificate store pin
  [3/29]: stopping certificate server instance to update CS.cfg
  [4/29]: backing up CS.cfg
  [5/29]: disabling nonces
  [6/29]: set up CRL publishing
  [7/29]: enable PKIX certificate path discovery and validation
  [8/29]: starting certificate server instance
  [9/29]: configure certmonger for renewals
  [10/29]: requesting RA certificate from CA
  [11/29]: setting up signing cert profile
  [12/29]: setting audit signing renewal to 2 years
  [13/29]: restarting certificate server
  [14/29]: publishing the CA certificate
  [15/29]: adding RA agent as a trusted user
  [16/29]: authorizing RA to modify profiles
  [17/29]: authorizing RA to manage lightweight CAs
  [18/29]: Ensure lightweight CAs container exists
  [19/29]: configure certificate renewals
  [20/29]: configure Server-Cert certificate renewal
  [21/29]: Configure HTTP to proxy connections
  [22/29]: restarting certificate server
  [23/29]: updating IPA configuration
  [24/29]: enabling CA instance
  [25/29]: migrating certificate profiles to LDAP
  [error] RemoteRetrieveError: Failed to authenticate to CA REST API
ipa.ipapython.install.cli.install_tool(CompatServerMasterInstall): ERROR    Failed to authenticate to CA REST API
ipa.ipapython.install.cli.install_tool(CompatServerMasterInstall): ERROR    The ipa-server-install command failed. See /var/log/ipaserver-install.log for more information

The certificate authority has already been configured at this point. A valid /etc/ipa/ca.crt has already been created, whose issuer and subject names are properly formatted.

Expected behavior

The installation should succeed.



Additional info:

Log file locations: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Linux_Domain_Identity_Authentication_and_Policy_Guide/config-files-logs.html
Troubleshooting guide: https://www.freeipa.org/page/Troubleshooting


Here's the generated CA certificate, showing that the base DN has been properly applied (meaning that the escaped comma was properly processed here).

The contents of /var/log/pki/pki-tomcat/ca/system show that it was "IPA RA" that failed to authenticate — and reveal the jumbled subject name given to that certificate:

0.localhost-startStop-1 - [04/Jan/2018:05:15:52 UTC] [13] [3] authz instance DirAclAuthz initialization failed and skipped, error=Property internaldb.ldapconn.port missing value
0.http-bio-8443-exec-3 - [04/Jan/2018:05:17:03 UTC] [3] [3] Cannot build CA chain. Error java.security.cert.CertificateException: Certificate is not a PKCS #11 certificate
0.http-bio-8443-exec-3 - [04/Jan/2018:05:17:06 UTC] [3] [3] CASigningUnit: Object certificate not found. Error Certificate object not found
0.Thread-14 - [04/Jan/2018:05:17:38 UTC] [8] [3] Publishing: Could not publish certificate serial number 0x7. Error Failed to publish using rule: No rules enabled
0.http-bio-8443-exec-3 - [04/Jan/2018:05:18:35 UTC] [6] [3] Cannot authenticate agent with certificate Serial 0x7 Subject DN CN=IPA RA,CN=Inc.,O=Acme\\,ST=Massachusetts,C=US. Error: User not found

Notice: CN=IPA RA,CN=Inc.,O=Acme\\,ST=Massachusetts,C=US

Similarly, /var/log/pki/pki-tomcat/ca/transactions shows that when the IPA RA certificate was first enrolled, the requested distinguished name already had this problem:

0.http-bio-8443-exec-5 - [04/Jan/2018:05:17:38 UTC] [20] [1] enrollment reqID 7 fromAgent userID: admin authenticated by certUserDBAuthMgr is completed DN requested: CN=IPA RA,CN=Inc.,O=Acme\\,ST=Massachusetts,C=US cert issued serial number: 0x7 time: 235

I've also attached the dogtag setup log.

I think I've traced it through to a certmonger issue.

How I got there:

1) ipaserver/install/cainstance.py is responsible for this particular enrollment:

            reqId = certmonger.request_and_wait_for_cert(
                certpath=(paths.RA_AGENT_PEM, paths.RA_AGENT_KEY),
                principal='host/%s' % self.fqdn,
                subject=str(DN(('CN', 'IPA RA'), self.subject_base)),

This isn't the problem. The assembled DN containing the subject base is fine; ipapython/dn.py smartly accounts for escaping. In the example I gave above, minus the state (to make these examples more convenient to illustrate), the subject is appropriately assembled as ('CN','IPA RA'),('O','Acme\, Inc.'),('C','US').

2) However, the parameter above is cast back to a string, instead of being passed as a DN object to certmonger.request_and_wait_for_cert().

Once passed to the certmonger module ipalib/install/certmonger.py, it gets switched back to a DN, reversed, and then back to a string again... this "workaround":

subject = str(DN(*reversed(DN(subject))))

Still not the problem. Despite the list operation and type changes, the ipapython/dn.py module is intelligent, so this line reverses the order of AVAs without damaging the escaping. At this point, the subject is still appropriately grouped, just reversed, as ('C','US'),('O','Acme\, Inc.'),('CN','IPA RA').

3) The Python code eventually sends this still okay DN string, in reversed order, to certmonger over dbus or a socket... something like (string) C=US,O=Acme\, Inc.,CN=IPA RA.

4) As certmonger#62 says, certmonger implements the subject field differently depending on the destined storage. Here, since storage="FILE", it's going to use the OpenSSL implementation in csrgen-o.c. (Unlike the NSS implementation in csrgen-n.c, which can invoke a library helper function CERT_AsciiToName(), the OpenSSL implementation is forced to do its own parsing of the DN, one AVA at a time...)

5) Which brings me to the certmonger source:

/* This isn't really correct, but it will
    * probably do for now. */
p = entry->cm_template_subject;
q = p + strcspn(p, ",");
subject = X509_NAME_new();
if (subject != NULL) {
    while (*p != '\0') {

That's it, then; because csrgen-o.c in certmonger naively splits the subject string by commas, all the hard work in ipapython/dn.py was for naught. Despite ipapython/dn.py's intelligent handling of RDN attributes, certmonger just treated it as a string without accounting for escaping or multivalue.

6) Notably, when the code excerpted above splits a string like C=US,O=Acme\, Inc.,CN=IPA RA, it's going to see C=US; O=Acme\; Inc.; CN=IPA RA. And lines 213-218 treat " Inc." as its own common name value:

    "CN", astring_type("CN", p, q - p),
    (unsigned char *) p, q - p,
    -1, 0);

I'm assuming the OpenSSL library adds its own escaping to the dangling \, resulting in the final, observed subject of CN=IPA RA,CN=Inc.,O=Acme\\,C=US.

As I see it, the simple workaround for users is to try the \2c escape code for the comma instead — i.e. O=Acme\2c Inc.,C=US, which the naive C code in certmonger won't damage. But this is definitely not an obvious requirement of --subject-base.

A smarter fix might avoid passing the information as a string. certmonger's code avoids the naive comma-delimiter parsing if a subject is provided in DER.

            if ((entry->cm_template_subject_der != NULL) &&
                (strlen(entry->cm_template_subject_der) != 0)) {
                i = strlen(entry->cm_template_subject_der);
                name = malloc(i);
                if (name != NULL) {
                    i = cm_store_hex_to_bin(entry->cm_template_subject_der,
                                name, i);
                    nametmp = name;
                    subject = d2i_X509_NAME(NULL, &nametmp, i);
            if ((subject == NULL) &&
/* ... go on to split by commas ... */

That might be a more optimal way of passing the information from Python to certmonger.

It seems to me that certmonger is re-inventing the wheel here.

certmonger already links in ldap which provides ldap_str2dn() so it should be straightforward to convert the string subject into an LDAP DN and then walk over that to create the X509 subject.

As it turns out it looks like the NSS call CERT_AsciiToName() also does the wrong thing with escaped commas (though it got saved properly).

The LDAP method I came up with ended up being issued as:

template_subject=CN=ipa.example.com,O="Acme, Inc.",ST=Massachusetts,C=US

The NSS method fails to issue with this template_subject:

template_subject=CN=ipa.example.com,O=Acme\, Inc.,ST=Massachusetts,C=US

I'll try to talk to one of the dogtag devs.

After talking with ftweedal this may be a bug either in dogtag, IPA, both and/or certmonger. He is going to do some further investigation. He thinks the CSR generated by NSS looks ok.

When installing with --subject-base 'OU=201801171517,O=Red Hat\, Inc.', the following CSR
was generated by certmonger and sent to Dogtag:


It has ASN.1 DN structure:

   11:d=2  hl=2 l=  75 cons:   SEQUENCE                                    
   13:d=3  hl=2 l=  17 cons:    SET                 
   15:d=4  hl=2 l=  15 cons:     SEQUENCE           
   17:d=5  hl=2 l=   3 prim:      OBJECT            :organizationName      
   22:d=5  hl=2 l=   8 prim:      UTF8STRING        :Red Hat\              
   32:d=3  hl=2 l=  14 cons:    SET                 
   34:d=4  hl=2 l=  12 cons:     SEQUENCE           
   36:d=5  hl=2 l=   3 prim:      OBJECT            :commonName            
   41:d=5  hl=2 l=   5 prim:      PRINTABLESTRING   : Inc.                 
   48:d=3  hl=2 l=  21 cons:    SET                 
   50:d=4  hl=2 l=  19 cons:     SEQUENCE                                  
   52:d=5  hl=2 l=   3 prim:      OBJECT            :organizationalUnitName
   57:d=5  hl=2 l=  12 prim:      UTF8STRING        :201801171517          
   71:d=3  hl=2 l=  15 cons:    SET                 
   73:d=4  hl=2 l=  13 cons:     SEQUENCE                                  
   75:d=5  hl=2 l=   3 prim:      OBJECT            :commonName            
   80:d=5  hl=2 l=   6 prim:      PRINTABLESTRING   :IPA RA                

Dogtag processes this CSR just fine (for the particular profile used for the IPA RA certificate),
but Certmonger generated the CSR wrong.

Looking at the request file under /var/lib/certmonger/requests, I see:

template_subject=O=Red Hat\, Inc.,OU=201801171517,CN=IPA RA

So FreeIPA has conveyed the correct Subject DN to Certmonger when creating the request.

Testing with three new file-based (OpenSSL-based) certificate requests created using the following commands:

% getcert request -f /tmp/tmp2.pem -k /tmp/tmp2.key -G rsa -g 1024 -N "o=Acme\, Inc." -I acme1     
New signing request "acme1" added.
% getcert request -f /tmp/tmp3.pem -k /tmp/tmp3.key -G rsa -g 1024 -N "ou=HR,o=Acme\, Inc." -I acme2
New signing request "acme2" added.
% getcert request -f /tmp/tmp4.pem -k /tmp/tmp4.key -G rsa -g 1024 -N "o=Acme\, Inc.,C=AU" -I acme3 
New signing request "acme3" added.

Resulted in the following (stringified) DNs in the CSRs (as displayed by openssl req -text):

O = Acme\\, CN = " Inc."
OU = HR, O = Acme\\, CN = " Inc."
O = Acme\\, CN = " Inc.", C = AU

All of these are incorrect parses of the string given in -N.

The parser for FILES just looks for comma tokens so is very simplistic. I have a candidate patch which should handle this better.


OK, reviewing the patch.

  1. It ignores multiple AVA RDNs (only takes rdn[0]). If we're already going this far we might as well fix that properly at this time.

  2. It seems like attribute strings (o, cn, etc) need to be upper cased for X509_NAME_add_entry_by_txt to recognise them. We should do this upper casing. we should probably also check the return value and log if it didn't work, instead of silenty dropping the AVA.

@rcritten here's a patch that will apply on top of yours that:

  • adds support for multi-attribute RDNs
  • if the attribute name is unrecognised, upper-cases it and tries again
  • logs unrecognised (after retry) attribute names, where currently they are silently dropped


And here's another patch on top of that one, that extracts the X509_NAME creation
procedures into separate subroutines:


Thanks for the patches!

I'd like to wait until we can do an end-to-end install in IPA before pushing these upstream in case we find anything that needs additional changes. I'll do a proper PR along with your patches when we get to that point.

That could be a little while... it seems that there's a bug in 389DS certmap code too, as well as the identified Dogtag and Certmonger issues.

@rcritten there a PR for the 389 fix now:

I've tested this now with the patched versions of Dogtag, Certmonger and 389.
So I think we can finish reviewing and merge the Certmonger patches hopefully
ASAP now.

Aged in oak. Can we re-review this issue and (hopefully) merge soon?

Fix pushed to master:

632fc96c test handling of DNs with escaped chars
bf91c284 csrgen-o: extract X509_NAME creation subroutines
47a60740 csrgen-o: handle multi-value RDNs and log dropped AVAs

@rcritten we can close this now. And also could you cut a new release and
Fedora build at your earliest convenience?

Metadata Update from @rcritten:
- Issue close_status updated to: fixed
- Issue status updated to: Closed (was: Open)

4 months ago

Metadata Update from @rcritten:
- Issue assigned to rcritten
- Issue set to the milestone: 0.79

4 months ago

Login to comment on this ticket.