#7347 ipa-server-install breaks if subject base RDN has an escaped comma
Closed: fixed 5 years ago by cheimes. Opened 6 years ago by frederickding.

Request for enhancement

ipa-server-install should work when a custom RDN is used, when the RDN contains escaped commas. Alternatively, if this is not possible due to interactions with Dogtag code, the documentation should expressly indicate that such escaped values cannot be used.

Issue

When running ipa-server-install with the --subject-base option to override the default realm name as the organization, a comma that is properly escaped will cause an authentication failure later in the setup script.

One legitimate reason that the RDN would include a comma is a corporation's legal name — for example, "O=Acme\, Inc.,ST=Massachusetts,C=US". According to RFC 4514, commas can be escaped with a prefixed backslash (section 2.4; examples in section 4).

Steps to Reproduce

  1. On a clean minimal CentOS installation, install the FreeIPA packages with yum install ipa-server from the default repositories. Ensure that the hostname for the local system has been added to /etc/hosts.
  2. Invoke ipa-server-install with a --subject-base option whose value contains an escaped comma.
  3. Accept default values for other installation prompts.

Actual behavior

The installation proceeds, but halts with an error at step 25 of the CA setup with the following message:

Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes
  [1/29]: configuring certificate server instance
  [2/29]: exporting Dogtag certificate store pin
  [3/29]: stopping certificate server instance to update CS.cfg
  [2/29]: exporting Dogtag certificate store pin
  [3/29]: stopping certificate server instance to update CS.cfg
  [4/29]: backing up CS.cfg
  [5/29]: disabling nonces
  [6/29]: set up CRL publishing
  [7/29]: enable PKIX certificate path discovery and validation
  [8/29]: starting certificate server instance
  [9/29]: configure certmonger for renewals
  [10/29]: requesting RA certificate from CA
  [11/29]: setting up signing cert profile
  [12/29]: setting audit signing renewal to 2 years
  [13/29]: restarting certificate server
  [14/29]: publishing the CA certificate
  [15/29]: adding RA agent as a trusted user
  [16/29]: authorizing RA to modify profiles
  [17/29]: authorizing RA to manage lightweight CAs
  [18/29]: Ensure lightweight CAs container exists
  [19/29]: configure certificate renewals
  [20/29]: configure Server-Cert certificate renewal
  [21/29]: Configure HTTP to proxy connections
  [22/29]: restarting certificate server
  [23/29]: updating IPA configuration
  [24/29]: enabling CA instance
  [25/29]: migrating certificate profiles to LDAP
  [error] RemoteRetrieveError: Failed to authenticate to CA REST API
ipa.ipapython.install.cli.install_tool(CompatServerMasterInstall): ERROR    Failed to authenticate to CA REST API
ipa.ipapython.install.cli.install_tool(CompatServerMasterInstall): ERROR    The ipa-server-install command failed. See /var/log/ipaserver-install.log for more information

The certificate authority has already been configured at this point. A valid /etc/ipa/ca.crt has already been created, whose issuer and subject names are properly formatted.

Expected behavior

The installation should succeed.

Version/Release/Distribution

ipa-server-4.5.0-22.el7.centos.x86_64
ipa-client-4.5.0-22.el7.centos.x86_64
389-ds-base-1.3.6.1-24.el7_4.x86_64
pki-ca-10.4.1-17.el7_4.noarch
krb5-server-1.15.1-8.el7.x86_64

Additional info:

Log file locations: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Linux_Domain_Identity_Authentication_and_Policy_Guide/config-files-logs.html
Troubleshooting guide: https://www.freeipa.org/page/Troubleshooting

ipaserver-install.log


Here's the generated CA certificate, showing that the base DN has been properly applied (meaning that the escaped comma was properly processed here).
/etc/ipa/ca.crt

The contents of /var/log/pki/pki-tomcat/ca/system show that it was "IPA RA" that failed to authenticate — and reveal the jumbled subject name given to that certificate:

0.localhost-startStop-1 - [04/Jan/2018:05:15:52 UTC] [13] [3] authz instance DirAclAuthz initialization failed and skipped, error=Property internaldb.ldapconn.port missing value
0.http-bio-8443-exec-3 - [04/Jan/2018:05:17:03 UTC] [3] [3] Cannot build CA chain. Error java.security.cert.CertificateException: Certificate is not a PKCS #11 certificate
0.http-bio-8443-exec-3 - [04/Jan/2018:05:17:06 UTC] [3] [3] CASigningUnit: Object certificate not found. Error Certificate object not found
0.Thread-14 - [04/Jan/2018:05:17:38 UTC] [8] [3] Publishing: Could not publish certificate serial number 0x7. Error Failed to publish using rule: No rules enabled
0.http-bio-8443-exec-3 - [04/Jan/2018:05:18:35 UTC] [6] [3] Cannot authenticate agent with certificate Serial 0x7 Subject DN CN=IPA RA,CN=Inc.,O=Acme\\,ST=Massachusetts,C=US. Error: User not found

Notice: CN=IPA RA,CN=Inc.,O=Acme\\,ST=Massachusetts,C=US

Similarly, /var/log/pki/pki-tomcat/ca/transactions shows that when the IPA RA certificate was first enrolled, the requested distinguished name already had this problem:

0.http-bio-8443-exec-5 - [04/Jan/2018:05:17:38 UTC] [20] [1] enrollment reqID 7 fromAgent userID: admin authenticated by certUserDBAuthMgr is completed DN requested: CN=IPA RA,CN=Inc.,O=Acme\\,ST=Massachusetts,C=US cert issued serial number: 0x7 time: 235

I've also attached the dogtag setup log.
/var/log/pki/pki-ca-spawn.20180104051539.log

Thanks for the detailed report! Someone (probably me) will look into this, hopefully in the next few days.

I think I've traced it through to a certmonger issue.

How I got there:

1) ipaserver/install/cainstance.py is responsible for this particular enrollment:

            reqId = certmonger.request_and_wait_for_cert(
                certpath=(paths.RA_AGENT_PEM, paths.RA_AGENT_KEY),
                principal='host/%s' % self.fqdn,
                subject=str(DN(('CN', 'IPA RA'), self.subject_base)),
                ...
                storage="FILE")

This isn't the problem. The assembled DN containing the subject base is fine; ipapython/dn.py smartly accounts for escaping. In the example I gave above, minus the state (to make these examples more convenient to illustrate), the subject is appropriately assembled as ('CN','IPA RA'),('O','Acme\, Inc.'),('C','US').

2) However, the parameter above is cast back to a string, instead of being passed as a DN object to certmonger.request_and_wait_for_cert().

Once passed to the certmonger module ipalib/install/certmonger.py, it gets switched back to a DN, reversed, and then back to a string again... this "workaround":

subject = str(DN(*reversed(DN(subject))))

Still not the problem. Despite the list operation and type changes, the ipapython/dn.py module is intelligent, so this line reverses the order of AVAs without damaging the escaping. At this point, the subject is still appropriately grouped, just reversed, as ('C','US'),('O','Acme\, Inc.'),('CN','IPA RA').

3) The Python code eventually sends this still okay DN string, in reversed order, to certmonger over dbus or a socket... something like (string) C=US,O=Acme\, Inc.,CN=IPA RA.

4) As certmonger#62 says, certmonger implements the subject field differently depending on the destined storage. Here, since storage="FILE", it's going to use the OpenSSL implementation in csrgen-o.c. (Unlike the NSS implementation in csrgen-n.c, which can invoke a library helper function CERT_AsciiToName(), the OpenSSL implementation is forced to do its own parsing of the DN, one AVA at a time...)

5) Which brings me to the certmonger source:

/* This isn't really correct, but it will
    * probably do for now. */
p = entry->cm_template_subject;
q = p + strcspn(p, ",");
subject = X509_NAME_new();
if (subject != NULL) {
    while (*p != '\0') {
...

That's it, then; because csrgen-o.c in certmonger naively splits the subject string by commas, all the hard work in ipapython/dn.py was for naught. Despite ipapython/dn.py's intelligent handling of RDN attributes, certmonger just treated it as a string without accounting for escaping or multivalue.

6) Notably, when the code excerpted above splits a string like C=US,O=Acme\, Inc.,CN=IPA RA, it's going to see C=US; O=Acme\; Inc.; CN=IPA RA. And lines 213-218 treat " Inc." as its own common name value:

X509_NAME_add_entry_by_txt(subject,
    "CN", astring_type("CN", p, q - p),
    (unsigned char *) p, q - p,
    -1, 0);

I'm assuming the OpenSSL library adds its own escaping to the dangling \, resulting in the final, observed subject of CN=IPA RA,CN=Inc.,O=Acme\\,C=US.

As I see it, the simple workaround for users is to try the \2c escape code for the comma instead — i.e. O=Acme\2c Inc.,C=US, which the naive C code in certmonger won't damage. But this is definitely not an obvious requirement of --subject-base.

A smarter fix might avoid passing the information as a string. certmonger's code avoids the naive comma-delimiter parsing if a subject is provided in DER.

            if ((entry->cm_template_subject_der != NULL) &&
                (strlen(entry->cm_template_subject_der) != 0)) {
                i = strlen(entry->cm_template_subject_der);
                name = malloc(i);
                if (name != NULL) {
                    i = cm_store_hex_to_bin(entry->cm_template_subject_der,
                                name, i);
                    nametmp = name;
                    subject = d2i_X509_NAME(NULL, &nametmp, i);
                }
            }
            if ((subject == NULL) &&
/* ... go on to split by commas ... */

That might be a more optimal way of passing the information from Python to certmonger.

Great job troubleshooting this!

It seems to me that certmonger is re-inventing the wheel here.

certmonger already links in ldap which provides ldap_str2dn() so it should be straightforward to convert the string subject into an LDAP DN and then walk over that to create the X509 subject.

This is really a certmonger issue so I'm going to close this and move the discussion instead to https://pagure.io/certmonger/issue/90

Metadata Update from @rcritten:
- Issue close_status updated to: duplicate
- Issue status updated to: Closed (was: Open)

6 years ago

Metadata Update from @ftweedal:
- Issue assigned to ftweedal
- Issue status updated to: Open (was: Closed)

6 years ago

Re-opening.

While testing the updated OpenSSL code the request is failing on a code path that should be using the NSS parser.

I tested by installing on F-27 from IPA master branch using:

# ipa-server-install -a password -p password -r EXAMPLE.COM --subject-base "O=Acme\, Inc.,ST=Massachusetts,C=US" -U

It failed with certmonger logging the error:

Invalid Subject Name CN=ipa.example.com, O=Acme, Inc.,ST=Massachusetts,C=US

dogtag logged this while processing the request:

EnrollProfile: fillPKCS10: Finish parsePKCS10 - CN=ipa.example.com,O=Acme\, Inc.,ST=Massachusetts,C=US
...
subjectName=CN=ipa.example.com, O=Acme, Inc.,ST=Massachusetts,C=US
SubjectNameDefault: failed to populate: java.io.IOException: Unknown AVA keyword 'INC.,ST'.

-----BEGIN NEW CERTIFICATE REQUEST-----
MIID0DCCArgCAQAwVDELMAkGA1UEBhMCVVMxFjAUBgNVBAgTDU1hc3NhY2h1c2V0
dHMxEzARBgNVBAoTCkFjbWUsIEluYy4xGDAWBgNVBAMTD2lwYS5leGFtcGxlLmNv
bTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALY1hRUBT2mP5lMeRTtj
2+OcDCW4dAF0NaXgTlh/MbCaViGZSScmXuH0884lrpr/nESlG1FLeVG3XXh+aNxq
jF3Q/z4A7UfaEDYBltpvKHkuhbyHkKFRnpFmcxoSJ53rpmew7ONjKHBRfF/rIuaQ
jZi+aHCZ7jmlSyYThf7KFAg5WOcIEYp0xN5k7LEv9nuD9DmokQkIcnl6nOeQXpe6
a0B5ALKydK+H21fF2Yh6iuc+mvnRfQr3vDCprAnYD40KzM4JOZrm/YzIrmrevKlr
tk4PG11J5lmKncHR1PNPOk0Eza/KFbkXBOYpG6fu9f8/Woqb0mqvPzGmyqHctdjY
+YkCAwEAAaCCATUwJQYJKoZIhvcNAQkUMRgeFgBTAGUAcgB2AGUAcgAtAEMAZQBy
AHQwggEKBgkqhkiG9w0BCQ4xgfwwgfkwgZIGA1UdEQEBAASBhzCBhIIPaXBhLmV4
YW1wbGUuY29toDAGCisGAQQBgjcUAgOgIgwgbGRhcC9pcGEuZXhhbXBsZS5jb21A
RVhBTVBMRS5DT02gPwYGKwYBBQICoDUwM6ANGwtFWEFNUExFLkNPTaEiMCCgAwIB
AaEZMBcbBGxkYXAbD2lwYS5leGFtcGxlLmNvbTAMBgNVHRMBAf8EAjAAMCAGA1Ud
DgEBAAQWBBTiVAqGBHh8i6+6IQPpBJgiVYnU0TAyBgkrBgEEAYI3FAIBAQAEIh4g
AGMAYQBJAFAAQQBzAGUAcgB2AGkAYwBlAEMAZQByAHQwDQYJKoZIhvcNAQELBQAD
ggEBAGTBSHfLBoW05Jpnj+/HH0moP5QMz5DgANbSdtBfOezo+3N8Q3dno7oxOyHG
oUlPF+xiIi1YOw+2QpeF/VFDfpzKe3pSZr2topcg8xvsIbiOtg9xJUuvJd17qW38
FfAhf8mncNF6a6HdN9kRvkagVg/Xdkk23F5ze5vH4g9KmBlxCeQ2fxgxC5F5aX1q
mKiuPglPH75rm/n5Uy25OjOB0ih6TbATN+QKSEKKNm6cjdflByr9cWV09ApWlqhx
TIXHXF31ly67jc4hXLglH2MSXxJRoYhpDNGTnqltZ3fXxbYtN4xu/wBhN/827alk
ThI5/PzhQWZfU3T8gtqRKcSHIHM=
-----END NEW CERTIFICATE REQUEST-----

Reopening... there may be bugs in multiple programs, including possibly IPA.
Further investigation needed.

@rcritten have you made progress on the certmonger issue? If you haven't started yet, I'll have a crack.

Based on our last discussion I was under the impression the issue is in dogtag.

You had said the CSR looked ok so I stopped looking.

I have mitigation for the reported issue with OpenSSL.

@rcritten I cannot reproduce the error you encountered, but I am encountering errors
that seem to be caused by Certmonger. (It may be that those errors are reproducible
but the current errors I'm hitting are getting in the way). I dumped more info in
https://pagure.io/certmonger/issue/90.

Feel free to attach your /var/log/pki/pki-tomcat/ca/debug (and other logs) so I can get a clearer picture of what the system was doing when that AVA error was thrown (e.g. what certificate was being requested at the time, etc).

@rcritten what is this mitigation of which you speak? Do you have a link, or patches I can test?

You likely need the patch from https://pagure.io/certmonger/issue/90 to see what I was seeing.

@rcritten yes, with that Certmonger patch I now proceed to the error you
described :)

Confirmed that the problem is in the Dogtag ProfileService class;
it uses the Properties class which strips the backslash from configuration
values. Investigating whether just switching to SimpleProperties resolves
the issue (and does introduce other issues).

@ftweedal would a simpler fix be to automatically replace escaped commas with \2c in IPA?

@rcritten no, these are real bugs in the other programs, not in IPA**.

(**Well there might be / probably are bugs in IPA around this too, but the other programs
should be updated to handle RFC 4514 DN string representations).

The issues probably go beyond commas to other special characters too (plus, pound,
equal, angle brackets etc).

Fix for the Dogtag issue (https://pagure.io/dogtagpki/issue/2909) has been pushed to master
and will be released in v10.6.

Fix for 389ds issue (https://pagure.io/389-ds-base/issue/49543) is not started yet. I intend to work with @firstyear on that but he is on PTO this week.

Metadata Update from @rcritten:
- Issue priority set to: normal
- Issue set to the milestone: FreeIPA 4.8
- Issue tagged with: tracker

6 years ago

I think https://pagure.io/dogtagpki/issue/2379 / https://bugzilla.redhat.com/show_bug.cgi?id=1346433 is a duplicate. The op has requested a subject with O=AmTrust North America\, Inc., please note the comma in the organization string.

Thanks @cheimes, I closed it as dup and updated the corresponding BZ.

Fixes for Dogtag, 389 and Certmonger (https://pagure.io/certmonger/pull-request/108) have been merged. All we need to do now is add some tests for the scenario to FreeIPA, then we can close this one out.

Notes about which released versions contain needed fixes:

  • 389-ds-base 1.4.0.20 and 1.3.9.1. (Patch also merged to 1.3.8 branch but no tag yet).

  • certmonger 0.79.7 (version bump and tag yet to be pushed to repo; also waiting on f28 and f29 builds)

  • pki-core 10.5.5

master:

  • 98b7fbe Fix installation when CA subject DN has escapes

ipa-4-7:

  • 487ab52 Fix installation when CA subject DN has escapes

Metadata Update from @cheimes:
- Issue close_status updated to: fixed
- Issue status updated to: Closed (was: Open)

5 years ago

At the time this was closed, there was not yet an integration test.
But a few months ago, the integration test we added for
https://pagure.io/freeipa/issue/8084 also covers this scenario.
The specific commit is:
https://pagure.io/freeipa/c/0a0e802bd47188fe31d6bf02b28ef0ea51567194

Login to comment on this ticket.

Metadata
Attachments 3