#9534 Nightly test failure (PKI) in test_integration/test_replica_promotion.py::TestSubCAkeyReplication
Closed: fixed a year ago by frenaud. Opened a year ago by amore.

Issue

The nightly test test_integration/test_replica_promotion.py::TestSubCAkeyReplication
TESTs:
test_subca_master
test_subca_replica
test_del_subca_master_on_replica
test_scale_add_subca
is failing in latest PKI build, see in PR 3391 with the following logs and report

Test test_subca_master and test_subca_replica is failing with error:

INFO     ipatests.pytest_ipa.integration.host.Host.master.IPAOpenSSHTransport:transport.py:391 RUN ['/usr/bin/certutil', '-d', '/etc/pki/pki-tomcat/alias', '-f', '/etc/pki/pki-tomcat/alias/pwdfile.txt', '-K', '-n', 'caSigningCert cert-pki-ca 77483854-14d3-4730-b32a-2cc5fef6bf4e']
DEBUG    ipatests.pytest_ipa.integration.host.Host.master.cmd76:transport.py:513 RUN ['/usr/bin/certutil', '-d', '/etc/pki/pki-tomcat/alias', '-f', '/etc/pki/pki-tomcat/alias/pwdfile.txt', '-K', '-n', 'caSigningCert cert-pki-ca 77483854-14d3-4730-b32a-2cc5fef6bf4e']
DEBUG    ipatests.pytest_ipa.integration.host.Host.master.cmd76:transport.py:557 certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services"
DEBUG    ipatests.pytest_ipa.integration.host.Host.master.cmd76:transport.py:557 certutil: problem listing keys: SEC_ERROR_INVALID_ARGS: security library: invalid arguments.
DEBUG    ipatests.pytest_ipa.integration.host.Host.master.cmd76:transport.py:217 Exit code: 255
ERROR    ipatests.pytest_ipa.integration.host.Host.master.cmd76:host.py:201 stderr: certutil: problem listing keys: SEC_ERROR_INVALID_ARGS: security library: invalid arguments.

Test test_del_subca_master_on_replica and test_scale_add_subca is failing with error:

INFO     ipatests.pytest_ipa.integration.host.Host.replica0.IPAOpenSSHTransport:transport.py:391 RUN ['ipa', 'ca-del', 'test_subca_master']
DEBUG    ipatests.pytest_ipa.integration.host.Host.replica0.cmd117:transport.py:513 RUN ['ipa', 'ca-del', 'test_subca_master']
DEBUG    ipatests.pytest_ipa.integration.host.Host.replica0.cmd117:transport.py:557 ipa: ERROR: Request failed with status 500: Non-2xx response from CA REST API: 500. 
DEBUG    ipatests.pytest_ipa.integration.host.Host.replica0.cmd117:transport.py:217 Exit code: 1
ERROR    ipatests.pytest_ipa.integration.host.Host.replica0.cmd117:host.py:201 stderr: ipa: ERROR: Request failed with status 500: Non-2xx response from CA REST API: 500.

Test scenario:
- enable PKI nightly build repo with dnf copr enable -y @pki/master
- install ipa server packages with dnf install -y freeipa-server
- configure ipa server with ipa-server-install --domain ipa.test --realm IPA.TEST -a Secret123 -p Secret123 -U
- create a sub ca with echo Secret123 | kinit admin; ipa ca-add test_subca_master --subject cn=test_subca_master --desc subca
Note the ca ID, for instance in this output the id is aece3f29-de70-41ce-b722-5d136e020089:

# ipa ca-add test_subca_master --subject cn=test_subca_master --desc subca
------------------------------
Created CA "test_subca_master"
------------------------------
  Name: test_subca_master
  Description: subca
  Authority ID: aece3f29-de70-41ce-b722-5d136e020089
  Subject DN: CN=test_subca_master
  Issuer DN: CN=Certificate Authority,O=IPA.TEST
  Certificate: MIIE...
  • try to display the subca key using certutil and the key nickname caSigningCert cert-pki-ca $ID:
# certutil -d /etc/pki/pki-tomcat/alias/ -f /etc/pki/pki-tomcat/alias/pwdfile.txt  -K -n 'caSigningCert cert-pki-ca aece3f29-de70-41ce-b722-5d136e020089'
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services"
certutil: problem listing keys: SEC_ERROR_INVALID_ARGS: security library: invalid arguments.

The key was created with a nickname composed of $token:caSigningCert cert-pki-ca $id instead of caSigningCert cert-pki-ca $id:

# certutil -d /etc/pki/pki-tomcat/alias/ -f /etc/pki/pki-tomcat/alias/pwdfile.txt  -K   
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services"
< 0> rsa      c1c86a51e5d8a0650a2883b8acac8f284921a128   NSS Certificate DB:caSigningCert cert-pki-ca
< 1> rsa      d2f34c5a32c7009a4a392fe8f2c939715b2a6a3f   NSS Certificate DB:ocspSigningCert cert-pki-ca
< 2> rsa      ebbf010cf4961844245a27bb298c5a2df183a21e   NSS Certificate DB:Server-Cert cert-pki-ca
< 3> rsa      f916c9944ce07184c8e609bd2654908fa7adca3d   NSS Certificate DB:subsystemCert cert-pki-ca
< 4> rsa      740c7f1572456292b867b58e0b60ab7e312366a5   NSS Certificate DB:auditSigningCert cert-pki-ca
< 5> rsa      c431be7f804cf3148b6174344cab6f701a6619d1   NSS Certificate DB:caSigningCert cert-pki-ca aece3f29-de70-41ce-b722-5d136e020089

The issue is in the test code that wrongly assumes the key nickname. It should be more robust and support either caSigningCert cert-pki-ca $idor $token:caSigningCert cert-pki-ca $id.

FWIW this is likely related to changes they made to support generated the keys on HSM tokens.

Metadata Update from @frenaud:
- Issue assigned to frenaud

a year ago

Metadata Update from @frenaud:
- Custom field on_review adjusted to https://github.com/freeipa/freeipa/pull/7232

a year ago

I wonder if this could be considered a regression. Typically one does not need to manually specify the token name for the NSS softoken. It is implied. Within PKI it refers to the default token name as "internal". See /etc/pki/pki-tomcat/password.conf as an example.

The issue resolved itself with recent updates on f40. I did not track down which package was the root cause but the test is now green (Report).
Closing as fixed.

Metadata Update from @frenaud:
- Issue close_status updated to: fixed
- Issue status updated to: Closed (was: Open)

a year ago

@frenaud Test is failing on f39 with report

@amore in the provided report, the error is not happening in a call to certutil -K but when adding subcas in a loop, with ipa: ERROR: Request failed with status 500: Non-2xx response from CA REST API: 500. This is reported in dogtagpki issue #4677 Freeipa: adding subca in loop fails with Non-2xx response from CA REST API: 500

Log in to comment on this ticket.

Metadata