#6898 Lightweight sub CA replication destroyed Dogtag's NSSDB
Closed: insufficientinfo 6 years ago Opened 6 years ago by cheimes.

While I was testing SELinux policies for ipa-custodia, I managed to completely destroy Dogtag's NSSDB with all its private keys on the primary master. As a consequence Dogtag is no longer able to start. It's stuck in an endless lseek() + read() loop with 100% CPU usage on a single core.

setup

  • vm-012 primary master with CA and KRA. This machine has the broken NSSDB.
  • vm-058-111 replica with CA and KRA

I created subca1 and subca3 on vm-058-111, subca2 and subca4 on vm-012. The creation of further sub cas on vm-012 (subca5 and subca6) failed. I don't recall the exact error message.

output on vm-058-111

$ ipa ca-find
-------------
5 CAs matched
-------------
  Name: ipa
  Description: IPA CA
  Authority ID: c5863c57-1c50-4b7e-b559-098c3bd3f274
  Subject DN: CN=Certificate Authority,O=IPA.EXAMPLE
  Issuer DN: CN=Certificate Authority,O=IPA.EXAMPLE

  Name: subca1
  Authority ID: 48c255d1-47cf-40f2-9d9e-b0524149669e
  Subject DN: CN=subca1,O=IPA.EXAMPLE
  Issuer DN: CN=Certificate Authority,O=IPA.EXAMPLE

  Name: subca2
  Authority ID: 7d46b7fc-02d5-44e2-85a0-2c0ad14a16f7
  Subject DN: CN=subca2,O=IPA.EXAMPLE
  Issuer DN: CN=Certificate Authority,O=IPA.EXAMPLE

  Name: subca3
  Authority ID: d944efbf-b832-4bf4-b791-df1e7015e5f8
  Subject DN: CN=subca3,O=IPA.EXAMPLE
  Issuer DN: CN=Certificate Authority,O=IPA.EXAMPLE

  Name: subca4
  Authority ID: 72eba70f-c7b1-463a-bbd9-99ff1c685f9a
  Subject DN: CN=subca4,O=IPA.EXAMPLE
  Issuer DN: CN=Certificate Authority,O=IPA.EXAMPLE
----------------------------
Number of entries returned 5
----------------------------
$ certutil -d /etc/pki/pki-tomcat/alias/ -L

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

auditSigningCert cert-pki-ca                                 u,u,Pu
caSigningCert cert-pki-ca                                    CTu,Cu,Cu
Server-Cert cert-pki-ca                                      u,u,u
storageCert cert-pki-kra                                     u,u,u
caSigningCert cert-pki-ca 48c255d1-47cf-40f2-9d9e-b0524149669e u,u,u
ocspSigningCert cert-pki-ca                                  u,u,u
auditSigningCert cert-pki-kra                                u,u,Pu
subsystemCert cert-pki-ca                                    u,u,u
transportCert cert-pki-kra                                   u,u,u
caSigningCert cert-pki-ca d944efbf-b832-4bf4-b791-df1e7015e5f8 u,u,u

symptoms

pki-tomcat start is stuck

$ journalctl -u pki-tomcatd@pki-tomcat.service
Apr 26 07:51:52 vm-012.ipa.example server[2580]: PKIListener: org.apache.catalina.core.StandardServer[before_init]

pstack / strace

$ pstack $(pidof java)
    Thread 20 (Thread 0x7f47e2c58700 (LWP 2603)):
    #0  0x00007f47e2839d8d in lseek64 () from /lib64/libpthread.so.0
    #1  0x00007f47c11acf25 in __get_page () from /lib64/libnssdbm3.so
    #2  0x00007f47c11ab8c9 in __get_buf () from /lib64/libnssdbm3.so
    #3  0x00007f47c11a9be9 in hash_seq () from /lib64/libnssdbm3.so
    #4  0x00007f47c1197dbe in keydb_Seq.isra.0 () from /lib64/libnssdbm3.so
    #5  0x00007f47c11989a0 in nsslowkey_TraverseKeys () from /lib64/libnssdbm3.so
    #6  0x00007f47c119f61c in lg_searchTokenList.constprop.5 () from /lib64/libnssdbm3.so
    #7  0x00007f47c119faca in lg_FindObjectsInit () from /lib64/libnssdbm3.so
    #8  0x00007f47c168d9b1 in sftkdb_FindObjectsInit () from /lib64/libsoftokn3.so
    #9  0x00007f47c1675d1e in sftk_searchDatabase () from /lib64/libsoftokn3.so
    #10 0x00007f47c167ae46 in NSC_FindObjectsInit () from /lib64/libsoftokn3.so
    #11 0x00007f47c216688f in pk11_FindObjectByTemplate () from /lib64/libnss3.so
    #12 0x00007f47c2166ceb in PK11_MatchItem () from /lib64/libnss3.so
    #13 0x00007f47c218efc8 in nssToken_IsPrivateKeyAvailable () from /lib64/libnss3.so
    #14 0x00007f47c2185b3c in NSSCertificate_IsPrivateKeyAvailable () from /lib64/libnss3.so
    #15 0x00007f47c218b3dc in nssTrust_GetCERTCertTrustForCert () from /lib64/libnss3.so
    #16 0x00007f47c218b748 in stan_GetCERTCertificate () from /lib64/libnss3.so
    #17 0x00007f47c218561d in nssCertificate_GetDecoding () from /lib64/libnss3.so
    #18 0x00007f47c218a4a8 in nssCertificateArray_FindBestCertificate () from /lib64/libnss3.so
    #19 0x00007f47c214f1ee in PK11_FindCertFromNickname () from /lib64/libnss3.so
    #20 0x00007f47c28d51ff in JSS_PK11_findCertAndSlotFromNickname () from /usr/lib64/jss/libjss4.so
    #21 0x00007f47c28d32d5 in Java_org_mozilla_jss_CryptoManager_findCertByNicknameNative () from /usr/lib64/jss/libjss4.so
    #22 0x00007f47ccbfb1d4 in ?? ()
    #23 0xfffffffe00000000 in ?? ()
    #24 0x00007f47ccbfaf22 in ?? ()
    #25 0x00007f47e2c569f0 in ?? ()
    #26 0x00007f47c2b99348 in ?? ()
    #27 0x00007f47e2c56a58 in ?? ()
    #28 0x00007f47c2ba1de0 in ?? ()
    #29 0x0000000000000000 in ?? ()
    Thread 19 (Thread 0x7f47cc6c8700 (LWP 2604)):
    #0  0x00007f47e2836945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
    #1  0x00007f47e1735aeb in os::PlatformEvent::park() () from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-4.b11.el7.x86_64/jre/lib/amd64/server/libjvm.so
    #2  0x00007f47e16f1f27 in Monitor::IWait(Thread*, long) () from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-4.b11.el7.x86_64/jre/lib/amd64/server/libjvm.so
    #3  0x00007f47e16f2a4f in Monitor::wait(bool, long, bool) () from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-4.b11.el7.x86_64/jre/lib/amd64/server/libjvm.so
    #4  0x00007f47e143f1ca in GCTaskManager::get_task(unsigned int) () from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-4.b11.el7.x86_64/jre/lib/amd64/server/libjvm.so
    #5  0x00007f47e1440cbe in GCTaskThread::run() () from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-4.b11.el7.x86_64/jre/lib/amd64/server/libjvm.so
    #6  0x00007f47e172de12 in java_start(Thread*) () from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-4.b11.el7.x86_64/jre/lib/amd64/server/libjvm.so
    #7  0x00007f47e2832e25 in start_thread () from /lib64/libpthread.so.0
    #8  0x00007f47e1f3734d in clone () from /lib64/libc.so.6
$ strace -p 2603
...
lseek(72, 28672, SEEK_SET)              = 28672
read(72, "\6\0\0\17\350\t\350\10\322\3\1\20\0\0\300\3\322\3\377\377\377\377\377\377\377\377\377\377\377\377\377\377"..., 4096) = 4096
lseek(72, 28672, SEEK_SET)              = 28672
read(72, "\6\0\0\17\350\t\350\10\322\3\1\20\0\0\300\3\322\3\377\377\377\377\377\377\377\377\377\377\377\377\377\377"..., 4096) = 4096
lseek(72, 28672, SEEK_SET)              = 28672
...

certutil

certutil is stuck and burns 100% CPU, too. strace shows the same infinitive lseek() + read() loop.

$ certutil -d /etc/pki/pki-tomcat/alias/ -L

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

(no further output, certutil is stuck)

was reproduced only once, closing - insufficient info

Metadata Update from @pvoborni:
- Issue close_status updated to: insufficientinfo
- Issue status updated to: Closed (was: Open)

6 years ago

Login to comment on this ticket.

Metadata
Attachments 1
Attached 6 years ago View Comment