Description of problem:

Testing IPA CA Certificate autorenewal I am seeing certificate autorenewal fail
because it's trying to renew everything at the same time.

Version-Release number of selected component (if applicable):
ipa-server-4.1.0-12.el7.x86_64
certmonger-0.75.14-2.el7.x86_64
pki-ca-10.1.2-5.el7.noarch


How reproducible:
always

Steps to Reproduce:

1.  Setup IPA server

ipa-server-install --setup-dns --forwarder=<IP> -r <REALM> -a <PASSWORD> -p
<PASSWORD> -U

2.  Walk the time forward till you reach the CA Certificate renewal threshold.

getcert list | egrep "status|expires|Request|subject|ca-error"
date -u <soonest expiration date - 4 weeks>
# check with getcert until everything back in monitoring then repeat

3.  When you reach time for CA expire, all certs should show same expiration
equal to that of CA certificate's.

4.  Change time to within 4 weeks of expiration.


Actual results:

[root@vm1 ~]# date -u; getcert list | egrep
"status|expires|Request|subject|ca-error"
Mon Nov 13 15:06:03 UTC 2034
Request ID '20141211150642':
        status: NOTIFYING_VALIDITY
        subject: CN=CA Audit,O=EXAMPLE.TEST
        expires: 2034-12-11 15:06:02 UTC
Request ID '20141211150643':
        status: NOTIFYING_VALIDITY
        subject: CN=OCSP Subsystem,O=EXAMPLE.TEST
        expires: 2034-12-11 15:06:02 UTC
Request ID '20141211150644':
        status: NOTIFYING_VALIDITY
        subject: CN=CA Subsystem,O=EXAMPLE.TEST
        expires: 2034-12-11 15:06:02 UTC
Request ID '20141211150645':
        status: NOTIFYING_VALIDITY
        subject: CN=Certificate Authority,O=EXAMPLE.TEST
        expires: 2034-12-11 15:06:02 UTC
Request ID '20141211150646':
        status: NOTIFYING_VALIDITY
        subject: CN=IPA RA,O=EXAMPLE.TEST
        expires: 2034-12-11 15:06:02 UTC
Request ID '20141211150647':
        status: GENERATING_CSR
        subject: CN=vm1.example.test,O=EXAMPLE.TEST
        expires: 2034-12-11 15:06:02 UTC
Request ID '20141211150648':
        status: GENERATING_CSR
        subject: CN=vm1.example.test,O=EXAMPLE.TEST
        expires: 2034-12-11 15:06:02 UTC
Request ID '20141211150715':
        status: GENERATING_CSR
        subject: CN=vm1.example.test,O=EXAMPLE.TEST
        expires: 2034-12-11 15:06:02 UTC

[root@vm1 ~]# date -u; getcert list | egrep
"status|expires|Request|subject|ca-error"
Mon Nov 13 15:06:46 UTC 2034
Request ID '20141211150642':
        status: CA_UNREACHABLE
        ca-error: Internal error
        subject: CN=CA Audit,O=EXAMPLE.TEST
        expires: 2034-12-11 15:06:02 UTC
Request ID '20141211150643':
        status: CA_UNREACHABLE
        ca-error: Internal error
        subject: CN=OCSP Subsystem,O=EXAMPLE.TEST
        expires: 2034-12-11 15:06:02 UTC
Request ID '20141211150644':
        status: CA_UNREACHABLE
        ca-error: Internal error
        subject: CN=CA Subsystem,O=EXAMPLE.TEST
        expires: 2034-12-11 15:06:02 UTC
Request ID '20141211150645':
        status: CA_UNREACHABLE
        ca-error: Internal error
        subject: CN=Certificate Authority,O=EXAMPLE.TEST
        expires: 2034-12-11 15:06:02 UTC
Request ID '20141211150646':
        status: CA_UNREACHABLE
        ca-error: Internal error
        subject: CN=IPA RA,O=EXAMPLE.TEST
        expires: 2034-12-11 15:06:02 UTC
Request ID '20141211150647':
        status: CA_UNREACHABLE
        ca-error: Error 7 connecting to
http://vm1.example.test:8080/ca/ee/ca/profileSubmit: Couldn't connect to
server.
        subject: CN=vm1.example.test,O=EXAMPLE.TEST
        expires: 2034-12-11 15:06:02 UTC
Request ID '20141211150648':
        status: CA_UNREACHABLE
        ca-error: Error setting up ccache for "host" service on client using
default keytab: Cannot contact any KDC for realm 'EXAMPLE.TEST'.
        subject: CN=vm1.example.test,O=EXAMPLE.TEST
        expires: 2034-12-11 15:06:02 UTC
Request ID '20141211150715':
        status: CA_UNREACHABLE
        ca-error: Error setting up ccache for "host" service on client using
default keytab: Cannot contact any KDC for realm 'EXAMPLE.TEST'.
        subject: CN=vm1.example.test,O=EXAMPLE.TEST
        expires: 2034-12-11 15:06:02 UTC

Expected results:

CA Certificate renews first, then others in proper order to prevent
CA_UNREACHABLE due to conflicts with renewing some at the same time.

Additional info:
I'll also attach the PKI debug log.