#47632 Replication GSSAPI authentication fails (IPA)
Closed: wontfix 3 years ago Opened 6 years ago by jcholast.

I have an IPA server and replica with 389-ds-base-1.3.2.8-1.fc20. Replication does not work, because GSSAPI authentication between the replicas fail. Both server and replica have error log filled with SASL/GSSAPI errors:

[10/Dec/2013:22:17:09 -0500] - 389-Directory/1.3.2.8 B2013.340.1954 starting up
[10/Dec/2013:22:17:09 -0500] schema-compat-plugin - warning: no entries set up under cn=ng, cn=compat,dc=idm,dc=lab,dc=bos,dc=redhat,dc=com
[10/Dec/2013:22:17:09 -0500] schema-compat-plugin - warning: no entries set up under ou=sudoers,dc=idm,dc=lab,dc=bos,dc=redhat,dc=com
[10/Dec/2013:22:17:09 -0500] NSACLPlugin - The ACL target cn=automember rebuild membership,cn=tasks,cn=config does not exist
[10/Dec/2013:22:17:09 -0500] - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=idm,dc=lab,dc=bos,dc=redhat,dc=com--no CoS Templates found, which should be added before the CoS Definition.
[10/Dec/2013:22:17:09 -0500] set_krb5_creds - Could not get initial credentials for principal [ldap/vm-053.idm.lab.bos.redhat.com@IDM.LAB.BOS.REDHAT.COM] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328324 (Gene
ric error (see e-text))
[10/Dec/2013:22:17:09 -0500] - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=idm,dc=lab,dc=bos,dc=redhat,dc=com--no CoS Templates found, which should be added before the CoS Definition.
[10/Dec/2013:22:17:09 -0500] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecifie
d GSS failure.  Minor code may provide more information (No Kerberos credentials available)) errno 0 (Success)
[10/Dec/2013:22:17:09 -0500] slapi_ldap_bind - Error: could not perform interactive bind for id [] authentication mechanism [GSSAPI]: error -2 (Local error)
[10/Dec/2013:22:17:09 -0500] NSMMReplicationPlugin - agmt="cn=meTovm-129.idm.lab.bos.redhat.com" (vm-129:389): Replication bind with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (No Kerberos credentials available))
[10/Dec/2013:22:17:09 -0500] - slapd started.  Listening on All Interfaces port 389 for LDAP requests
[10/Dec/2013:22:17:09 -0500] - Listening on All Interfaces port 636 for LDAPS requests
[10/Dec/2013:22:17:09 -0500] - Listening on /var/run/slapd-IDM-LAB-BOS-REDHAT-COM.socket for LDAPI requests
[10/Dec/2013:22:17:12 -0500] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error 49 (Invalid credentials) (SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context) errno 0 (Success)
[10/Dec/2013:22:17:12 -0500] slapi_ldap_bind - Error: could not perform interactive bind for id [] authentication mechanism [GSSAPI]: error 49 (Invalid credentials)
[10/Dec/2013:22:17:12 -0500] NSMMReplicationPlugin - agmt="cn=meTovm-129.idm.lab.bos.redhat.com" (vm-129:389): Replication bind with GSSAPI auth failed: LDAP error 49 (Invalid credentials) (SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context)
[10/Dec/2013:22:17:18 -0500] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error 49 (Invalid credentials) (SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context) errno 0 (Success)
[10/Dec/2013:22:17:18 -0500] slapi_ldap_bind - Error: could not perform interactive bind for id [] authentication mechanism [GSSAPI]: error 49 (Invalid credentials)
...
[11/Dec/2013:11:48:30 -0500] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error 49 (Invalid credentials) (SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context) errno 0 (Success)
[11/Dec/2013:11:48:30 -0500] slapi_ldap_bind - Error: could not perform interactive bind for id [] authentication mechanism [GSSAPI]: error 49 (Invalid credentials)
[11/Dec/2013:23:48:50 -0500] NSMMReplicationPlugin - agmt="cn=meTovm-129.idm.lab.bos.redhat.com" (vm-129:389): Replication bind with GSSAPI auth resumed
[12/Dec/2013:23:48:50 -0500] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Ticket expired)) errno 0 (Success)
[12/Dec/2013:23:48:50 -0500] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Ticket expired)) errno 0 (Success)
[12/Dec/2013:23:48:50 -0500] slapi_ldap_bind - Error: could not perform interactive bind for id [] authentication mechanism [GSSAPI]: error -2 (Local error)
[12/Dec/2013:23:48:50 -0500] NSMMReplicationPlugin - agmt="cn=meTovm-129.idm.lab.bos.redhat.com" (vm-129:389): Replication bind with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Ticket expired))
[12/Dec/2013:23:48:54 -0500] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Ticket expired)) errno 2 (No such file or directory)
[12/Dec/2013:23:48:54 -0500] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Ticket expired)) errno 2 (No such file or directory)
[12/Dec/2013:23:48:54 -0500] slapi_ldap_bind - Error: could not perform interactive bind for id [] authentication mechanism [GSSAPI]: error -2 (Local error)
...

How easy is it to reproduce the error? Have you see this error with 389-ds-base 1.3.1 - on F19 or on F20?

I encountered the error quite often lately, so I think it should be easy to reproduce. I have seen this only with 1.3.2, not with 1.3.1 (on both F19 and F20).

I have set up two F20 systems - ipa-server-install on one and ipa-replica-install on the other. I did not set up bind/named.

I did see some errors in the replica like this:
{{{
[16/Dec/2013:13:57:58 -0500] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No Kerberos credentials available)) errno 0 (Success)
[16/Dec/2013:13:57:58 -0500] slapi_ldap_bind - Error: could not perform interactive bind for id [] authentication mechanism [GSSAPI]: error -2 (Local error)
[16/Dec/2013:13:57:58 -0500] NSMMReplicationPlugin - agmt="cn=meTodell-pem710-01.rhts.eng.bos.redhat.com" (dell-pem710-01:389): Replication bind with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No Kerberos credentials available))
[16/Dec/2013:13:58:01 -0500] NSMMReplicationPlugin - agmt="cn=meTodell-pem710-01.rhts.eng.bos.redhat.com" (dell-pem710-01:389): Replication bind with GSSAPI auth resumed
}}}

But once replication resumed, everything seems to be working. On each server, I am running a script to add many users:
{{{
ii=1
out=test.out
err=test.err
rm -f $out $err
errcnt=0
while [ $ii -le 10000 ] ; do
ipa user-add --first=first --last=last user$ii >> $out 2>> $err || errcnt=expr $errcnt + 1
ii=expr $ii + 2
done
echo found $errcnt errors
}}}

On the replica I use a starting value of 2.

I have confirmed that each server is replicating the new users to the other server. This script has been running for hours - I have had no GSSAPI issues, and no issues with users missing from either server.

So I cannot reproduce this issue. I need more information.

Tried again, this time with --setup-dns. Same results - cannot reproduce - replication works fine. Test has been running for several hours with no errors.

I need some help to reproduce this problem.

I can't reproduce it now either.

Closing this ticket. Please reopen or open another one if the problem happens again.

I am able to reproduce this issue in cert renewal scenario and i think this is causing cert renewal failure on Replica.

Steps to Reproduce:
1. Install Master and a Replica
2. Change system date ahead (not after cert expire date) so that cert renewal takes place on Master for every certificate.
3. Wait till renewal is complete on Master.
4. Now see log file /var/log/dirsrv/<ipa-ldap-instance>/errors on Replica

following message shown in /var/log/dirsrv/<ipa-ldap-instance>/errors on Replica

[28/Feb/2014:10:11:59 -0500] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error 49 (Invalid credentials) (SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context) errno 0 (Success)
[28/Feb/2014:10:11:59 -0500] slapi_ldap_bind - Error: could not perform interactive bind for id [] authentication mechanism [GSSAPI]: error 49 (Invalid credentials)
[28/Feb/2014:10:11:59 -0500] NSMMReplicationPlugin - agmt="cn=meTodell-pe840-01.testrelm.test" (dell-pe840-01:389): Replication bind with GSSAPI auth failed: LDAP error 49 (Invalid credentials) (SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context)
[28/Feb/2014:10:12:03 -0500] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error 49 (Invalid credentials) (SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context) errno 0 (Success)

Also following log shown in /var/log/dirsrv/<ipa-ldap-instance>/errors on Replica when system date changed ahead on Master.

[28/Feb/2014:10:07:29 -0500] - csngen_adjust_time: adjustment limit exceeded; value - 61687825, limit - 86400
[28/Feb/2014:10:07:29 -0500] - CSN generator's state:
[28/Feb/2014:10:07:29 -0500] - replica id: 97
[28/Feb/2014:10:07:29 -0500] - sampled time: 1393600049
[28/Feb/2014:10:07:29 -0500] - local offset: 0
[28/Feb/2014:10:07:29 -0500] - remote offset: 1
[28/Feb/2014:10:07:29 -0500] - sequence number: 2
[28/Feb/2014:10:07:29 -0500] NSMMReplicationPlugin - conn=28 op=4 repl="o=ipaca": Excessive clock skew from supplier RUV
[28/Feb/2014:10:07:29 -0500] NSMMReplicationPlugin - conn=28 op=4 replica="o=ipaca": Unable to acquire replica: error: excessive clock skew

Replying to [comment:9 ksiddiqu]:

I am able to reproduce this issue in cert renewal scenario and i think this is causing cert renewal failure on Replica.

Steps to Reproduce:
1. Install Master and a Replica
2. Change system date ahead (not after cert expire date) so that cert renewal takes place on Master for every certificate.

It is very strongly discouraged to change the system date - this will cause problems with other features, notably replication (with time skew, below) and possibly kerberos.

I think this is a very highly artificial scenario, not something that a customer will likely run into.

  1. Wait till renewal is complete on Master.
  2. Now see log file /var/log/dirsrv/<ipa-ldap-instance>/errors on Replica

following message shown in /var/log/dirsrv/<ipa-ldap-instance>/errors on Replica

[28/Feb/2014:10:11:59 -0500] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error 49 (Invalid credentials) (SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context) errno 0 (Success)
[28/Feb/2014:10:11:59 -0500] slapi_ldap_bind - Error: could not perform interactive bind for id [] authentication mechanism [GSSAPI]: error 49 (Invalid credentials)
[28/Feb/2014:10:11:59 -0500] NSMMReplicationPlugin - agmt="cn=meTodell-pe840-01.testrelm.test" (dell-pe840-01:389): Replication bind with GSSAPI auth failed: LDAP error 49 (Invalid credentials) (SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context)
[28/Feb/2014:10:12:03 -0500] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error 49 (Invalid credentials) (SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context) errno 0 (Success)

Also following log shown in /var/log/dirsrv/<ipa-ldap-instance>/errors on Replica when system date changed ahead on Master.

[28/Feb/2014:10:07:29 -0500] - csngen_adjust_time: adjustment limit exceeded; value - 61687825, limit - 86400
[28/Feb/2014:10:07:29 -0500] - CSN generator's state:
[28/Feb/2014:10:07:29 -0500] - replica id: 97
[28/Feb/2014:10:07:29 -0500] - sampled time: 1393600049
[28/Feb/2014:10:07:29 -0500] - local offset: 0
[28/Feb/2014:10:07:29 -0500] - remote offset: 1
[28/Feb/2014:10:07:29 -0500] - sequence number: 2
[28/Feb/2014:10:07:29 -0500] NSMMReplicationPlugin - conn=28 op=4 repl="o=ipaca": Excessive clock skew from supplier RUV
[28/Feb/2014:10:07:29 -0500] NSMMReplicationPlugin - conn=28 op=4 replica="o=ipaca": Unable to acquire replica: error: excessive clock skew

Metadata Update from @rmeggins:
- Issue set to the milestone: N/A

3 years ago

I think it's been a long time since this was opened. It may not be relevant. If this issue still exists, please open a new issue.

Metadata Update from @firstyear:
- Issue close_status updated to: invalid
- Issue status updated to: Closed (was: Open)

3 years ago

389-ds-base is moving from Pagure to Github. This means that new issues and pull requests
will be accepted only in 389-ds-base's github repository.

This issue has been cloned to Github and is available here:
- https://github.com/389ds/389-ds-base/issues/969

If you want to receive further updates on the issue, please navigate to the github issue
and click on subscribe button.

Thank you for understanding. We apologize for all inconvenience.

Metadata Update from @spichugi:
- Issue close_status updated to: wontfix (was: invalid)

13 days ago

Login to comment on this ticket.

Metadata