#47889 DS crashed during ipa-server-install on test_ava_filter
Closed: wontfix None Opened 6 years ago by mkosek.

ipa-server-install crashed as DS crashed:

# ipa-server-install
...
Global DNS configuration in LDAP server is empty
You can use 'dnsconfig-mod' command to set global DNS options that
would override settings in local named.conf files

Restarting the web server
Unexpected error - see /var/log/ipaserver-install.log for details:
CalledProcessError: Command ''/bin/systemctl' 'restart' 'ipa.service'' returned non-zero exit status 1

# ipactl restart
Starting Directory Service
Starting krb5kdc Service
Starting kadmin Service
Starting named Service
Starting ipa_memcached Service
Starting httpd Service
Starting pki-tomcatd Service
Failed to start pki-tomcatd Service
Shutting down
Aborting ipactl

# systemctl status dirsrv@MKOSEK-FEDORA20-TEST.service 
dirsrv@MKOSEK-FEDORA20-TEST.service - 389 Directory Server MKOSEK-FEDORA20-TEST.
   Loaded: loaded (/usr/lib/systemd/system/dirsrv@.service; enabled)
   Active: failed (Result: signal) since Fri 2014-09-05 14:29:30 CEST; 6min ago
  Process: 2744 ExecStopPost=/bin/rm -f /var/run/dirsrv/slapd-%i.pid (code=exited, status=0/SUCCESS)
  Process: 2653 ExecStart=/usr/sbin/ns-slapd -D /etc/dirsrv/slapd-%i -i /var/run/dirsrv/slapd-%i.pid -w /var/run/dirsrv/slapd-%i.startpid (code=exited, status=0/SUCCESS)
 Main PID: 2676 (code=killed, signal=SEGV)

Sep 05 14:29:27 ipa.mkosek-fedora20.test systemd[1]: Started 389 Directory Server MKOSEK-FEDORA20-TEST..
Sep 05 14:29:29 ipa.mkosek-fedora20.test ns-slapd[2676]: GSSAPI server step 1
Sep 05 14:29:29 ipa.mkosek-fedora20.test ns-slapd[2676]: GSSAPI server step 2
Sep 05 14:29:29 ipa.mkosek-fedora20.test ns-slapd[2676]: GSSAPI server step 3
Sep 05 14:29:29 ipa.mkosek-fedora20.test ns-slapd[2676]: GSSAPI server step 1
Sep 05 14:29:29 ipa.mkosek-fedora20.test ns-slapd[2676]: GSSAPI server step 2
Sep 05 14:29:29 ipa.mkosek-fedora20.test ns-slapd[2676]: GSSAPI server step 3
Sep 05 14:29:30 ipa.mkosek-fedora20.test systemd[1]: dirsrv@MKOSEK-FEDORA20-TEST.service: main pro...EGV
Sep 05 14:29:30 ipa.mkosek-fedora20.test systemd[1]: Unit dirsrv@MKOSEK-FEDORA20-TEST.service ente...te.
Hint: Some lines were ellipsized, use -l to show in full.

Versions:

freeipa-server-4.0.1GIT5de8d88-0.fc20.x86_64
389-ds-base-1.3.2.23-1.fc20.x86_64
pki-ca-10.1.1-1.fc20.noarch

Reproduced the crash the seems related to an access to a freed entry. {{{ (gdb) where #0 0x00007f4f054273e3 in sdn_is_nulldn (sdn=sdn@entry=0x7f4ec400f600) at ldap/servers/slapd/mapping_tree.c:2092 #1 0x00007f4f054299b7 in slapi_get_mapping_tree_node_by_dn (dn=0x7f4ec400f600) at ldap/servers/slapd/mapping_tree.c:2920 #2 0x00007f4f05429cc9 in slapi_be_select (sdn=<optimized out>) at ldap/servers/slapd/mapping_tree.c:3100 #3 0x00007f4f054721f0 in vattr_test_filter (pb=pb@entry=0x7f4ec4009810, e=e@entry=0x7f4ec400f600, f=f@entry=0x7f4f07b1b6f0, filter_type=FILTER_TYPE_AVA, type=0x7f4f07b94d30 "objectClass") at ldap/servers/slapd/vattr.c:432 #4 0x00007f4f0540e73f in slapi_vattr_filter_test_ext_internal (pb=pb@entry=0x7f4ec4009810, e=e@entry=0x7f4ec400f600, f=f@entry=0x7f4f07b1b6f0, verify_access=verify_access@entry=0, only_check_access=only_check_access@entry=0, access_check_done=access_check_done@entry=0x7f4ed1fedfe4) at ldap/servers/slapd/filterentry.c:897 #5 0x00007f4f0540ea75 in vattr_test_filter_list (pb=pb@entry=0x7f4ec4009810, e=e@entry=0x7f4ec400f600, flist=<optimized out>, ftype=161, verify_access=0, only_check_access=0, access_check_done=0x7f4ed1fedfe4) at ldap/servers/slapd/filterentry.c:1038 #6 0x00007f4f0540e64d in slapi_vattr_filter_test_ext_internal (pb=0x7f4ec4009810, e=0x7f4ec400f600, f=0x7f4f07b306b0, verify_access=verify_access@entry=0, only_check_access=0, access_check_done=access_check_done@entry=0x7f4ed1fedfe4) at ldap/servers/slapd/filterentry.c:965 #7 0x00007f4f0540f74c in slapi_vattr_filter_test_ext (pb=pb@entry=0x7f4ec4009810, e=0x7f4ec400f600, f=0x7f4f07b306b0, verify_access=verify_access@entry=0, only_check_access=only_check_access@entry=0) at ldap/servers/slapd/filterentry.c:827 #8 0x00007f4f0540f788 in slapi_vattr_filter_test (pb=pb@entry=0x7f4ec4009810, e=<optimized out>, f=<optimized out>, verify_access=verify_access@entry=0) at ldap/servers/slapd/filterentry.c:790 #9 0x00007f4efb9d8832 in dna_be_txn_pre_op (pb=0x7f4ec4009810, modtype=4) at ldap/servers/plugins/dna/dna.c:3994 #10 0x00007f4f0543e0b5 in plugin_call_func (list=0x7f4f074ef4d0, operation=operation@entry=461, pb=pb@entry=0x7f4ec4009810, call_one=call_one@entry=0) at ldap/servers/slapd/plugin.c:1489 #11 0x00007f4f0543e268 in plugin_call_list (pb=0x7f4ec4009810, operation=461, list=<optimized out>) at ldap/servers/slapd/plugin.c:1451 #12 plugin_call_plugins (pb=pb@entry=0x7f4ec4009810, whichfunction=whichfunction@entry=461) at ldap/servers/slapd/plugin.c:413 #13 0x00007f4ef7bdda08 in ldbm_back_modify (pb=<optimized out>) at ldap/servers/slapd/back-ldbm/ldbm_modify.c:670 #14 0x00007f4f0542e171 in op_shared_modify (pb=pb@entry=0x7f4ec4009810, pw_change=pw_change@entry=0, old_pw=0x0) at ldap/servers/slapd/modify.c:1081 #15 0x00007f4f0542ec34 in modify_internal_pb (pb=0x7f4ec4009810) at ldap/servers/slapd/modify.c:631 #16 0x00007f4efafb4bd3 in ipalockout_postop () from /usr/lib64/dirsrv/plugins/libipa_lockout.so #17 0x00007f4f0543e0b5 in plugin_call_func (list=0x7f4f074ff580, operation=operation@entry=501, pb=pb@entry=0x7f4ed1ff2ae0, call_one=call_one@entry=0) at ldap/servers/slapd/plugin.c:1489 #18 0x00007f4f0543e268 in plugin_call_list (pb=0x7f4ed1ff2ae0, operation=501, list=<optimized out>) at ldap/servers/slapd/plugin.c:1451 #19 plugin_call_plugins (pb=pb@entry=0x7f4ed1ff2ae0, whichfunction=whichfunction@entry=501) at ldap/servers/slapd/plugin.c:413 #20 0x00007f4f058ff48f in do_bind (pb=pb@entry=0x7f4ed1ff2ae0) at ldap/servers/slapd/bind.c:424 #21 0x00007f4f05905ebf in connection_dispatch_operation (pb=0x7f4ed1ff2ae0, op=0x7f4f07bde8a0, conn=0x7f4ef01a26b0) at ldap/servers/slapd/connection.c:635 #22 connection_threadmain () at ldap/servers/slapd/connection.c:2534 #23 0x00007f4f03831e3b in _pt_root (arg=0x7f4f07bd4780) at ../../../nspr/pr/src/pthreads/ptthread.c:212 #24 0x00007f4f031d1f35 in start_thread (arg=0x7f4ed1ff3700) at pthread_create.c:309 #25 0x00007f4f02f00c3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 }}} It can conduct to different kind of crash, so the stack itself may change.

Glad you could reproduce it, Thierry! Did you have a chance to run DS via valgrind?

  • The crash occurs because the modified entry (in pblock) is corrupted.
    When processing a MOD, the target entry is duplicated and mod_ops are processed on the duplicated entry.
    This duplicated entry (SLAPI_MODIFY_EXISTING_ENTRY) is corrupted (freed ??) when the be-txn-pre-op (dna_be_txn_pre_op)
    is called.
    The only possibilities I can think of are:
    - one of the preop/txnpreop frees that entry
    - There is a heap corruption before that this function is called and this thread is victim of this corruption.

  • Crashing thread is the only active thread at this time

  • The crash occurs quite often but not systematically.
  • I was not able to reproduce it with valgrind
  • An other possibility is that SLAPI_MODIFY_EXISTING_ENTRY is not always set. It may happened if the modify needs to be retried (in case of transient db failure). Then SLAPI_MODIFY_EXISTING_ENTRY entry is removed from the entry cache and freed. The retry will occur with a freed entry.
  • recent fix (https://fedorahosted.org/389/ticket/47834) removes the setting of SLAPI_MODIFY_EXISTING_ENTRY in case of retry (thanks Ludwig for catching that !)
  • testing a fix (sets SLAPI_MODIFY_EXISTING_ENTRY in case of retry) on top of 47834, I was not able to reproduce the crash (in >100 restart. without the fix it crashed 1 times out of 20 restart).

Pushed to master on behalf of Thierry (Thank you, Thierry!!)
0f1a203..3b5f3fa master -> master
commit 3b5f3fa

Pushed to 389-ds-base-1.3.3 branch:
cad5b96..0363fa4 389-ds-base-1.3.3 -> 389-ds-base-1.3.3
commit 0363fa4

Pushed to 389-ds-base-1.3.2 branch:
0056b61..1db611e 389-ds-base-1.3.2 -> 389-ds-base-1.3.2
commit 1db611e

I leave this ticket opened. Thierry, if the fix is verified, please close it.
Thank you!!

Crash with 389-ds-base-1.3.3.2-1.fc20.x86_64
stacktrace.1410511113.txt.gz

I tested with 389-ds-base-1.3.3.2-1.fc20.x86_64 (rebuilt F21 SRPM from Fedora) and I still got a crash - see attached stacktrace.

This is the same crash. What is weird is that the source is not containing the fix (setting of SLAPI_MODIFY_EXISTING_ENTRY is not present) :

{{{
(gdb) l 520,535
520 / reset ec set cache in id2entry_add_ext /
521 if (ec) {
522 / must duplicate ec before returning it to cache,
523 * which could free the entry.
/
524 if ( (tmpentry = backentry_dup( ec )) == NULL ) {
525 ldap_result_code= LDAP_OPERATIONS_ERROR;
526 goto error_return;
527 }
528 if (cache_is_in_cache(&inst->inst_cache, ec)) {
529 CACHE_REMOVE(&inst->inst_cache, ec);
530 }
531 CACHE_RETURN(&inst->inst_cache, &ec);
532 ec = original_entry;
533 original_entry = tmpentry;
534 tmpentry = NULL;
535 }

}}}

However the branch 1.3.3 contains the fix and the version 1.3.3.2 contains it.

It is looking that the test was done from http://koji.fedoraproject.org/koji/buildinfo?buildID=576829 (389-ds-base-1.3.3.2-1.fc21.src.rpm) but this build is missing some of the fixes (dna crash 47889 and possibly some part of 47885).

The source tarball for 389-ds-base-1.3.3.2-1 was incorrect. I have corrected this in 389-ds-base-1.3.3.2-2:

http://koji.fedoraproject.org/koji/taskinfo?taskID=7572257

I have successfully tested copr builds of master branch and 1.3.2. (I needed to backport Ludwig fix for freeipa Update-SSL-ciphers-configured-in-389-ds-base to configure the instance).

freeipa (+dns) restart did not trigger crash after more than 100 restarts.

Metadata Update from @mkosek:
- Issue assigned to tbordaz
- Issue set to the milestone: 1.3.2.24

3 years ago

389-ds-base is moving from Pagure to Github. This means that new issues and pull requests
will be accepted only in 389-ds-base's github repository.

This issue has been cloned to Github and is available here:
- https://github.com/389ds/389-ds-base/issues/1220

If you want to receive further updates on the issue, please navigate to the github issue
and click on subscribe button.

Thank you for understanding. We apologize for all inconvenience.

Metadata Update from @spichugi:
- Issue close_status updated to: wontfix (was: Fixed)

3 months ago

Login to comment on this ticket.

Metadata