#1894 sssd_be crashes while processing ASQ dereference request
Closed: Fixed None Opened 6 years ago by jhrozek.

This bug was reported on #sssd by John Hodrien while running master HEAD:

==17518== Memcheck, a memory error detector
==17518== Copyright (C) 2002-2012, and GNU GPL'd, by Julian Seward et al.
==17518== Using Valgrind-3.8.1 and LibVEX; rerun with -h for copyright info
==17518== Command: /usr/libexec/sssd/sssd_be --domain default
==17518== Parent PID: 17517
==17518==
==17518== Invalid write of size 8
==17518==    at 0xCEB658A: sdap_nested_group_deref_direct_done (sdap_async_nested_groups.c:2140)
==17518==    by 0xCE9796C: sdap_deref_search_done (sdap_async.c:2167)
==17518==    by 0xCE9730D: sdap_asq_search_done (sdap_async.c:2025)
==17518==    by 0xCE9BBBA: sdap_get_generic_ext_done (sdap_async.c:1449)
==17518==    by 0xCEA13DE: sdap_process_result (sdap_async.c:366)
==17518==    by 0x332F207BD8: tevent_common_loop_timer_delay (tevent_timed.c:254)
==17518==    by 0x332F2072AA: std_event_loop_once (tevent_standard.c:560)
==17518==    by 0x332F2038EF: _tevent_loop_once (tevent.c:507)
==17518==    by 0x332F20395A: tevent_common_loop_wait (tevent.c:608)
==17518==    by 0x4C908D2: server_loop (server.c:602)
==17518==    by 0x412DE5: main (data_provider_be.c:2766)
==17518==  Address 0x107b1018 is 0 bytes after a block of size 136 alloc'd
==17518==    at 0x4A069EE: malloc (vg_replace_malloc.c:270)
==17518==    by 0x333F605228: _talloc_zero (talloc.c:560)
==17518==    by 0xCEB1FD2: sdap_nested_group_deref_send (sdap_async_nested_groups.c:1952)
==17518==    by 0xCEB4AE3: sdap_nested_group_process_send (sdap_async_nested_groups.c:824)
==17518==    by 0xCEB51AC: sdap_nested_group_recurse_step (sdap_async_nested_groups.c:984)
==17518==    by 0xCEB5CF5: sdap_nested_group_recurse_send (sdap_async_nested_groups.c:954)
==17518==    by 0xCEB6344: sdap_nested_group_deref_direct_done (sdap_async_nested_groups.c:2177)
==17518==    by 0xCE9796C: sdap_deref_search_done (sdap_async.c:2167)
==17518==    by 0xCE9730D: sdap_asq_search_done (sdap_async.c:2025)
==17518==    by 0xCE9BBBA: sdap_get_generic_ext_done (sdap_async.c:1449)
==17518==    by 0xCEA13DE: sdap_process_result (sdap_async.c:366)
==17518==    by 0x332F207BD8: tevent_common_loop_timer_delay (tevent_timed.c:254)
==17518==
==17518== Invalid read of size 8
==17518==    at 0xCEB51A1: sdap_nested_group_recurse_step (sdap_async_nested_groups.c:984)
==17518==    by 0xCEB5286: sdap_nested_group_recurse_done (sdap_async_nested_groups.c:1011)
==17518==    by 0xCEB6C84: sdap_nested_group_process_done (sdap_async_nested_groups.c:902)
==17518==    by 0xCEB2415: sdap_nested_group_single_done (sdap_async_nested_groups.c:1324)
==17518==    by 0x332F2044C7: tevent_common_loop_immediate (tevent_immediate.c:135)
==17518==    by 0x332F207299: std_event_loop_once (tevent_standard.c:555)
==17518==    by 0x332F2038EF: _tevent_loop_once (tevent.c:507)
==17518==    by 0x332F20395A: tevent_common_loop_wait (tevent.c:608)
==17518==    by 0x4C908D2: server_loop (server.c:602)
==17518==    by 0x412DE5: main (data_provider_be.c:2766)
==17518==  Address 0x107b1018 is 0 bytes after a block of size 136 alloc'd
==17518==    at 0x4A069EE: malloc (vg_replace_malloc.c:270)
==17518==    by 0x333F605228: _talloc_zero (talloc.c:560)
==17518==    by 0xCEB1FD2: sdap_nested_group_deref_send (sdap_async_nested_groups.c:1952)
==17518==    by 0xCEB4AE3: sdap_nested_group_process_send (sdap_async_nested_groups.c:824)
==17518==    by 0xCEB51AC: sdap_nested_group_recurse_step (sdap_async_nested_groups.c:984)
==17518==    by 0xCEB5CF5: sdap_nested_group_recurse_send (sdap_async_nested_groups.c:954)
==17518==    by 0xCEB6344: sdap_nested_group_deref_direct_done (sdap_async_nested_groups.c:2177)
==17518==    by 0xCE9796C: sdap_deref_search_done (sdap_async.c:2167)
==17518==    by 0xCE9730D: sdap_asq_search_done (sdap_async.c:2025)
==17518==    by 0xCE9BBBA: sdap_get_generic_ext_done (sdap_async.c:1449)
==17518==    by 0xCEA13DE: sdap_process_result (sdap_async.c:366)
==17518==    by 0x332F207BD8: tevent_common_loop_timer_delay (tevent_timed.c:254)
==17518==
==17518== Invalid read of size 8
==17518==    at 0xCEB51A1: sdap_nested_group_recurse_step (sdap_async_nested_groups.c:984)
==17518==    by 0xCEB5CF5: sdap_nested_group_recurse_send (sdap_async_nested_groups.c:954)
==17518==    by 0xCEB6344: sdap_nested_group_deref_direct_done (sdap_async_nested_groups.c:2177)
==17518==    by 0xCE9796C: sdap_deref_search_done (sdap_async.c:2167)
==17518==    by 0xCE9730D: sdap_asq_search_done (sdap_async.c:2025)
==17518==    by 0xCE9BBBA: sdap_get_generic_ext_done (sdap_async.c:1449)
==17518==    by 0xCEA13DE: sdap_process_result (sdap_async.c:366)
==17518==    by 0x332F207BD8: tevent_common_loop_timer_delay (tevent_timed.c:254)
==17518==    by 0x332F2072AA: std_event_loop_once (tevent_standard.c:560)
==17518==    by 0x332F2038EF: _tevent_loop_once (tevent.c:507)
==17518==    by 0x332F20395A: tevent_common_loop_wait (tevent.c:608)
==17518==    by 0x4C908D2: server_loop (server.c:602)
==17518==  Address 0xd42e580 is 0 bytes after a block of size 80 alloc'd
==17518==    at 0x4A069EE: malloc (vg_replace_malloc.c:270)
==17518==    by 0x333F605228: _talloc_zero (talloc.c:560)
==17518==    by 0xCEB1FD2: sdap_nested_group_deref_send (sdap_async_nested_groups.c:1952)
==17518==    by 0xCEB4AE3: sdap_nested_group_process_send (sdap_async_nested_groups.c:824)
==17518==    by 0xCEB584C: sdap_nested_group_send (sdap_async_nested_groups.c:641)
==17518==    by 0xCEB072D: sdap_get_groups_process (sdap_async_groups.c:1649)
==17518==    by 0xCE97CDD: sdap_get_generic_done (sdap_async.c:1558)
==17518==    by 0xCE9C08C: sdap_get_generic_ext_done (sdap_async.c:1407)
==17518==    by 0xCEA13DE: sdap_process_result (sdap_async.c:366)
==17518==    by 0x332F207BD8: tevent_common_loop_timer_delay (tevent_timed.c:254)
==17518==    by 0x332F2072AA: std_event_loop_once (tevent_standard.c:560)
==17518==    by 0x332F2038EF: _tevent_loop_once (tevent.c:507)
==17518==
--17518-- VALGRIND INTERNAL ERROR: Valgrind received a signal 11 (SIGSEGV) - exiting
--17518-- si_code=1;  Faulting address: 0x24F46708;  sp: 0x402f04dc0

valgrind: the 'impossible' happened:
   Killed by fatal signal
==17518==    at 0x3803E66F: vgPlain_arena_free (m_mallocfree.c:291)
==17518==    by 0x38003667: create_MC_Chunk (mc_malloc_wrappers.c:165)
==17518==    by 0x38003BE0: vgMemCheck_new_block (mc_malloc_wrappers.c:283)
==17518==    by 0x3800409A: vgMemCheck_malloc (mc_malloc_wrappers.c:301)
==17518==    by 0x3807A58A: vgPlain_scheduler (scheduler.c:1665)
==17518==    by 0x380A5A19: run_a_thread_NORETURN (syswrap-linux.c:103)

sched status:
  running_tid=1

Thread 1: status = VgTs_Runnable
==17518==    at 0x4A069EE: malloc (vg_replace_malloc.c:270)
==17518==    by 0x333F604CA0: talloc_named_const (talloc.c:560)
==17518==    by 0xCEB9352: build_membership_diff (sdap_async_initgroups.c:627)
==17518==    by 0xCEBCE2B: rfc2307bis_group_memberships_build (sdap_async_initgroups.c:1949)
==17518==    by 0x332F600D71: hash_iterate (dhash.c:697)
==17518==    by 0xCEBE041: sdap_initgr_rfc2307bis_done (sdap_async_initgroups.c:1834)
==17518==    by 0xCEC65A7: rfc2307bis_nested_groups_done (sdap_async_initgroups.c:2499)
==17518==    by 0xCEC6C68: rfc2307bis_nested_groups_process (sdap_async_initgroups.c:2438)
==17518==    by 0xCE97CDD: sdap_get_generic_done (sdap_async.c:1558)
==17518==    by 0xCE9BBBA: sdap_get_generic_ext_done (sdap_async.c:1449)
==17518==    by 0xCEA13DE: sdap_process_result (sdap_async.c:366)
==17518==    by 0x332F207BD8: tevent_common_loop_timer_delay (tevent_timed.c:254)
==17518==    by 0x332F2072AA: std_event_loop_once (tevent_standard.c:560)
==17518==    by 0x332F2038EF: _tevent_loop_once (tevent.c:507)
==17518==    by 0x332F20395A: tevent_common_loop_wait (tevent.c:608)
==17518==    by 0x4C908D2: server_loop (server.c:602)
==17518==    by 0x412DE5: main (data_provider_be.c:2766)

John also managed to gether the backtrace:

    Program received signal SIGABRT, Aborted.
    0x000000332ce328a5 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
    64 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
    Missing separate debuginfos, use: debuginfo-install libgcc-4.4.7-3.el6.x86_64
    (gdb) bt
    #0 0x000000332ce328a5 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
    #1 0x000000332ce34085 in abort () at abort.c:92
    #2 0x000000332ce707b7 in __libc_message (do_abort=2, fmt=0x332cf57f80 "*** glibc detected *** %s: %s: 0x%s ***\n")
    at ../sysdeps/unix/sysv/linux/libc_fatal.c:198
    #3 0x000000332ce760e6 in malloc_printerr (action=3, str=0x332cf582e0 "double free or corruption (!prev)", ptr=<value optimized out>)
    at malloc.c:6311
    #4 0x000000332ce78c13 in _int_free (av=0x332d18ee80, p=0x8750e0, have_lock=0) at malloc.c:4811
    #5 0x000000333f602389 in ?? () from /usr/lib64/libtalloc.so.2
    #6 0x000000333f602323 in ?? () from /usr/lib64/libtalloc.so.2
    #7 0x000000333f602323 in ?? () from /usr/lib64/libtalloc.so.2
    #8 0x00007fa116a61c71 in sdap_nested_group_process_done (subreq=0x8a70e0) at src/providers/ldap/sdap_async_nested_groups.c:869
    #9 0x00007fa116a5bbd6 in sdap_nested_group_deref_done (subreq=0x0) at src/providers/ldap/sdap_async_nested_groups.c:2216
    #10 0x00007fa116a6029d in sdap_nested_group_recurse_done (subreq=0x0) at src/providers/ldap/sdap_async_nested_groups.c:1015
    #11 0x00007fa116a61c85 in sdap_nested_group_process_done (subreq=0x0) at src/providers/ldap/sdap_async_nested_groups.c:902
    #12 0x00007fa116a5d416 in sdap_nested_group_single_done (subreq=0x0) at src/providers/ldap/sdap_async_nested_groups.c:1324
    #13 0x000000332f2044c8 in tevent_common_loop_immediate () from /usr/lib64/libtevent.so.0
    #14 0x000000332f20729a in ?? () from /usr/lib64/libtevent.so.0
    #15 0x000000332f2038f0 in _tevent_loop_once () from /usr/lib64/libtevent.so.0
    #16 0x000000332f20395b in tevent_common_loop_wait () from /usr/lib64/libtevent.so.0
    #17 0x00007fa11e8d38d3 in server_loop (main_ctx=0x80e6a0) at src/util/server.c:602
    #18 0x0000000000412de6 in main (argc=<value optimized out>, argv=<value optimized out>) at src/providers/data_provider_be.c:2766

Fields changed

priority: major => critical

Fields changed

milestone: NEEDS_TRIAGE => SSSD 1.10.0
rhbz: => 0

Fields changed

changelog: =>
owner: somebody => pbrezina
status: new => assigned

Fields changed

patch: 0 => 1

resolution: => fixed
status: assigned => closed

Two additional fixes landed in master:
- fc0d76a
- e6dee51

Metadata Update from @jhrozek:
- Issue assigned to pbrezina
- Issue set to the milestone: SSSD 1.10.0

2 years ago

Login to comment on this ticket.

Metadata