During the test case, a very very rare issue can be observed (maybe 1 in a million executions)
This appears to be a race between internal job rearm's state assertion, and the thread shutdown clearing the queues.
[ RUN ] ns_set_data_test Assertion failure: job->state == NS_JOB_NEEDS_ARM, at src/nunc-stans/ns/ns_thrpool.c:254 Assertion failure: job->state == NS_JOB_NEEDS_ARM, at src/nunc-stans/ns/ns_thrpool.c:254 ns_thrpool_new(): max threads, (4) ns_thrpool_new(): max threads, (4) stacksize (0), event q size (unbounded), work q size (unbounded) stacksize (0), event q size (unbounded), work q size (unbounded) new_ns_job 580 initial NS_JOB_WAITING new_ns_job 580 initial NS_JOB_WAITING setup_event_q_wakeup 580 state 7 moving NS_JOB_ARMED setup_event_q_wakeup 580 state 7 moving NS_JOB_ARMED new_ns_job 660 initial NS_JOB_WAITING new_ns_job 660 initial NS_JOB_WAITING ns_add_job 660 state 7 moving to NS_JOB_ARMED ns_add_job 660 state 7 moving to NS_JOB_ARMED internal_ns_rearm_job 660 state 4 moving to NS_JOB_ARMED internal_ns_rearm_job 660 state 4 moving to NS_JOB_ARMED work_q_notify 660 state 5 work_q_notify 660 state 5 work_job_execute 660 state 5 moving to NS_JOB_RUNNING work_job_execute 660 state 5 moving to NS_JOB_RUNNING work_job_execute 660 state 6 job func complete move to NS_JOB_WAITING work_job_execute 660 state 6 job func complete move to NS_JOB_WAITING ns_rearm_job 660 state 7 moving to NS_JOB_NEEDS_ARM ns_rearm_job 660 state 7 moving to NS_JOB_NEEDS_ARM internal_ns_rearm_job 660 state 4 moving to NS_JOB_ARMED internal_ns_rearm_job 660 state 4 moving to NS_JOB_ARMED work_q_notify 660 state 5 work_q_notify 660 state 5 work_job_execute 660 state 5 moving to NS_JOB_RUNNING work_job_execute 660 state 5 moving to NS_JOB_RUNNING work_job_execute 660 state 6 job func complete move to NS_JOB_WAITING work_job_execute 660 state 6 job func complete move to NS_JOB_WAITING ns_job_done 660 tp shutdown -> false state 7 setting NS_JOB_NEEDS_DELETE and queuing ns_job_done 660 tp shutdown -> false state 7 setting NS_JOB_NEEDS_DELETE and queuing event_q_notify enqueuing 660 with state 2 event_q_notify enqueuing 660 with state 2 event_q_wake attempting to wake event queue. event_q_wake attempting to wake event queue. event_q_wake result. 0 event_q_wake result. 0 ns_thrpool_shutdown initiated ... ns_thrpool_shutdown initiated ... new_ns_job 740 initial NS_JOB_WAITING new_ns_job 740 initial NS_JOB_WAITING internal_ns_rearm_job 740 state 4 moving to NS_JOB_ARMED internal_ns_rearm_job 740 state 4 moving to NS_JOB_ARMED work_q_notify 740 state 5 work_q_notify 740 state 5 event_cb 580 state 5 non-threaded, execute right meow event_cb 580 state 5 non-threaded, execute right meow work_job_execute 580 state 5 moving to NS_JOB_RUNNING work_job_execute 580 state 5 moving to NS_JOB_RUNNING wakeup_cb 580 state 6 wakeup_cb wakeup_cb 580 state 6 wakeup_cb new_ns_job 820 initial NS_JOB_WAITING new_ns_job 820 initial NS_JOB_WAITING internal_ns_rearm_job 820 state 4 moving to NS_JOB_ARMED internal_ns_rearm_job 820 state 4 moving to NS_JOB_ARMED work_q_notify 820 state 5 work_q_notify 820 state 5 new_ns_job 900 initial NS_JOB_WAITING new_ns_job 900 initial NS_JOB_WAITING internal_ns_rearm_job 900 state 4 moving to NS_JOB_ARMED internal_ns_rearm_job 900 state 4 moving to NS_JOB_ARMED work_q_notify 900 state 5 work_q_notify 900 state 5 worker_thread_func notified to shutdown! worker_thread_func notified to shutdown! internal_ns_job_done 740 state 5 moving to NS_JOB_DELETED internal_ns_job_done 740 state 5 moving to NS_JOB_DELETED worker_thread_func shutdown complete! worker_thread_func shutdown complete! get_new_event_requests Dequeuing 660 with state 2 get_new_event_requests Dequeuing 660 with state 2 update_event 660 state 2 update_event 660 state 2 internal_ns_job_done 660 state 2 moving to NS_JOB_DELETED internal_ns_job_done 660 state 2 moving to NS_JOB_DELETED work_job_execute PERSIST and RUNNING, remarking 580 as NS_JOB_NEEDS_ARM work_job_execute PERSIST and RUNNING, remarking 580 as NS_JOB_NEEDS_ARM work_job_execute 580 state 4 job func complete, sending to rearm... work_job_execute 580 state 4 job func complete, sending to rearm... worker_thread_func notified to shutdown! worker_thread_func notified to shutdown! new_ns_job 9e0 initial NS_JOB_WAITING new_ns_job 9e0 initial NS_JOB_WAITING worker_thread_func notified to shutdown! worker_thread_func notified to shutdown! internal_ns_job_done 900 state 5 moving to NS_JOB_DELETED internal_ns_job_done 900 state 5 moving to NS_JOB_DELETED internal_ns_rearm_job 9e0 state 4 moving to NS_JOB_ARMED internal_ns_rearm_job 9e0 state 4 moving to NS_JOB_ARMED work_q_notify 9e0 state 5 work_q_notify 9e0 state 5 worker_thread_func shutdown complete! worker_thread_func shutdown complete! ns_thrpool_wait has begun ns_thrpool_wait has begun ns_thrpool_wait joined thread, result 0 ns_thrpool_wait joined thread, result 0 internal_ns_job_done 820 state 5 moving to NS_JOB_DELETED internal_ns_job_done 820 state 5 moving to NS_JOB_DELETED worker_thread_func shutdown complete! worker_thread_func shutdown complete! worker_thread_func notified to shutdown! worker_thread_func notified to shutdown! internal_ns_job_done 9e0 state 5 moving to NS_JOB_DELETED internal_ns_job_done 9e0 state 5 moving to NS_JOB_DELETED worker_thread_func shutdown complete! worker_thread_func shutdown complete! ns_thrpool_wait joined thread, result 0 ns_thrpool_wait joined thread, result 0 ns_thrpool_wait joined thread, result 0 ns_thrpool_wait joined thread, result 0 ns_thrpool_wait joined thread, result 0 ns_thrpool_wait joined thread, result 0 ns_thrpool_wait complete, retval 0 ns_thrpool_wait complete, retval 0 ns_thrpool_destroy ns_thrpool_destroy ns_thrpool_destroy 580 state 4 moving to NS_JOB_NEEDS_DELETE ns_thrpool_destroy 580 state 4 moving to NS_JOB_NEEDS_DELETE event_q_notify enqueuing 580 with state 2 event_q_notify enqueuing 580 with state 2 event_q_wake attempting to wake event queue. event_q_wake attempting to wake event queue. event_q_wake result. 0 event_q_wake result. 0 FAIL test_nuncstans (exit status: 134) FAIL test_nuncstans (exit status: 134) RPM build errors: RPM build errors:
Note, this will not affect production releases, on our test builds.
Metadata Update from @firstyear: - Issue assigned to firstyear
Metadata Update from @mreynolds: - Custom field component adjusted to None - Custom field origin adjusted to None - Custom field reviewstatus adjusted to None - Custom field type adjusted to None - Custom field version adjusted to None - Issue set to the milestone: 1.4 backlog
<img alt="0001-Ticket-49435-Fix-NS-race-condition-on-loaded-test-sy.patch" src="/389-ds-base/issue/raw/files/ea330cc702ff4919acb103d80609f87b16040b78c5ec3d7199f02b83a330785a-0001-Ticket-49435-Fix-NS-race-condition-on-loaded-test-sy.patch" />
Metadata Update from @firstyear: - Custom field reviewstatus adjusted to review (was: None)
Metadata Update from @mreynolds: - Custom field reviewstatus adjusted to ack (was: review)
commit 11974a0 To ssh://git@pagure.io/389-ds-base.git e622d95..11974a0 master -> master
Metadata Update from @firstyear: - Issue close_status updated to: fixed - Issue status updated to: Closed (was: Open)
60f66df..d67e8ae 389-ds-base-1.3.7 -> 389-ds-base-1.3.7
Metadata Update from @mreynolds: - Issue set to the milestone: 1.3.7.0 (was: 1.4 backlog)
389-ds-base is moving from Pagure to Github. This means that new issues and pull requests will be accepted only in 389-ds-base's github repository.
This issue has been cloned to Github and is available here: - https://github.com/389ds/389-ds-base/issues/2494
If you want to receive further updates on the issue, please navigate to the github issue and click on subscribe button.
subscribe
Thank you for understanding. We apologize for all inconvenience.
Metadata Update from @spichugi: - Issue close_status updated to: wontfix (was: fixed)
Login to comment on this ticket.