#49435 NS race condition during test case
Closed: wontfix 6 years ago Opened 6 years ago by firstyear.

Issue Description

During the test case, a very very rare issue can be observed (maybe 1 in a million executions)

This appears to be a race between internal job rearm's state assertion, and the thread shutdown clearing the queues.

[ RUN      ] ns_set_data_test
Assertion failure: job->state == NS_JOB_NEEDS_ARM, at src/nunc-stans/ns/ns_thrpool.c:254
Assertion failure: job->state == NS_JOB_NEEDS_ARM, at src/nunc-stans/ns/ns_thrpool.c:254
ns_thrpool_new():  max threads, (4)
ns_thrpool_new():  max threads, (4)
stacksize (0), event q size (unbounded), work q size (unbounded)
stacksize (0), event q size (unbounded), work q size (unbounded)
new_ns_job 580 initial NS_JOB_WAITING
new_ns_job 580 initial NS_JOB_WAITING
setup_event_q_wakeup 580 state 7 moving NS_JOB_ARMED
setup_event_q_wakeup 580 state 7 moving NS_JOB_ARMED
new_ns_job 660 initial NS_JOB_WAITING
new_ns_job 660 initial NS_JOB_WAITING
ns_add_job 660 state 7 moving to NS_JOB_ARMED
ns_add_job 660 state 7 moving to NS_JOB_ARMED
internal_ns_rearm_job 660 state 4 moving to NS_JOB_ARMED
internal_ns_rearm_job 660 state 4 moving to NS_JOB_ARMED
work_q_notify 660 state 5
work_q_notify 660 state 5
work_job_execute 660 state 5 moving to NS_JOB_RUNNING
work_job_execute 660 state 5 moving to NS_JOB_RUNNING
work_job_execute 660 state 6 job func complete move to NS_JOB_WAITING
work_job_execute 660 state 6 job func complete move to NS_JOB_WAITING
ns_rearm_job 660 state 7 moving to NS_JOB_NEEDS_ARM
ns_rearm_job 660 state 7 moving to NS_JOB_NEEDS_ARM
internal_ns_rearm_job 660 state 4 moving to NS_JOB_ARMED
internal_ns_rearm_job 660 state 4 moving to NS_JOB_ARMED
work_q_notify 660 state 5
work_q_notify 660 state 5
work_job_execute 660 state 5 moving to NS_JOB_RUNNING
work_job_execute 660 state 5 moving to NS_JOB_RUNNING
work_job_execute 660 state 6 job func complete move to NS_JOB_WAITING
work_job_execute 660 state 6 job func complete move to NS_JOB_WAITING
ns_job_done 660 tp shutdown -> false state 7 setting NS_JOB_NEEDS_DELETE and queuing
ns_job_done 660 tp shutdown -> false state 7 setting NS_JOB_NEEDS_DELETE and queuing
event_q_notify enqueuing 660 with state 2
event_q_notify enqueuing 660 with state 2
event_q_wake attempting to wake event queue.
event_q_wake attempting to wake event queue.
event_q_wake result. 0
event_q_wake result. 0
ns_thrpool_shutdown initiated ...
ns_thrpool_shutdown initiated ...
new_ns_job 740 initial NS_JOB_WAITING
new_ns_job 740 initial NS_JOB_WAITING
internal_ns_rearm_job 740 state 4 moving to NS_JOB_ARMED
internal_ns_rearm_job 740 state 4 moving to NS_JOB_ARMED
work_q_notify 740 state 5
work_q_notify 740 state 5
event_cb 580 state 5 non-threaded, execute right meow
event_cb 580 state 5 non-threaded, execute right meow
work_job_execute 580 state 5 moving to NS_JOB_RUNNING
work_job_execute 580 state 5 moving to NS_JOB_RUNNING
wakeup_cb 580 state 6 wakeup_cb
wakeup_cb 580 state 6 wakeup_cb
new_ns_job 820 initial NS_JOB_WAITING
new_ns_job 820 initial NS_JOB_WAITING
internal_ns_rearm_job 820 state 4 moving to NS_JOB_ARMED
internal_ns_rearm_job 820 state 4 moving to NS_JOB_ARMED
work_q_notify 820 state 5
work_q_notify 820 state 5
new_ns_job 900 initial NS_JOB_WAITING
new_ns_job 900 initial NS_JOB_WAITING
internal_ns_rearm_job 900 state 4 moving to NS_JOB_ARMED
internal_ns_rearm_job 900 state 4 moving to NS_JOB_ARMED
work_q_notify 900 state 5
work_q_notify 900 state 5
worker_thread_func notified to shutdown!
worker_thread_func notified to shutdown!
internal_ns_job_done 740 state 5 moving to NS_JOB_DELETED
internal_ns_job_done 740 state 5 moving to NS_JOB_DELETED
worker_thread_func shutdown complete!
worker_thread_func shutdown complete!
get_new_event_requests Dequeuing 660 with state 2
get_new_event_requests Dequeuing 660 with state 2
update_event 660 state 2
update_event 660 state 2
internal_ns_job_done 660 state 2 moving to NS_JOB_DELETED
internal_ns_job_done 660 state 2 moving to NS_JOB_DELETED
work_job_execute PERSIST and RUNNING, remarking 580 as NS_JOB_NEEDS_ARM
work_job_execute PERSIST and RUNNING, remarking 580 as NS_JOB_NEEDS_ARM
work_job_execute 580 state 4 job func complete, sending to rearm...
work_job_execute 580 state 4 job func complete, sending to rearm...
worker_thread_func notified to shutdown!
worker_thread_func notified to shutdown!
new_ns_job 9e0 initial NS_JOB_WAITING
new_ns_job 9e0 initial NS_JOB_WAITING
worker_thread_func notified to shutdown!
worker_thread_func notified to shutdown!
internal_ns_job_done 900 state 5 moving to NS_JOB_DELETED
internal_ns_job_done 900 state 5 moving to NS_JOB_DELETED
internal_ns_rearm_job 9e0 state 4 moving to NS_JOB_ARMED
internal_ns_rearm_job 9e0 state 4 moving to NS_JOB_ARMED
work_q_notify 9e0 state 5
work_q_notify 9e0 state 5
worker_thread_func shutdown complete!
worker_thread_func shutdown complete!
ns_thrpool_wait has begun
ns_thrpool_wait has begun
ns_thrpool_wait joined thread, result 0
ns_thrpool_wait joined thread, result 0
internal_ns_job_done 820 state 5 moving to NS_JOB_DELETED
internal_ns_job_done 820 state 5 moving to NS_JOB_DELETED
worker_thread_func shutdown complete!
worker_thread_func shutdown complete!
worker_thread_func notified to shutdown!
worker_thread_func notified to shutdown!
internal_ns_job_done 9e0 state 5 moving to NS_JOB_DELETED
internal_ns_job_done 9e0 state 5 moving to NS_JOB_DELETED
worker_thread_func shutdown complete!
worker_thread_func shutdown complete!
ns_thrpool_wait joined thread, result 0
ns_thrpool_wait joined thread, result 0
ns_thrpool_wait joined thread, result 0
ns_thrpool_wait joined thread, result 0
ns_thrpool_wait joined thread, result 0
ns_thrpool_wait joined thread, result 0
ns_thrpool_wait complete, retval 0
ns_thrpool_wait complete, retval 0
ns_thrpool_destroy
ns_thrpool_destroy
ns_thrpool_destroy 580 state 4 moving to NS_JOB_NEEDS_DELETE
ns_thrpool_destroy 580 state 4 moving to NS_JOB_NEEDS_DELETE
event_q_notify enqueuing 580 with state 2
event_q_notify enqueuing 580 with state 2
event_q_wake attempting to wake event queue.
event_q_wake attempting to wake event queue.
event_q_wake result. 0
event_q_wake result. 0
FAIL test_nuncstans (exit status: 134)
FAIL test_nuncstans (exit status: 134)
RPM build errors:
RPM build errors:

Note, this will not affect production releases, on our test builds.


Metadata Update from @firstyear:
- Issue assigned to firstyear

6 years ago

Metadata Update from @mreynolds:
- Custom field component adjusted to None
- Custom field origin adjusted to None
- Custom field reviewstatus adjusted to None
- Custom field type adjusted to None
- Custom field version adjusted to None
- Issue set to the milestone: 1.4 backlog

6 years ago

Metadata Update from @firstyear:
- Custom field reviewstatus adjusted to review (was: None)

6 years ago

Metadata Update from @mreynolds:
- Custom field reviewstatus adjusted to ack (was: review)

6 years ago

commit 11974a0
To ssh://git@pagure.io/389-ds-base.git
e622d95..11974a0 master -> master

Metadata Update from @firstyear:
- Issue close_status updated to: fixed
- Issue status updated to: Closed (was: Open)

6 years ago

60f66df..d67e8ae 389-ds-base-1.3.7 -> 389-ds-base-1.3.7

Metadata Update from @mreynolds:
- Issue set to the milestone: 1.3.7.0 (was: 1.4 backlog)

6 years ago

389-ds-base is moving from Pagure to Github. This means that new issues and pull requests
will be accepted only in 389-ds-base's github repository.

This issue has been cloned to Github and is available here:
- https://github.com/389ds/389-ds-base/issues/2494

If you want to receive further updates on the issue, please navigate to the github issue
and click on subscribe button.

Thank you for understanding. We apologize for all inconvenience.

Metadata Update from @spichugi:
- Issue close_status updated to: wontfix (was: fixed)

3 years ago

Login to comment on this ticket.

Metadata