Hello.
I need your help with debugging a problem uncovered by OpenQA tests in Fedora. We have prepared an update for Python from 3.9.7 to 3.9.8 and an OpenQA test fails with the new version. The situation is not completely clear because there is no actual failure with some errors or tracebacks but systemctl stop ipa.service takes 95 seconds to finish which is more than the test is willing to wait so it's marked as failed by that timeout. In other updates for Fedora 34, the same command takes 8 seconds to finish.
systemctl stop ipa.service
All the information I currently have are recorded in the update: https://bodhi.fedoraproject.org/updates/FEDORA-2021-e152ce5f31
I have no clue where the problem might be so I'll start with reproducing the steps form the test and see whether I'm able to reproduce it but you might know where to take a look or what to test.
Any help is appreciated. Lumír
I am also afraid this might eventually happen on Python 3.10.1 in Fedora 35/36 and that it might also affect CentOS 9 Stream with Python 3.9.8.
We use mod_wsgi in Apache for our API. I wonder if that is hung up in some way. If you can get a reproducer then strace against Apache may help point to where it is deadlocking.
I'm now able to reproduce the problem and I've tried strace yet I don't have a clue where the problem actually lies. Note that you have to actually use the web UI to reproduce the problem.
Could you please take a look at the provided logs?
After one more day of debugging, I've created an upstream issue because it does not seem to me that the problem is in FreeIPA: https://github.com/GrahamDumpleton/mod_wsgi/issues/730
Could you please take a look and add your comments?
After a few comments in the mentioned upstream issue, it seems that it actually might be a problem in FreeIPA. Could you please take a look and comment there?
I'm looking into the issue now.
@cheimes do you plan to fix this also in Fedora 34? I mean there is a possible fix implemented in mod_wsgi already but it's not released and we probably won't update it in F34.
For the record, this underlying problem still exists with mod_wsgi 4.9.1 and Python 3.11. We had the problematic Python commit reverted in Fedora's Python 3.9 and 3.10 packages, but that reversion was dropped in the update to 3.11, and the problem showed up again.
Oh my :/
Was there any progress at all? I don't want to carry this revert forever.
Well, there's this comment from the mod_wsgi developer explaining he's added an option you can toggle to try a different behaviour on shutdown, and this one where he says that his opinion is this was a breaking change upstream, which perhaps implies we should file a Python issue to see if it can be revisited somehow?
Login to comment on this ticket.