In MBS, we often encounter a "too many open files" error due to a file descriptor leak (e.g. #7652). In recent months, we've discovered two leaks in the internal MBS instance that contributed to this issue. One of which was https://pagure.io/koji/issue/1652. The fix for that was deployed within the last couple of weeks in Fedora. That was the worst of the two leaks.
The other leak relates to python-krbv leaking several file descriptors relating to the Kerberos replay cache. For example, lsof -a -p <pid> on the MBS backend shows several entries in the format of "/var/tmp/krb5_RC* (deleted)". python-krbv is used when performing Kerberos authentication with Brew.
lsof -a -p <pid>
In the internal MBS stage instance, we updated python-requests-kerberos to an internal build of v0.10.0 and python-kerberos to an internal build of v1.2.5. This causes the Kerberos authentication with Brew to use the newer "GSSAPI login". We've been meaning to do this anyways so that MBS can be fully Python 3 compliant since python-krbv is not supported on Python 3. Since this change, the leak has not reappeared.
Could the Fedora Infrastructure team please provide newer builds of python-kerberos and python-requests-kerberos in epel7-infra-stg so that MBS in stage can install them to get rid of this file descriptor leak? If it helps, I can attach the SRPMs used for those builds internally.
After testing in stage, we can do the same in production.
No particular date, but doing this soon is appreciated.
Please let us know which versions of these packages were used (and src.rpm etc if possible). This will help make sure we don't add more misery.
Metadata Update from @smooge: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: mbs
Thanks @smooge! Here are the SRPMs that were used for the internal builds: https://mprahl.fedorapeople.org/python-requests-kerberos-0.10.0-8.el7eso.src.rpm https://mprahl.fedorapeople.org/python-kerberos-1.2.5-5.el7eso.src.rpm
Metadata Update from @smooge: - Issue assigned to smooge
Hi @smooge, any luck with this?
I got sidelined before my PTO with some 'important projects'. I hope to look at this when I get back next week from my parents.
Thanks @smooge! If possible, it'd be nice to have this done by January 10th. No rush on this though.
Hi @smooge, do you think you'll be able to get to this soon?
Metadata Update from @mohanboddu: - Issue assigned to mohanboddu (was: smooge)
Metadata Update from @cverna: - Assignee reset
Metadata Update from @cverna: - Issue assigned to nphilipp
@mprahl I'll take this one and assume that getting the current Fedora versions is fine for you, too, right?
Notes to myself:
BuildError: package python-kerberos not in list for tag epel7-infra-candidate
epel7-infra
root.log
DEBUG util.py:600: No matching package to install: 'python3-pytest' DEBUG util.py:600: No matching package to install: 'python3-mock' DEBUG util.py:600: No matching package to install: 'python2-kerberos' DEBUG util.py:600: No matching package to install: 'python3-kerberos' DEBUG util.py:600: No matching package to install: 'python3-cryptography'
python*-kerberos is clear, the other ones I have to find out if we should build them in EPEL proper or only for infra.
python*-kerberos
I found out that the pytest, mock and cryptography packages are available as python36-* packages for EPEL7. I'm not sure if the mock one is even needed because as unittest.mock, it's part of the Python 3 stdlib.
pytest
mock
cryptography
python36-*
unittest.mock
For this, we need to get Python3-only versions of python-kerberos and python-requests-kerberos into EPEL7. Here are the review tickets:
python-kerberos
python-requests-kerberos
@mprahl Just checking, with the above I'm assuming that it's okay for you to get Python 3 only versions of the packages. This makes maintenance easier, as they can be normal packages -- for Python 2 versions we'd have to work around the older versions present in plain RHEL.
Metadata Update from @nphilipp: - Issue priority set to: Waiting on Reporter (was: Waiting on Assignee)
@nphilipp sorry about the confusion. This is just to add newer Python 2 versions in the Fedora Infra epel7-infra-stg tags/repo and not in EPEL 7.
@mprahl, ahh the confusion was a little self-induced, if Python 3 packages would work for you, we could get by without special infra sauce here. Anyway, the packages are ready to be reviewed (and as I understand it, would be necessary so that you can make the jump to Python 3 with MBS eventually).
I'll look into building the SRPMs you linked above in the infra tag, or do you need/want newer versions (as some time has passed since you linked them)?
This needs python-kerberos cleared for the infra tag (as there is an older version in RHEL7), see issue #8843.
Metadata Update from @nphilipp: - Issue marked as depending on: #8843 - Issue priority set to: Waiting on Assignee (was: Waiting on Reporter)
Hi @nphilipp, the MBS code does now work on Python 3 but I don't see a major benefit in having the deployed instance of MBS run on Python 3 on RHEL 7. If such an effort were to occur, I'd rather see it on RHEL 8 so that newer packages can be used.
Regardless, I no longer maintain MBS and so it is not my decision. @mikem and @breilly are the maintainers, so I'll defer this to them.
@nphilipp by the way, thank you for working on this ticket. :smile:
Ah, then I'll kill off the reviews.
python-kerberos-1.2.5-5.el7.infra.1 and python-requests-kerberos-0.10.0-8.el7.infra are available in epel7-infra-stg, closing.
python-kerberos-1.2.5-5.el7.infra.1
python-requests-kerberos-0.10.0-8.el7.infra
epel7-infra-stg
Metadata Update from @nphilipp: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.