taskotron

#235 fedmsg-hub is running out of memory
Closed: Fixed Opened by kparal.

Closed: Fixed
Reopen Issue

This has happened for the second time (that I know of), it's no coincidence. On all dev, stg and prod at roughly the same time fedmsg-hub stopped triggering new jobs. When you expect the available memory, the process consumes all available memory (~3GB) and all swap space (~2GB). Restarting the process "fixes" the problem.

There's clearly some memory leak. But it's not trivial to find out where it is, whether in fedmsg-hub itself or in our adjustments of it (taskotron-trigger). It can be happening for each message and growing over time, or it can happen just for some specific messages, we don't know yet.

On taskotron-stg01, I saved a core file using gdb into /root/fedmsg-hub-leak.core. Since the issue doesn't seem to be going away, we'll need to figure out what leaks in there.

Metadata Update from @kparal:
- Issue priority set to: High
- Issue tagged with: infra

A few pointers for start:
https://stackoverflow.com/questions/1435415/python-memory-leaks
http://blog.korbakov.com/2013/09/22/hunting-memory-leaks-in-running-python-process.html

It seems pyopenssl is to blame here:

<bowlofeggs> jcline recently fixed some kind of memory leak in pyopenssl that had some relationship to fedmsg - i wonder if that was the cause here?
<jcline> kparal, bowlofeggs, that is indeed the pyopenssl fixes. Both leaks should be fixed in pyOpenSSL 17.3.0, and I backported one of them to older releases. I can get the other one in now it's merged upstream.
<jcline> Although it looks like the maintainer actually just closed all my PRs without accepting them so...
<jbowen> Good <time of day>
<jbowen> Is there any chance I can get membership approval for the QA group (FAS username is the same as my IRC username)
<jcline> kparal, is this consumer on f25?
<tflink> jcline: yeah, it is
<jcline> tflink, ah okay, thanks
* jcline backports patches further :(

Meeting outcome:

  * to fix the fedmsg memory issue, the plan is to use side-builds in
    the short term if needed, upgrade the master to f26 in the medium
    term and upgrade everything to f26 in the longer term  (tflink,
    14:36:52)

https://meetbot.fedoraproject.org/fedora-meeting-1/2017-09-18/fedora-qadevel.2017-09-18-14.01.log.html

These updates fixed the problem on dev:
https://bodhi.fedoraproject.org/updates/FEDORA-2017-235298fa58
https://bodhi.fedoraproject.org/updates/pyOpenSSL-16.2.0-2.fc25

We need to deploy the same to stg and prod.

The fix was deployed to prod, stg is not updated yet (due to login issues).

Deployed even to stg, closing.

Metadata Update from @kparal:
- Issue assigned to kparal
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)
Metadata

infra

None
None
Lowest
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.