gssproxy

#215 gssproxy consumes >1024 fds under normal operation?
Closed: Deferred by simo. Opened by ralston.

Closed: Deferred
Reopen Issue

We're having a bizarre problem with gssproxy-0.4.1-13.el7.x86_64 on RHEL 7.4, and we're wondering if anyone can help explain what we're seeing.

We're using the gssproxy interposer feature with httpd, as described here:

https://pagure.io/gssproxy/blob/master/f/proxy/docs/Apache.md

The httpd instance runs Atlassian JIRA via Tomcat.

On a fairly reliable basis, gssproxy exhausts all file descriptors, at which point it becomes non-responsive.

We captured an lsof (attached) of gssproxy during one such episode, and are confused about what we're seeing.

Specifically, it looks like the lsof shows 5 simultaneous in-flight requests (5 socket connections), and in the process of serving those requests, gssproxy has managed to consume 1K open file descriptors, mostly with deleted (but still open) temporary files, because it looks like every open fd is replicated across every gssproxy thread.

We're going to try increasing LimitNOFILE for gssproxy.service to 8192 (from the default of 1024:2048), to see if that prevents the problem, as /proc/sys/fs/file-nr typically reports >300,000 system-wide fds available.

But it doesn't seem right that the gssproxy interposer can consume 1K of fds just by serving a small number of simultaneous requests.

Are we misinterpreting the lsof of gssproxy? Or does gssproxy really need that many fds available to serve even a small number of simultaneous requests?
gssproxy-lsof.txt

This is a downtream bug; please file it downstream (i.e., RHEL bugs on Red Hat's bugtracker) unless you can reproduce it with master. Thanks!

Metadata Update from @rharwood:
- Issue close_status updated to: Invalid

@rharwood but is it a bug? Or is it actually normal/expected for gssproxy to need a boatload of fds to serve simultaneous requests?

(That's why I opened an issue here first, instead of opening a support case with Red Hat—I was hoping for clarification that the behavior we were seeing was not expected behavior for gssproxy. Because if the answer is that it is expected behavior, the correct resolution is for us to just increase LimitNOFILE, instead of pound a support case at Red Hat.)

Metadata Update from @ralston:
- Issue status updated to: Open (was: Closed)

@ralston it's a bug
@rharwood please do not close this way

@ralston can you please verify this bug goes away if you start gssproxy with the environemt variable KRB5RCACHETYPE=none set ?

@simo confirmed: if we set the environment variable KRB5RCACHETYPE=none, the problem does not occur. (It looks like an infinite loop is occurring somewhere, which is what eventually leads to gssproxy running out of fds.)

Also, this seems to be specifically a problem with mod_auth_kerb when used with httpd. If we use mod_auth_gssapi, we cannot reproduce the problem.

(We have tended to use mod_auth_kerb, even on RHEL7, because we have a mix of RHEL6 and RHEL7 web servers, and since RHEL6 lacks mod_auth_gssapi, sticking with mod_auth_kerb was the path of least resistance.)

I am seeing similar symptoms but the setup is different:
- gssproxy + Apache + mod_auth_gssapi (gssproxy-0.7.0-4.el7.x86_64, httpd-2.4.6-67.el7.centos.6.x86_64, mod_auth_gssapi-1.5.1-2.el7.x86_64)
- The machine has been joined to an AD domain, using SSSD, but gssproxy is using a different domain, not in same forest

In this case, gssproxy is using all 6k open files (almost all sockets) and ends up in an about endless loop, for example:
Dec 1 11:42:21 k01 gssproxy: gssproxy[631]: Error connecting client: (24:Too many open files)
Dec 1 11:42:21 k01 gssproxy: gssproxy[631]: Error connecting client: (24:Too many open files)
Dec 1 11:42:21 k01 gssproxy: gssproxy[631]: Error connecting client: (24:Too many open files)
Dec 1 11:42:21 k01 gssproxy: gssproxy[631]: Error connecting client: (24:Too many open files)
Dec 1 11:42:21 k01 gssproxy: gssproxy[631]: Error connecting client: (24:Too many open files)
Dec 1 11:42:21 k01 gssproxy: gssproxy[631]: Error connecting client: (24:Too many open files)
Dec 1 11:42:21 k01 gssproxy: gssproxy[631]: Error connecting client: (24:Too many open files)
Dec 1 11:42:21 k01 gssproxy: gssproxy[631]: Error connecting client: (24:Too many open files)
Dec 1 11:42:21 k01 gssproxy: gssproxy[631]: Error connecting client: (24:Too many open files)

The above are just few lines in the same second - total log events count over 2000 per second!

Not sure what threw gssproxy off, but it seems like SSSD (that talks to different AD domain from gssproxy) has "Preauthentication failed." error as well... edit: probably KVNO mismatch caused the SSSD error, perhaps related, perhaps not.
Edited by vojamo

I think we have this fixed in master and Fedora right now (gssproxy >= 0.7.0-25.fc27). If you're unable to test with either master or Fedora, I may be able to make you some RPMs for el7 to try.

After 13 days it has recurred.. I cannot compile anything on this box (the issue manifests only on one production EL7 box).

Could the fc27 RPM be installed as a drop-in replacement for vanilla EL7 gssproxy?

You'll need to rebuild it for el7 (I recommend using gssproxy-0.7.0-29 as a starting point) but otherwise I believe it should work. If you run into trouble or would like me to do builds instead, I'm happy to; just let me know.

@vojamo, any updates on this?

W.r.t mod_auth_kerb, it is buggy. In Gentoo we developed a patch to avoid SEGV:
https://bugs.gentoo.org/673066

@rharwood I had abandoned the project where two domains were used and the problem did not happen on any of the several servers that use only one domain with the similar setup.

I may revisit the project later.

OK, but you should look into the mod_auth_kerb bug as it will SEGV anyhow randomly

Project has moved please reopen here if still an issue:
https://github.com/gssapi/gssproxy/issues

Metadata Update from @simo:
- Issue close_status updated to: Deferred
- Issue status updated to: Closed (was: Open)
Metadata
None
None
None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.