#3169 [HOTFIX] Fix python hash table collisions across all of our WSGI apps
Closed: Upstream 2 years ago Opened 7 years ago by lmacken.

= bug description =

https://bugzilla.redhat.com/show_bug.cgi?id=750555

= bug analysis =

To mitigate this issue, we must set the PYTHONHASHSEED=random environment variable before python loads.

According to the [https://code.google.com/p/modwsgi/wiki/ConfigurationGuidelines#Application_Configuration mod_wsgi documentation], there is no way to change os.environ via apache directives.

The only way I could figure out how to solve this issue is by altering the apache initscript.

= fix recommendation =

{{{

!diff

--- /etc/init.d/httpd.orig 2012-03-01 01:14:29.071120965 +0000
+++ /etc/init.d/httpd 2012-03-01 01:14:31.203430537 +0000
@@ -27,6 +27,9 @@
. /etc/sysconfig/httpd
fi

+# Allow environment variables to be set in /etc/sysconfig/httpd
+eval $ENVSET
+
# Start httpd in the C locale by default.
HTTPD_LANG=${HTTPD_LANG-"C"}

--- /etc/sysconfig/httpd.orig 2012-03-01 01:14:45.499113785 +0000
+++ /etc/sysconfig/httpd 2012-03-01 01:14:48.501114555 +0000
@@ -29,3 +29,6 @@
# location needs to be reported in the PIDFILE.
#
#PIDFILE=/var/run/httpd/httpd.pid
+
+# Mitigate Python hash table collisions
+ENVSET='export PYTHONHASHSEED=random'

}}}


We'll also want to confirm that sys.flags.hash_randomization is properly set, once the patched Python package gets released.

I just added support for hash seed randomization to mod_wsgi, and sent the patch upstream:

https://groups.google.com/group/modwsgi/browse_thread/thread/f8b71ca48a13e244#

This is a much cleaner approach as opposed to having to mess with apache init scripts.

I just implemented the initial fix recommendation in staging.

Replying to [comment:3 lmacken]:

I just implemented the initial fix recommendation in staging.

I did this by copying modules/{hotfix,httpd} over to modules-staging, and included the hotfix in the httpd::mod_wsgi class, and on the {pkgs,koji}01.stg nodes.

Should we wait for a fixed mod_wsgi? or push forward with the init script change for now?

Definitely push forward with the init script change. At least some of the hosts are going to need to keep the init script change even after mod_wsgi is deployed (the pkgs.fp.o upload.cgi, for instance).

ok. I'd say we should push it out to production later today then.

It's been in staging for a week or so without ill effect I can see.

Yeah, I agree we should push forward with our current hotfix. Pushing it into production tonight sounds good to me.

We have pushed out the init script version of this.

When a fixed mod_wsgi shows up, we will need to remember to drop this at that time for mod_wsgi hosts.

We will still need to carry this or something like it for non mod_wsgi applications, or get some kind of fix upstream in httpd or the like.

I just enabled this hotfix in production and did a hard httpd restart on all effected machines.

It looks like 4.1.0 upstream supports the WSGIPythonHashSeed directive, but Fedora is still on 3.5. So we'll need to keep our environment variable workaround for a bit longer.

http://modwsgi.readthedocs.org/en/latest/release-notes/version-4.1.0.html

@lmacken did this ever get resolved?

It didnt in RHEL7 at least... so we are still carrying hotfix.

I guess we can close this however, as there's not really anything to do here. Once there is a fixed version (RHEL8 I guess) we can drop the hotfix.

Metadata Update from @kevin:
- Issue close_status updated to: Upstream
- Issue status updated to: Closed (was: Open)

2 years ago

Login to comment on this ticket.

Metadata