With a distro hat on,I think that -s would be a good change, probably for any thing that we ship. With a python user hat on I am somewhat conflicted. I know there's people who rely on being able to override system packages with what is in their local directories for certain system packages ("i need to override the system requests to work around a bug in fedora's version", type uses). But there's also people who are surprised when it happens - they expect that for their locally created software but not for the system packages.

Maybe we could convince the people who make use of it that they're doing it wrong? Or perhaps limiting it to daemons is a good compromise. It's probably best to discuss this with other distro maintainers and take it to python-dev for discussion (dmalcolm took the question of setting the default encoding to utf-8 to python-dev and was saved from making a grave mistake because of that discussion.)

-E is not a good change. The pythonpath env var is widely used and relied upon. If there's specific software that needs to disregard the env vars they should unset the env vars when they startup (probably in a shell wrapper so the user can modify it in a normal way.). But be wary, this is the equivalent of a shell script or c program "sanitizing" path and ld_library_path. There may be specific circumstances where this is warranted but it is changing a fundamental assumption that a user may have about their system. (And again, using the shell wrapper makes it easy for us to sanitize both path and ld_library_path at the same time and allow the user to specify different values if they need to. -E does neither of those things.)

Thanks for the discussion. Sounds like python-dev would be a good place to get more comments.

This appears to be able to be set for setuptools users with:

python setup.py build --executable="%{__python} -Es"

or whatever we decide upon. -E does seem bad - tests in %check for some packages and MPI modules are setting PYTHONPATH.

dwalsh - thoughts on dropping -E ?

According to http://article.gmane.org/gmane.comp.python.devel/151991 debian may be moving this way as well, with -Es for python2 and -I for python3.

Also (http://article.gmane.org/gmane.comp.python.devel/151993) musings over having /usr/bin/spython (system python) that is locked down.

Given that I had no idea what this did, and sys.path was the same for me both before and after adding -s ... then for everyone else, here is roughly what it does:

-s (and other things) sets sys.flags.no_user_site which is used in check_enableusersite() (and others) to set ENABLE_USER_SITE, which (among other things) adds ~/.local/lib/python?.?/site-packages to the path if it exists (USER_SITE in: python -m site).

My personal guess is that a lot of Fedora python apps. should have this set, as they'll be a lot of confusion if a user accidentally overrides something and stuff stops working. But maybe someone is using this to test bugfixes (use VMs?) or randomly monkey patch something (use override rpms?) so I'm open to arguments that we shouldn't force all system packages to set it.

I'm going to add it to the meeting for tomorrow.

To be honest, if someone is using this to test bugfixes, they should just copy the executable to ~/bin and change the first line. I can't imagine this is any real impediment to that workflow (xkcd coming about every change breaking someone's workflow notwithstanding) and if it gives any measurable security benefit or even reduction in end-user surprise then we should do it wherever possible.

Can't say I have enough knowledge to argue the benefit of -sE versus just -s.

Also, only semi-related, why don't we macro-ize %{__python2} setup.py build --executable="%{python} -Es"

Oh, my semi-related comment is addressed by https://fedorahosted.org/fpc/ticket/281 which is unfortunately stalled.

I can say with certainty that people are doing both of the things that James mentions. In fact, I've done it myself from time to time. (And then later sat on the other side of the line and wondered why my rpm update didn't appear to have any effect (only to discover that I'd forgotten about something I installed into ~/.local ) but as tibbs says, any change will break someone's workflow. And -s 's benefits for system installed programs might overwhelm the change in workflow for those who use it.

If we do macros we need to be sure to use python2 in the shebang rather than just python.

Some additional context here:

1. "python -m site" is the quickest and simplest way to check your current sys.path setup

2. As Slavek says, system packages should likely at least be using "-s" (to disable user site-packages), as packages installed there can override the system site-packages (they can't easily override the standard library though).

3. Failing to set "-E" is even more of a concern, as packages on PYTHONPATH can override the standard library in addition to system site packages

4. Even "-Es" is potentially inadequate, as packages co-located with the script being run will always take precedence over the standard library and system site-packages, which is why Python 3.4 added the -I option to enable "isolated mode": https://docs.python.org/3/whatsnew/3.4.html#whatsnew-isolated-mode (-Es remains the highest level of isolation available to Python 2.7)

5. When isolation is enabled in the shebang line, you can still invoke the Python interpreter directly to run a script "unisolated". It's only Python scripts invoked indirectly that you wouldn't be able to influence any more, and I consider that a feature of isolated mode, not a bug

6. I would indeed like to see a dedicated "system Python" binary for CPython with different default settings, but it requires some architectural work to make it happen: https://www.python.org/dev/peps/pep-0432/#a-system-python-executable (I haven't updated that to reflect the fact we accepted the "-I" option as a simpler interim enhancement, but the general description of the problem is still right)

Oh, and one more point: if this change causes problems for someone's workflow, it's a fairly strong sign that they should probably be using virtualenv, conda, or an SCL rather than using the system Python and site-packages directly.

In particular, "virtualenv --system-site-packages" gives you a virtualenv with full read access to the system site packages, but anything installed into the virtualenv itself will take precedence.

In my opinion, we should use -s, and we shouldn't use -E.

I completely agree with Toshio's comment 9 - every change will break someone's workflow, the question is whether the benefits outweigh the breakages. I've already talked to several developers who broke DNF by installing a different version of "requests" in their ~/.local/ directory.

(Just FTR I also think that if/when there is a /usr/local/lib[64]/... directory for pip-installed packages, system binaries shouldn't see it as well.)

As for not using -E, my reasoning is that we should stay compatible with e.g. LD_LIBRARY_PATH and similar. If I define LD_LIBRARY_PATH to a directory containing different libfoo, the linker will use that over the system one. I have the same expectation for PYTHONPATH (and other env variables).

Speaking of -I, I like that it also omits the script directory, but then again it implies -E, which I think we shouldn't use.

We discussed this at today's meeting (http://meetbot.fedoraproject.org/fedora-meeting-1/2015-03-19/fpc.2015-03-19-17.01.txt):

• 513 Use python -Es in shbang (geppetto, 17:11:52)

• LINK: https://fedorahosted.org/fpc/ticket/513 (geppetto, 17:11:57)
• ACTION: Mandatory use of -s in she-bang of python programs (+1:5,
  0:0, -1:0) (geppetto, 17:26:28)
• ACTION: tibbs Will work on macros to vote on next week. (geppetto,
  17:32:22)
• LINK: https://bugzilla.redhat.com/show_bug.cgi?id=953704 (tibbs|w,
  17:32:42)

Here's a better link to the python-dev thread https://mail.python.org/pipermail/python-dev/2015-March/138812.html but it went off topic pretty quickly.

As tibbs noted this could be macroized:

For example:[[BR]]
%{python2_build} which would expand to %{!__python2} setup.py build --executable="%{!__python2} -s"[[BR]]
%{python2_install} which would expand to %{!__python2} setup.py install [[BR]]
similar macros for python3_build and python3_install[[BR]]
%{python_build} (and install) which would use %{!__python} macro inside

Please do not promote any more unversioned macros and if you feel you absolutely must do that, use %{__python2} inside of them. As a distribution starting to transition to python3 we have to get people using /usr/bin/python2 and /usr/bin/python3 and not using /usr/bin/python or we'll get to a time where a lot of things break.

I would like to use unversioned macros within applications specfiles which use only the default platform python which is %{!__python}.

I'd forgotten about that. If that was approved then it can work as long as {{{%{__python} }}} maps to {{{ %{__python2} }}} or {{{ %{__python3} }}} (whatever was approved previously). We want #!/usr/bin/python2 and #!/usr/bin/python3 to show up in application shebang lines. We just don't want to have to specify the version explicitly in the spec file.

I don't recall if that was approved, though. So if it wasn't then it won't make any difference here.

FWIW, I'm also in favor of the "system python" idea. In addition to more locked-down defaults, we could split out space-consuming modules like 2to3 and idle.

We discussed this at this weeks meeting (http://meetbot.fedoraproject.org/fedora-meeting-1/2015-06-11/fpc.2015-06-11-16.00.txt):

• 513 Use python -Es in shbang (geppetto, 16:24:53)

• LINK: https://fedorahosted.org/fpc/ticket/513 (geppetto, 16:24:54)
• ACTION: tibbs Deal with non-voted on parts of 513 in other tickets,
  macros. etc. (geppetto, 16:47:33)