DNF was [https://bugzilla.redhat.com/show_bug.cgi?id=1230820 accused] that it wrongfully uses the "-OO" switch in the shebang line no matter that there is a Fedora guideline that forbids this switch.
Nick Coghlan [https://bugzilla.redhat.com/show_bug.cgi?id=1230820#c4 said] that there is a bug in Python < 3.5 regarding this switch and that the switch should not be used in a general purpose environment like Fedora:
The main practical benefit of -OO is actually the memory reduction - there's no point consuming memory for docstrings when you don't need them at the runtime. However, for Python 3.4 and earlier, using -OO in a shared environment with other code that uses -O instead is a bug, because the two optimisation levels conflict on the ".pyo" extension when caching bytecode files. Since IDEs (et al) need the docstrings, that means general purpose environments (like a Linux distro) need to disallow the use of -OO for execution of Python programs. PEP 488 fixes that collision for Python 3.5+ by including the optimisation level in the generated filenames used by the bytecode cache: https://www.python.org/dev/peps/pep-0488/ I'd suggest adjusting the Python packaging policy to state the following: -OO should never be used in Python 2 shebang lines (or other Python invocations) -OO should never be used in Python 3 shebang lines (or other Python invocations) for Python 3.4 and earlier In the context of current Fedora, that means this bug report is valid, and dnf should stop using -OO for the time being. That then raises the question of "When will Fedora switch to Python 3.5?". We haven't actually set a target release yet, but the two immediately relevant schedule docs are: Fedora 23: https://fedoraproject.org/wiki/Releases/23/Schedule Python 3.5: https://www.python.org/dev/peps/pep-0478/ The upstream 3.5rc1 release is due on August 9, the final release is due on September 13. To switch in F23 that would mean: getting a system-wide change for a Python 3 upgrade approved by the F23 deadline on Jun 26 getting a beta release incorporated by the testability deadline on July 28 (this would correspond to 3.5b3 upstream, which is due for release on July 5) F23 Alpha would ship with a Python 3.5 beta release F23 Beta would ship with a Python 3.5 release candidate F23 final would ship with the Python 3.5.0 final release I'll start a thread on the Fedora python-devel list about the idea of pursuing that more aggressive Python 3.5 adoption cycle, rather than waiting until Fedora 24.
The main practical benefit of -OO is actually the memory reduction - there's no point consuming memory for docstrings when you don't need them at the runtime.
However, for Python 3.4 and earlier, using -OO in a shared environment with other code that uses -O instead is a bug, because the two optimisation levels conflict on the ".pyo" extension when caching bytecode files. Since IDEs (et al) need the docstrings, that means general purpose environments (like a Linux distro) need to disallow the use of -OO for execution of Python programs.
PEP 488 fixes that collision for Python 3.5+ by including the optimisation level in the generated filenames used by the bytecode cache: https://www.python.org/dev/peps/pep-0488/
I'd suggest adjusting the Python packaging policy to state the following:
In the context of current Fedora, that means this bug report is valid, and dnf should stop using -OO for the time being.
That then raises the question of "When will Fedora switch to Python 3.5?". We haven't actually set a target release yet, but the two immediately relevant schedule docs are:
Fedora 23: https://fedoraproject.org/wiki/Releases/23/Schedule Python 3.5: https://www.python.org/dev/peps/pep-0478/
The upstream 3.5rc1 release is due on August 9, the final release is due on September 13. To switch in F23 that would mean:
I'll start a thread on the Fedora python-devel list about the idea of pursuing that more aggressive Python 3.5 adoption cycle, rather than waiting until Fedora 24.
I would like to suggest adjusting the guidelines as he suggests above. I believe that the priority is critical since using the switch supposedly breaks some software.
Replying to [ticket:542 rholy]:
no matter that there is a Fedora guideline that forbids this switch.
I mean "no matter that there is '''no''' Fedora guideline that forbids this switch." I hope you understand me.
Nick Coghlan said that there is a bug in Python < 3.5
Well, he didn't say that it's a '''bug'''. It's just that the feature was not designed to be used in such environments.
I would suggest adding ncoghlan's notes to the byte compiling section of the guidelines. Here's an attempt at a Draft:
https://fedoraproject.org/w/index.php?title=User%3AToshio%2FPython_Optimization_Level&diff=415635&oldid=415626
I don't think that it is necessary to forbid -OO-compiled files in packages. IIUIC, the only problem is the recompilation caused by running python as root. I think that if a package is OK with the fact that IDEs cannot inspect it, it's its choice.
Hm, I was thinking about it again. Since packages already has to own their "pyo" files and since these files doesn't seem to be regenerated if "-OO" is used (unless the "py" file is changed), I think that Fedora doesn't need this policy. I believe that the modification of the "py" files is at the sysadmin's own risk (I mean, not something supported) and the recompilation is one of these risks.
We already de facto disallow OO compiled files in packages. This is just putting that into the guidelines as you requested instead of leaving it as what the brp-python-bytecompile script does.
A sysadmin should be able to safely modify a .py file without fearing that scripts that Fedora ships will cause that to break their system in new and mysterious ways.
I believe that the modification of the "py" files is at the sysadmin's own risk and the recompilation is one of these risks. A sysadmin should be able to safely modify a .py file without fearing that scripts that Fedora ships will cause that to break their system in new and mysterious ways.
Yeh, recompilation as a risk seems fine ... nothing working because of the recompilation not so much. But it's kind of a weird edge case.
Yeah, I wouldn't have thought this deserved a guideline but rholy wanted one before removing the -OO from dnf's shebang. dnf seems like it is especially prone to causing problems since it is both run as root and it runs from a cron job.
We really don't need a guideline for everything. Can't common sense win on occasion? I mean, the breakage here is obvious.
If people are tripping over this then sure, it warrants a line somewhere. But it seems in this case the parties involved should know better, and the fact that things break makes the proper course of action pretty clear regardless of whether there's a guideline about it.
@rholy what do you think? Does the opinion of ncoghlan, myself, seemingly a few of the FPC that we shouldn't be using -OO here make you rethink your stance on removing -OO in dnf expressed in https://bugzilla.redhat.com/show_bug.cgi?id=1230820#c3 and https://fedorahosted.org/fpc/ticket/542#comment:4 ? I would have thought that this was common sense as well but if you still want a guideline in place before making the change, well, I'll do what I can to make for a better Fedora experience.
This could maybe be added to some kind of common bugs wiki page outside of the packaging wiki, so that it is generally editable.
Replying to [comment:6 james]:
I believe that the modification of the "py" files is at the sysadmin's own risk and the recompilation is one of these risks. A sysadmin should be able to safely modify a .py file without fearing that scripts that Fedora ships will cause that to break their system in new and mysterious ways. Yeh, recompilation as a risk seems fine ... nothing working because of the recompilation not so much. But it's kind of a weird edge case.
Well, it isn't that "nothing will work". The only valid use case of docstrings seems to be the IDE usage. With all the hate of IDEs in linux community (I am not one of them), I'm really surprised that someone cares about it. Moreover in such uncommon (unsupported?) cases.
Anyway, yes, I don't care whether it is mentioned in the guidelines or not. So, despite the fact that I do not agree with you, it's enough to me that FPC thinks that the "-OO" usage is a problem. The decision about the guidelines is up to you.
We discussed this at this weeks meeting (http://meetbot.fedoraproject.org/fedora-meeting-1/2015-06-18/fpc.2015-06-18-16.00.txt):
We discussed this at this weeks meeting (http://meetbot.fedoraproject.org/fedora-meeting-1/2015-06-25/fpc.2015-06-25-16.01.txt):
Announcement text (will be in a separate section regarding python guidelines):
A small section was added to indicate that -OO must not be used for versions of python less than 3.5:
Metadata Update from @toshio: - Issue assigned to tibbs
Login to comment on this ticket.