The build flags (CFLAGS, CXXFLAGS and LDFLAGS) saved in the Python's distutils module for building extension modules are switched from:
%{build_cflags}, %{build_cxxflags} and %{build_ldflags}
to:
%{extension_cflags}, %{extension_cxxflags} and %{extension_ldflags}.
This currently means that no GCC plugins (such as annobin) are activated and no GCC spec files (-specs= arguments) are used by default when building Python extension modules (e.g. with python3 setup.py build).
The Package-x-generic-16.pngpython3-devel package will lose its runtime dependency on Package-x-generic-16.pngredhat-rpm-config (which was only required for annobin support and GCC spec files).
The change will affect building extension modules by users, outside of the RPM environment. The Python standard library and Fedora's Python 3 RPM packages are still built with the "traditional" set of flags (%{build_cflags} and friends), unless the package uses nonstandard methods to build the extensions.
Only Python 3.7 and 3.6 will be changed.
+1
The change proposal says that this will make python modules built on Fedora not being security hardened and this would be beneficial since upstream does not do this (if I understand it correctly) . How is weakening the security of custom python modules beneficial for the user? One could argue that these are the modules that these are probably less audited that therefore less secure by themselves.
Yeah, I understand not wanting to get the set of flags that came from the build of python itself, but I'd much prefer to see that we at least honor the standard distro CFLAGS. I'm -1 on this as it stands.
This is highly problematic as those flags are quite Fedora specific and not a common set among different linux distros. This change will affect custom built binaries and libraries, aka github fetched or user created code. This increases flexibility for developing on Fedora. Also I'm not sure of the validity of the security argument. Does Fedora, as distro, want to handle the security of random user created C code, built with distutils?
The rpm builds will not change (apart from a cosmetic change at the flag set as outlined at the proposal).
And as one of the current maintainers of CPython in Fedora and co-owner of the change, I consider the current behavior a serious bug which should be addressed.
This is highly problematic as those flags are quite Fedora specific and not a common set among different linux distros. This change will affect custom built binaries and libraries, aka github fetched or user created code. This increases flexibility for developing on Fedora.
I'm guessing this change was motivated by some specific problems? Can you provide examples? In general, the flags we're discussing are used across a huge codebase - I would expect them to be compatible with most code.
Also I'm not sure of the validity of the security argument. Does Fedora, as distro, want to handle the security of random user created C code, built with distutils?
Yes, I think we'd like to improve the security of random user created C code built with distutils - that sounds like a benefit to Fedora users. I don't see that this somehow makes us responsible for security problems that aren't fixed by the compilation options.
This is highly problematic as those flags are quite Fedora specific and not a common set among different linux distros. This change will affect custom built binaries and libraries, aka github fetched or user created code. This increases flexibility for developing on Fedora. I'm guessing this change was motivated by some specific problems? Can you provide examples? In general, the flags we're discussing are used across a huge codebase - I would expect them to be compatible with most code.
One good example is that it's not guaranteed that those flags will be the same across different toolchains. Users can compile the extensions with the compiler and flags of their choice but those flags will still be hardcoded into distutils. e.g. we compile cpython with link time optimizations, but it can be buggy in some cases when using clang.
Also I'm not sure of the validity of the security argument. Does Fedora, as distro, want to handle the security of random user created C code, built with distutils? Yes, I think we'd like to improve the security of random user created C code built with distutils - that sounds like a benefit to Fedora users. I don't see that this somehow makes us responsible for security problems that aren't fixed by the compilation options.
There is the rationale on the change, and also there is some extensive discussion here: https://bugzilla.redhat.com/show_bug.cgi?id=1217376
Just some example what type of issues this fixes:
https://github.com/mesonbuild/meson/issues/4709
I think Ruby is struggling of the same problem… @vondruch ?
(tl;dr in bold)
How is weakening the security of custom python modules beneficial for the user?
I kinda expected somebody to ask this, thanks @till. I realize that anything that even potentially lowers security is a red flag. And I admit that this is indeed the case.
Before the change: Users building custom Python extension got theoretically more secure builds but they also got:
Any compiler/linker flag that is bringing security and is generally compatible with anything should stay in %{extension_...flags}. However the problematic noncompatible flags should not be there. I trust @fweimer to make the right choices about what flags go where. We've just decided to use this.
%{extension_...flags}
This makes Fedora better for developers who use it as their development platform of choice. I suspect that it can certainly make it a bit less secure for people who use Fedora as their deployment platform and who don't know anything about hardening flags and hence are not motivated to explicitly set them. In my eyes the first beats the second.
As a counter example: When users compile their C code with gcc, we don't force any flags upon them. Does that make Fedora less secure? Why do we do it for Python extensions and not for everything?
Yeah, I understand not wanting to get the set of flags that came from the build of python itself, but I'd much prefer to see that we at least honor the standard distro CFLAGS.
What this change does is that it honors the standard distro CFLAGS except the flags that were selected as not applicable to extension builders by the folks that maintain the set of CFLAGS. That works for me. If there are specific flags that we are afraid of being removed, we can work with @fweimer and move them back. However those would need to:
Yes, Ruby is struggling with similar issues but I am not in favor of this change proposal and I don't plan to work on a similar proposal for Ruby. Ruby, neither Python or anything else should embed its compilation flags for future use. This mindset comes from the times there was no other option than installing software by "./configure && make && make install" and it does not take in account binary distributions, where the SW is compiled on a different system then it is used.
There should be some generic mechanism, which would set proper (based on some generic distribution settings, such as hardening) and optimized flags (allowing to compile code specifically for my machine) for my environment, once the compiler is installed and they should be used from now on.
There should be some generic mechanism...
Yet there is not. In the meantime, this is the best we can do.
Ruby, neither Python or anything else should embed its compilation flags for future use.
The problem is that currently embedding the flags is the only thing that makes it work. What we are trying to do here is only embed the necessary flags to make it work (or at least get rid of those that clearly don't work).
OK, I'm not sure I adequately stated my concern above, so I'm going to try to rephrase it.
When compiling a python extension in the Koji build-system, it must be compiled with all of the CFLAGS (and LDFLAGS, etc.) that any standard C compilation would get in that situation.
I don't really care which flags the extensions get when they're compiled outside of our build-system, as that's the developer's responsibility.
If this is already the case in the current proposal, I'm good with that. However, it's not clear to me right now what happens with packaged extensions.
Packaged extensions will have all same flags as before. It's just flags which are passed when you don't specify CFLAGS in env.
OK, I'm not sure I adequately stated my concern above, so I'm going to try to rephrase it. When compiling a python extension in the Koji build-system, it must be compiled with all of the CFLAGS (and LDFLAGS, etc.) that any standard C compilation would get in that situation. I don't really care which flags the extensions get when they're compiled outside of our build-system, as that's the developer's responsibility. If this is already the case in the current proposal, I'm good with that. However, it's not clear to me right now what happens with packaged extensions.
I'll try to clear this up a bit, but your understanding is correct. Basically packaged C extensions get two sets of flags. One that is coming from distutils, aka the ones cpython was built with (or better say the ones cpython says that C extensions should be built with). The other is coming from the standard rpm build environment.
Those flags get concatenated, so in the build logs you will see a huge set of flags, coming from both sets, with many of those overlapping.
With this change, the hardening flags will get removed when the C extensions inherits the distutils flags, but they will still be applied due to the environment.
You can see the comparison here: https://fedoraproject.org/wiki/Changes/Python_Extension_Flags#Impact_on_RPM_packages_building_extension_modules
So essentially the effect of this change to rpm built C extensions is mostly a cosmetic one.
The change states:
The Python standard library and Fedora's Python 3 RPM packages are still built with the "traditional" set of flags (%{build_cflags} and friends), unless the package uses nonstandard methods to build the extensions.
See https://fedoraproject.org/wiki/Changes/Python_Extension_Flags#Impact_on_RPM_packages_building_extension_modules for detialed breakdown of how this really works which I have documented for exactly this purpose.
If you worry about the nonstandard methods to build the extensions imagine somebody calling just gcc foo.c from the spec - it would not apply any flags either.
gcc foo.c
Thank you for answering those questions. I'm +1 with this information.
+1, and agreed that it is the user's responsibility to select their own build flags when building outside of Fedora's build system.
So most security flags are still present and AFAIU the missing hardened specs only ensure that -pie or -fPIEare set when appropriate but they are removed because --specs does not work wihth clang. Is there are way to include this flags without breaking clang?
-pie
-fPIE
We do not trust all the upstreams that we package to select the build flags responsibly. If we can help them to use better defaults, then it is good. If we did this only accidentally, then we should also properly (explicitly) communicate this in the release notes, i.e. highlight that the binaries of user-compiled extensions will not be position independent anymore by default and explain how users could get the behavior again.
If we did this only accidentally, then we should also properly (explicitly) communicate this in the release notes, i.e. highlight that the binaries of user-compiled extensions will not be position independent anymore by default and explain how users could get the behavior again.
Agreed. Adding this to the release notes section fo the change proposal.
https://fedoraproject.org/wiki/Changes/Python_Extension_Flags#Release_Notes
Metadata Update from @churchyard: - Issue tagged with: meeting
Metadata Update from @churchyard: - Issue untagged with: meeting - Issue tagged with: pending announcement
This just got approved with +7,0,0
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/LWYHDYIZSMWZUSC5F6UAAN4OX44ZHYYX/
Metadata Update from @churchyard: - Issue untagged with: pending announcement - Issue close_status updated to: Accepted - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.