#1384 F23 System Wide Change: Harden All Packages - https://fedoraproject.org/wiki/Changes/Harden_All_Packages
Closed None Opened 9 years ago by jreznik.

For the 2015-01-14 meeting as the Change Proposal was announced on devel-announce list on 2015-01-07.

Harden all packages with position-independent code to limit the damage from certain security vulnerabilities.


I believe everyone got carried away in the last meeting with the GCC5 change. GCC5 probably has PIE performance improvements. But this change works even with the current GCC.

Change '''line 130''' here https://pkgs.fedoraproject.org/cgit/redhat-rpm-config.git/tree/macros from

{{{

_hardened_build 0

}}}

to

{{{
%_hardened_build 1
}}}

40 hours = 1.66 days

Contingency mechanism: Rebuild '''only''' packages that do not work because of this without PIC.

More Details:

  1. For packages that fail to build, Rel-Eng will run a '''test''' by adding ''%global _hardened_build 0'' to the top of the spec file.

  2. If the package fails to build even with ''%global _hardened_build 0'' then it is '''not''' a PIC bug.

  3. If the build completes successfully with ''%global _hardened_build 0'' then it '''is''' a PIC bug. In this case, Rel-Eng will commit this change ('%global _hardened_build 0') to the spec file, and simultaneously file a bug against that package.

Can all of the above not be scripted? You will need to add more time for compilation from the regular 1.66 days to maybe 3 days because, this time, it will be compiling so that the end result is PIE code.

It looks to me that all of this can be done ''well before the Feb 6th Side Tag Builds Deadline.''

Maybe I have missed the big picture or something?


I am using Centos7 right now and have rebuilt many of packages with PIE enabled. All my rebuilds* work fine. They also pass the 'sudo checksec --proc-all' test.

*The exception being VBox, where R0 (ring0) code won't compile with -pie.

Replying to [comment:1 moezroy]:

Maybe I have missed the big picture or something?

Yes, the constraint is not the compute time it takes to actually do a mass rebuild. The constraint is the fixes that would be needed accross the package collection by maintainers in the wake of a mass rebuild.

Even if enabling PIE caused 0 build failures, there would almost surely be other build failures for other reasons since our last mass rebuild that would have to be dealt with.

Change is not approved.

AGREED: System Wide Change: Harden all packages with position-independent code NOT accepted for F22 (0,-7,0)

F23 decision defered until 2015-01-21 meeting.

Replying to [comment:2 kevin]:

Replying to [comment:1 moezroy]:

Maybe I have missed the big picture or something?

Yes, the constraint is not the compute time it takes to actually do a mass rebuild. The constraint is the fixes that would be needed accross the package collection by maintainers in the wake of a mass rebuild.

Even if enabling PIE caused 0 build failures, there would almost surely be other build failures for other reasons since our last mass rebuild that would have to be dealt with.

Okay. Thanks for the explanation above. :)

As I won't be able to attend the today's FESCo meeting I am +1 to build everything with PIE on 64 bit architectures, -1 on 32 bit architectures for F23. Exceptions if major performance regression is found in some applications (f. e. some mathematics computation apps) should be allowed but they should be handled in the individual spec files.

I forgot the % sign in the previous post. Edited it to read:

{{{
%_hardened_build 1
}}}

This ticket will be discussed in the FESCo meeting on Wednesday at 18:00UTC in #fedora-meeting on irc.freenode.net.

When this is discussed, can you please also decide whether it is ok to enable PIE for all executables that run after login after a fresh default installation of any product in Fedora 22?

Also in case it is not acceptable to harden all packages, please consider to at least make {{{%_hardened_build 1}}} the default and allow to disable it if there are valid reasons. But then these reasons need to be stated in the SPEC next to the line that changes the setting (this is for Fedora 23).

Now that we have branched I am in favor of changing the default to hardened_build 1 in rawhide, and asking change owners to retarget this change for f23.

From today’s FESCo meeting:
F23 System Wide Change: Harden all packages with position-independent code approved (+5)

FYI: I opened a bug to get this changed in redhat-rpm-config:
https://bugzilla.redhat.com/show_bug.cgi?id=1192183

I want to reopen the discussion on this. -z now is actually a pretty brutal change in terms of exec performance. For a package like xorg-x11-proto-devel (where the compilation time is almost entirely spent running ./configure) mockbuild time went from about 2:30 to 3:30 between F22 and F23. That's pretty steep.

In addition there's a pretty common plugin pattern where not all of the services a plugin might need are necessarily available at dlopen() time. For example, X video drivers are expected to explicitly load the rendering backend they need (libfb, typically), but then call those functions by name. Since fb won't be loaded yet, loading the driver will fail.

I propose limiting the '-z now' default behaviour to executables only, and not DSOs. Since most executables export relatively few symbols on their own, the overhead from eagerly resolving symbols should be much more limited. now-ness is per-object so setting it on the executable doesn't force now-ness on the libraries it loads, which typically export many more symbols (and there's a lot more libs than executables).

I think the changes look like this:

https://ajax.fedorapeople.org/patches/0001-Hardening-changes-default-to-_hardened_build-0-link-.patch

Replying to [comment:13 ajax]:

I think the changes look like this:

https://ajax.fedorapeople.org/patches/0001-Hardening-changes-default-to-_hardened_build-0-link-.patch

If that patch gets applied, then working around
https://bugzilla.redhat.com/show_bug.cgi?id=1197501 by undefining _hardened_build will no longer work, and I'll have to hack the macros much more invasively to be able to build any Ada programs.

By the way, changing _hardened_build from 1 to 0 has no effect, because "%{?_hardened_build:...}" tests only whether the macro exists and ignores its value.

Countless hours has been spent on debugging the -z now change already, and in one fell swoop the attached patch will destroy all that work, without even letting it see the light of day in F23 :( :(

Replying to [comment:16 moezroy]:

Countless hours has been spent on debugging the -z now change already, and in one fell swoop the attached patch will destroy all that work

So what do you propose instead, and why is that a better course of action? (This does include “just keep the current setup” as an option, but let’s actually discuss the advantages and disadvantages.)

To be blunt, the countless spent hours are a sunk cost; to get the best possible decision we not only can, but ''must'' ignore them.

E.g. the following would be useful to know, rather than that time has been spent:

  • Is the -z now change “debugged” now?
  • If it is debugged, or after it will have been debugged, how will X11 drivers and Guile modules be packaged and linked?
  • Is there anything specific we should wait for before discussing Adam’s patch? How much does it decrease the effectiveness of the mitigations? Does it have other disadvantages?
  • Is there something else I have forgotten to think about?

Replying to [comment:17 mitr]:

  • Is the -z now change “debugged” now?
  • If it is debugged, or after it will have been debugged, how will X11 drivers and Guile modules be packaged and linked?

-z now will be disabled in the packages having problems.

The following needs to be added under %build section in the spec file to disable -z now:

{{{
CFLAGS="$RPM_OPT_FLAGS -Wl,-z,lazy"
CXXFLAGS="$RPM_OPT_FLAGS -Wl,-z,lazy"

export CFLAGS
export CXXFLAGS
}}}

That was the step taken by the gnutls maintainer. See: https://bugzilla.redhat.com/show_bug.cgi?id=1196556#c11 He suggested a special macro for applications that rely on modules with undefined symbols.

Replying to [comment:14 rombobeorn]:

Replying to [comment:13 ajax]:

I think the changes look like this:

https://ajax.fedorapeople.org/patches/0001-Hardening-changes-default-to-_hardened_build-0-link-.patch

If that patch gets applied, then working around
https://bugzilla.redhat.com/show_bug.cgi?id=1197501 by undefining _hardened_build will no longer work, and I'll have to hack the macros much more invasively to be able to build any Ada programs.

I've commented on the bug, the workaround doesn't look too bad to me tbh.

By the way, changing _hardened_build from 1 to 0 has no effect, because "%{?_hardened_build:...}" tests only whether the macro exists and ignores its value.

Heh, indeed. I've uploaded an updated patch, which also fixes incremental linking.

Replying to [comment:16 moezroy]:

Countless hours has been spent on debugging the -z now change already, and in one fell swoop the attached patch will destroy all that work, without even letting it see the light of day in F23 :( :(

Let's not engage in hyperbole. Countable hours have been spent, and the patch would undo that work only for shared libraries. We'd still be changing executables to full-relro PIE. That's still a material improvement in ASLR.

Am I really the only person who's done even a vague performance comparison between the two regimes? Slowing down builds (and by implication other exec-bound workloads) by 40% seems like a really steep cost. And diverging Fedora that much from what binutils defaults to presents a latent unpleasant surprise for developers coming to Fedora from the rest of the world. If we're going to start breaking binary compatibility between Fedora and Linux in general, this isn't where I'd start.

Replying to [comment:20 ajax]:

Let's not engage in hyperbole. Countable hours have been spent, and the patch would undo that work only for shared libraries. We'd still be changing executables to full-relro PIE. That's still a material improvement in ASLR.

Sure, some ASLR is better than no ASLR; and ASLR is way more important than relro.

OTOH, looking at relro only, shared libraries are about 25% of ELF objects on my system, i.e. about 25% ELF objects would not be protected against .data overflows. Because only a subset of executables allow escalation of privilege, this could well be diminishing the relro effectiveness by half. So keeping it on by default, and asking packages that rely on (non-weak!) undefined symbols to work to add a linker option seems preferable: it results in a more secure system, and puts the burden on the more questionable (though probably legal) shared object / symbol usage code.

Slowing down builds (and by implication other exec-bound workloads) by 40% seems like a really steep cost.

At some point we should just simplify / fix / speed up autoconf, or perhaps just set the expectation that it needs to be fixed, instead of optimizing the system for this, really rather ridiculous, kind of usage. I am not sure that ''this'' is the point where moving the burden towards autoconf should happen, but I also don’t see any reason why not.

And diverging Fedora that much from what binutils defaults to presents a latent unpleasant surprise for developers coming to Fedora from the rest of the world. If we're going to start breaking binary compatibility between Fedora and Linux in general, this isn't where I'd start.

Are you calling a slowdown a binary compatibility breakage, or is there some other binary compatibility breakage? (AFAIK the redhat-rpm-config changes only apply to Fedora RPMs, not to anything custom-built.)

Replying to [comment:19 ajax]:

https://ajax.fedorapeople.org/patches/0001-Hardening-changes-default-to-_hardened_build-0-link-.patch
Heh, indeed. I've uploaded an updated patch, which also fixes incremental linking.

The ld man page says this regarding -r :

{{{
-r
--relocatable
Generate relocatable output---i.e., generate an output file that can in turn serve as input to ld. This is
often called partial linking. As a side effect, in environments that support standard Unix magic numbers, this
option also sets the output file's magic number to "OMAGIC". If this option is not specified, an absolute file
is produced. When linking C++ programs, this option will not resolve references to constructors; to do that,
use -Ur.

       When an input file does not have the same format as the output file, partial linking is only supported if that
       input file does not contain any relocations.  Different output formats can have further restrictions; for
       example some "a.out"-based formats do not support partial linking with input files in other formats at all.

       This option does the same thing as -i.

}}}

In the updated patch you are disabling -pie and -z now if there is -r? Why?

{{{
+self_spec:
++ %{!r:%{!shared:-pie}}
+
+
link:
++ %{!r:%{!shared:-z now}}
}}}

Try to remember how slow your computer was in 2006. At around this time (or most likely before) the Gentoo Hardened project started applying -z now to everything, where it was possible. See: http://binutils.sourceware.narkive.com/8GzJSgtx/adding-z-nonow-to-complement-z-now

Now come back to the present (2015 – nine years later) and compare how much your computer's performance has changed since 2006.

Right now if something is not shared, it will get the -pie flag.

So the shared library's will escape the -pie, but atleast there is -z now to harden them.

The alternative proposal will allow the shared libraries to escape from being hardened.

If you take a look at this: https://books.google.ca/books?id=NivmOf2J7qQC&pg=PA67&lpg=PA67&dq=relro+now&source=bl&ots=CSaoWB9DlM&sig=2w2LzxgZHHWY0kKB52e2Ppjkmec&hl=en&sa=X&ei=Ci0JVbmEIIuSyASgx4GYBA&redir_esc=y#v=onepage&q=relro%20now&f=false it shows how due to a vulnerability in the FFmpeg library (shared library) the attacker was able to overwrite the global addressing table and execute arbitrary code. Here the attack was carried using the VLC player.

Using the above scenario as an example, you will be hardening the VLC player executable, but you won't be making the FFmpeg library's global addressing table read only.

Based on this https://code.google.com/p/android-source-browsing/source/detail?r=233d460f21d372f964f8078b8b0d5fd17af7c6b6&repo=platform--build and it shows that Android enabled Full RELRO in 2012. This https://www.duosecurity.com/blog/exploit-mitigations-in-android-jelly-bean-4-1 talks about how they were able to root the phone because Full RELRO was not enabled.

This bug https://bugzilla.redhat.com/show_bug.cgi?id=1197692 shows a good example of where the hardening flags helped: “I will remove support for the non-existing symbol. I believe the bug was there for many years but never noticed until F23 enabled hardened build flags with non-lazy DSO relocations.”

Replying to [comment:19 ajax]:

Replying to [comment:14 rombobeorn]:

If that patch gets applied, then working around
https://bugzilla.redhat.com/show_bug.cgi?id=1197501 by undefining _hardened_build will no longer work, and I'll have to hack the macros much more invasively to be able to build any Ada programs.

I've commented on the bug, the workaround doesn't look too bad to me tbh.

By "the workaround", do you mean "jamming -fPIC on the end of %Ada_optflags"?

I assume that there were good reasons for writing redhat-hardened-cc1 with all those conditionals instead of just sticking -fPIC in optflags. The reasons appear to be that -fPIE is desired for programs but -fPIC is needed for libraries, and that there shouldn't be more than one of -fpic, -fPIC, -fpie and -fPIE in the same command line. I don't see why those reasons would be any less relevant for Ada than for C.

Can we please keep the ability to disable the hardening when it doesn't work? Perhaps we can have three levels:

  • When _hardened_build is undefined: Same meaning as today: no -fPIE, -pie or -z now.
  • When _hardened_build is defined to "partial": Use -fPIE and -pie like today, and apply -z now to programs but not to libraries like Adam wants.
  • When _hardened_build is defined to "full", or to any other value for backwards-compatibility: Use -fPIE, -pie and -z now like today.

Then working around problems by undefining _hardened_build will still work, and you guys can debate whether the default should be "partial" or "full".

Replying to [comment:18 moezroy]:

{{{
CFLAGS="$RPM_OPT_FLAGS -Wl,-z,lazy"
CXXFLAGS="$RPM_OPT_FLAGS -Wl,-z,lazy"

export CFLAGS
export CXXFLAGS
}}}

-Wl,-z,lazy is a linker flag and has therefore be part of LDFLAGS. In general, the above will not work.

Replying to [comment:14 rombobeorn]:

If that patch gets applied, then working around
https://bugzilla.redhat.com/show_bug.cgi?id=1197501 by undefining _hardened_build will no longer work, and I'll have to hack the macros much more invasively to be able to build any Ada programs.

I've asked here why the Ada specs file lacks support for flags injection:

https://gcc.gnu.org/ml/gcc/2015-04/msg00000.html

If there is no reason to have this Ada-specific discrepancy, I'll propose a patch upstream to add %(cc1_options), and we can backport it in Fedora.

Replying to [comment:22 moezroy]:

In the updated patch you are disabling -pie and -z now if there is -r? Why?

Because not to do so is wrong. -r is for when the output file is another ET_REL, not an executable or a shared library. If we ignored the presence of -r, we would ask for -r -pie, which is like asking for a black pink unicorn.

Replying to [comment:24 rombobeorn]:

By "the workaround", do you mean "jamming -fPIC on the end of %Ada_optflags"?

I assume that there were good reasons for writing redhat-hardened-cc1 with all those conditionals instead of just sticking -fPIC in optflags. The reasons appear to be that -fPIE is desired for programs but -fPIC is needed for libraries, and that there shouldn't be more than one of -fpic, -fPIC, -fpie and -fPIE in the same command line. I don't see why those reasons would be any less relevant for Ada than for C.

If there's a way to examine the gnat compile command and discover whether the output is destined for a shared object or an executable, then we could make the same distinction for Ada.

Can we please keep the ability to disable the hardening when it doesn't work?

Of course! I wasn't suggesting otherwise.

Agreed on today’s FESCo meeting:

Drop meeting keyword, continue tracking work in the ticket (+6) (mitr, 18:15:49)

TLDR, tbh. But was the bug https://bugzilla.redhat.com/show_bug.cgi?id=952946
considered before we turned on pie globally?

Seems like there are basically no guaranties about stack size, It could hurt
32-bit architectures a bit..

FWIW, ad comment 1, '%define _hardened_build 0' does not work:
https://bugzilla.redhat.com/show_bug.cgi?id=1211296

I see this ticket is open for tracking the work for this Change. We have already passed Change Checkpoint: Completion deadline (testable). Is there something to share here?

I'm not sure there is anything left to track here, so I am closing this ticket.

If anyone feels there is still something here for fesco to look at/do, please re-open with your rationale. Thanks.

The global enforcement of the '-z now' linker flag silently broke loading of zsh modules:

https://bugzilla.redhat.com/1277996

http://www.zsh.org/mla/workers/2015/msg02981.html

Just so you know...

Login to comment on this ticket.

Metadata