#1104 Expanding the list of "Hardened Packages"

Created 4 years ago by halfie
Modified 4 years ago

= phenomenon =

[http://fedoraproject.org/wiki/Hardened_Packages] page mentions that
"FESCo requires some packages to use PIE and relro hardening by default."

It would be great if this list could be expanded to include packages which are at comparatively more risk of being exploited (locally or remotely).

Such packages will typically include various system daemons, network daemons and network enabled applications.

(Implementing this will require changes to "Packaging Guidelines")

= background analysis =

Lot of network daemons are already using PIE and RELRO (e.g. httpd, MariaDB). So a natural question is why aren't packages in same "network daemons" class like PostgreSQL, Dovecot and MongoDB aren't being hardened?

= implementation recommendation =

I believe that hardening flags should be turned on (by default) for all packages which are at the risk of being exploited.

"Packaging Guidelines" say that "Other packages may enable the flags at the maintainer's discretion."

Thinking from a security perspective, I find "Hardening flags can be disabled for other packages at the maintainer's discretion provided enough justification is given to FESCo" to be more appropriate.

For a start, packages from the following RPM groups can be targeted,

{{{
"Applications/CGI", "Network/Daemons", "Applications/Communications",
"Applications/Internet", "System Environment/Daemons", "Applications/Databases"
}}}

http://fedoraproject.org/wiki/Packaging:Guidelines#PIE also says:

{{{
If your package meets any of the following criteria you MUST enable the PIE compiler flags:

Your package is long running. This means it's likely to be started and keep running until the machine is rebooted, not start on demand and quit on idle.

Your package has suid binaries, or binaries with capabilities.

Your package runs as root.

}}}

which is a superset of the criteria suggested in "phenomenon" above.

I read the guideline as "if a package matches the above criteria OR is on http://fedoraproject.org/wiki/Hardened_Packages , it MUST use PIE - but it could definitely be clarified. Anyway, this is probably up to FPC.

Note that RPM groups are almost entirely obsolete - they don't make a good guide here.

I'm afraid "all packages which are at the risk of being exploited" is pretty broad and could include most everything.

I'd rather suggest adding specific packages to the must list, or figuring out a better critera that maintainers could use to decide.

I'm afraid "all packages which are at the risk of being exploited" is pretty broad and could include most everything.

[[BR]]

You are right. I think packages which meet the criteria described in "comment:1" by mitr are suitable candidates for hardening.

However lot of packages (like PostgreSQL, Dovecot and MongoDB) which meet the criteria are still not being hardened. I am not sure if I should file bugs against individual packages or if FESCo can expand its list to include mentioned packages as well as similar packages (the latter option being more preferable).

I can work towards identifying more packages which could use hardening.

[[BR]]

Note that RPM groups are almost entirely obsolete - they don't make a good guide here.
[[BR]]

Yes, people in #fedora-devel told me the same. I am still looking for better ways to identify suitable packages. Suggestions are welcome!

Replying to [comment:4 halfie]:

However lot of packages (like PostgreSQL, Dovecot and MongoDB) which meet the criteria are still not being hardened. I am not sure if I should file bugs against individual packages or if FESCo can expand its list to include mentioned packages as well as similar packages (the latter option being more preferable).
If you think that some package meets the current criteria and it is not hardened yet, I think that filling a bug on the package cannot do any harm. So feel free to identify such packages and fill the bugs.

[https://wiki.ubuntu.com/Security/Features] says "PIE on x86_64 does not have the same penalties, and will eventually be made the default, but more testing is required."

What about Fedora taking the lead on this one?

I guess PIE has some impact on performance. Therefore I'd rather use PIE on limited list of packages. Databases might be a good addition into the current group.

If you have a list what you'd like to see there, then please add it here.

PIE disables use of prelink - so this is another performance impact on startup. On the other hand we should evaluate the impact of non-prelinked vs. prelinked startup time on modern computers, maybe it is no longer much relevant.

Replying to [comment:8 tmraz]:

PIE disables use of prelink - so this is another performance impact on startup. On the other hand we should evaluate the impact of non-prelinked vs. prelinked startup time on modern computers, maybe it is no longer much relevant.

[[BR]]

Please see http://people.redhat.com/~gmurphy/files/pie.odt for such an evaluation.

In short. "... the average delay of a PIE application over a non-PIE application was significantly below perceivable threshold."

I'm not sure that in the study above the non-PIE binaries were prelinked or not. Also it would be more interesting to see the result for bigger desktop applications like LibreOffice, Firefox, Evolution.

Replying to [comment:7 mmaslano]:

I guess PIE has some impact on performance. Therefore I'd rather use PIE on limited list of packages. Databases might be a good addition into the current group.

[ftp://ftp.inf.ethz.ch/doc/tech-reports/7xx/766.pdf] mentions an average overhead of 3.6% on AMD64 (x64) which is not too bad.

However, the overhead is too high on x86 (32-bit) in my opinion. Is it possible to use different compilation flags on different platforms?

If you have a list what you'd like to see there, then please add it here.

I am working on it and it should be somewhat ready by early next week.

Would it make sense to move the dicussion to fedora-devel to get a wider audience and more comments, and then return to the FESCo trac (and is it FESCo or FPC?) to discuss a specific policy proposal?

Replying to [comment:10 tmraz]:

I'm not sure that in the study above the non-PIE binaries were prelinked or not. Also it would be more interesting to see the result for bigger desktop applications like LibreOffice, Firefox, Evolution.

[http://en.wikipedia.org/wiki/Prelink] mentions "Jakub JelĂ­nek points out that position independent executables ignore prelinking on Red Hat Enterprise Linux and Fedora Core, and recommends that network and SUID programs be built PIE to facilitate a more secure environment."

I agree with your suggestion about measuring the impact of PIE on bigger Desktop applications. I will work on it (after the list is done).

Replying to [comment:12 mitr]:

Would it make sense to move the dicussion to fedora-devel to get a wider audience and more comments, and then return to the FESCo trac (and is it FESCo or FPC?) to discuss a specific policy proposal?

Agreed. I have posted the proposal to fedora-devel, [http://lists.fedoraproject.org/pipermail/devel/2013-March/180827.html]

Here is a list of packages (daemons only) which are (possibly) violating
packaging guidelines with respect to hardening.

http://dl.dropbox.com/u/1522424/probable-violations-F19.xls

http://dl.dropbox.com/u/1522424/probable-violations-F19.csv

(Please note that this list was generated by a program and might be buggy!)

To summarize,

  1. Many packages which meet the hardening criteria (described at [http://fedoraproject.org/wiki/Packaging:Guidelines#PIE]) are not being hardened.

For example, postgresql which is "long running" is not being built with PIE flags. This is a clear violation of the packaging guidelines.

  1. After many discussions, I believe that the existing "hardening criteria" is good enough but '''I would like to see the hardening policy actually being enforced.'''

I have posted tools and data to help in doing this (in an almost automated fashion).

  1. For postgresql, I have opened a bug [https://bugzilla.redhat.com/show_bug.cgi?id=947022] requesting hardening to be enabled but so far I haven't got any response.

In my opinion, any bugs reporting violation of packaging guidelines should be looked at in a prompt fashion.

  1. Opening bugs against tons of packages (possibly) manually is a bit tedious. Ideas on how to solve this bit of the problem are welcome!

  2. "hard data that demonstrate the impact, if any, in a situation relevant to Fedora (in particular, taking into account prelink as it is deployed by default) would be very welcome but is not a strict requirement".

Please see comment:10. In short, "... the average delay of a PIE application over a non-PIE application was significantly below perceivable threshold". Please note that "position independent executables ignore prelinking on Red Hat Enterprise Linux and Fedora Core", so the numbers presented in the report linked to in comment:10 should be good as they are).

Additionally, I have posted benchmarking numbers for Firefox ("big Desktop application") at [https://lists.fedoraproject.org/pipermail/devel/2013-April/181170.html]. In short, post-hardening numbers are same as before!

  1. Currently, I am working on getting more benchmarking numbers out (in order to address Jakub's concerns, see [https://lists.fedoraproject.org/pipermail/devel/2013-April/181171.html]). This is being done to study the possibility of bringing even more packages under hardening criteria in future.

Remember when Sun said "the network is the computer"? All packages handle untrusted user input and network input.

I'm strongly in favour of dropping prelink, and enabling hardening per default. Especially if the only price we pay is "a few seconds of waiting on initial loading of large applications". libreoffice is so slow to start, adding three, five or ten seconds isn't going to make a difference.

Short of that, I'd say any every daemon or service started through systemd/(x)inetd/ should enable hardening.

Replying to [comment:17 pwouters]:

I'm strongly in favour of dropping prelink, and enabling hardening per default. Especially if the only price we pay is "a few seconds of waiting on initial loading of large applications". libreoffice is so slow to start, adding three, five or ten seconds isn't going to make a difference.

Well, the difference in loading time (for "large Desktop applications") does not even amount to a second (according to my tests which may be flawed).

Please see [https://gist.github.com/kholia/807f463235ab6eacac96] for Gimp benchmarks.

Thanks for the summary.

Replying to [comment:16 halfie]:

  1. Many packages which meet the hardening criteria (described at [http://fedoraproject.org/wiki/Packaging:Guidelines#PIE]) are not being hardened.

I think that's a subset of point 4 and possibly point 2.

  1. After many discussions, I believe that the existing "hardening criteria" is good enough but '''I would like to see the hardening policy actually being enforced.'''

I have posted tools and data to help in doing this (in an almost automated fashion).

That would be great; could this be integrated into fedora-review (and maybe rpmlint)?

  1. For postgresql, I have opened a bug [https://bugzilla.redhat.com/show_bug.cgi?id=947022] requesting hardening to be enabled but so far I haven't got any response.

In my opinion, any bugs reporting violation of packaging guidelines should be looked at in a prompt fashion.

I'm sorry to say I'm much more worried about unhandled crash reports (which may indicate possibly exploitable bugs) than missing hardening; that's not to say the bug shouldn't be handled - just that I don't think some kind of forceful prioritization of these bugs makes sense.

  1. Opening bugs against tons of packages (possibly) manually is a bit tedious. Ideas on how to solve this bit of the problem are welcome!

There certainly are automated tools; is python-bugzilla the best one nowadays?

  1. "hard data that demonstrate the impact, if any, in a situation relevant to Fedora (in particular, taking into account prelink as it is deployed by default) would be very welcome but is not a strict requirement".

Please see comment:10. In short, "... the average delay of a PIE application over a non-PIE application was significantly below perceivable threshold".

That measures startup, not long-term performance impact. (PIE affects both, prelink optimizes primarily startup.)

Please note that "position independent executables ignore prelinking on Red Hat Enterprise Linux and Fedora Core", so the numbers presented in the report linked to in comment:10 should be good as they are).
So the PIE numbers are relevant, but the non-PIE ones are not?

Additionally, I have posted benchmarking numbers for Firefox ("big Desktop application") at [https://lists.fedoraproject.org/pipermail/devel/2013-April/181170.html]. In short, post-hardening numbers are same as before!

It might be a 5% performance difference, which is quite big, but it in the unexpected direction... Surprising.

  1. Currently, I am working on getting more benchmarking numbers out (in order to address Jakub's concerns, see [https://lists.fedoraproject.org/pipermail/devel/2013-April/181171.html]). This is being done to study the possibility of bringing even more packages under hardening criteria in future.

(FWIW, I'm looking for hard data as much preferable to opinions of the kind "I always disable $foo because it is slow". I don't currently have a specific threshold in mind that would sway me pro/con any decision.)

So in summary, your proposal is to keep the existing guidelines and enforce them better? I'm in any case fine with that.

Replying to [comment:17 pwouters]:

Remember when Sun said "the network is the computer"? All packages handle untrusted user input and network input.

I'm strongly in favour of dropping prelink, and enabling hardening per default. Especially if the only price we pay is "a few seconds of waiting on initial loading of large applications".

That's the price for prelink (and btw. preferable overall latency is 0.1s), not for PIE, which is an ongoing cost (... for code located in the executable and not in libraries).

libreoffice is so slow to start, adding three, five or ten seconds isn't going to make a difference.

Actually libreoffice seems not to be getting slower, unlike the rest of the system... On my desktop even a cold start of libreoffice takes less than a second, which is curiously faster than gnome-documents as far as I can see (using a clock and my eyes, not at all scientific).

For the substance of the proposal, see my recent mail on fedora-devel about mutable/static ASLR - after thinking about this more I can't see prelink as a significant problem (... which is not making the decision easier - as argued above PIE is probably the decision that has larger cost).

Replying to [comment:19 mitr]:

  1. For postgresql, I have opened a bug [https://bugzilla.redhat.com/show_bug.cgi?id=947022] requesting hardening to be enabled but so far I haven't got any response.

In my opinion, any bugs reporting violation of packaging guidelines should be looked at in a prompt fashion.

I'm sorry to say I'm much more worried about unhandled crash reports (which may indicate possibly exploitable bugs) than missing hardening; that's not to say the bug shouldn't be handled - just that I don't think some kind of forceful prioritization of these bugs makes sense.

I should not have used the word "prompt". I meant to say that such bugs should be handled in a reasonable amount of time instead of just being ignored and moved from one release to another.
[[BR]]

  1. "hard data that demonstrate the impact, if any, in a situation relevant to Fedora (in particular, taking into account prelink as it is deployed by default) would be very welcome but is not a strict requirement".

Please see comment:10. In short, "... the average delay of a PIE application over a non-PIE application was significantly below perceivable threshold".

That measures startup, not long-term performance impact. (PIE affects both, prelink optimizes primarily startup.)

You are right. For performance impact please see the ETH paper I have already linked to in comment:11

[[BR]]

So in summary, your proposal is to keep the existing guidelines and enforce them better? I'm in any case fine with that.

Yes.

Is this ticket ready now to be discussed in the upcoming FESCo meeting?

Replying to [comment:21 halfie]:

Replying to [comment:19 mitr]:

So in summary, your proposal is to keep the existing guidelines and enforce them better? I'm in any case fine with that.

Yes.

Thanks, closing.

For the other discussed aspects (PIE everywhere, dropping prelink), if you are interested please open a separate ticket with a proposal.

Login to comment on this ticket.