What version(s) of openssl and httpd do we need to grok "HIGH" correctly? The servers are rhel6.

Good question. httpd should grok "HIGH" in the SSLCipherSuite settings. I suspect the latest version of OpenSSL in RHEL6 will support all the "good" 256-bit ciphers.

It would be really nice to support TLSv1.2 but I think that's only in OpenSSL 1.0.1 and it isn't finalized (IIRC).

Unfortunately RHEL does not ship any TLSv1.2 capable browser so far. Additionally, most recent browsers still do not handle TLSv1.2 out of the box, see e.g. http://en.wikipedia.org/wiki/Transport_Layer_Security#Web_browsers for an overview (requires FF28+, IE11+, Opera 17+). If you can not use TLSv1.2, RC4 is unfortunately the only immune cipher AFAIK (see e.g. http://en.wikipedia.org/wiki/RC4#Security). Newer OpenSSL and newer NSS will help for RHEL 6.5+ (which is still not released).

The currently used/configured cipher suite is IMHO the only result which makes Qualys SSL labs that much happy as it is now...while having old OpenSSL without EC/GCM. Removing RC4 will mark us as vulnerable to BEAST - some time ago not having RC4 even degraded the overall rating (no idea how it is nowadays, did not recheck).

By the way, even Google has RC4 still in their cipher suite - and I guess it's for the same reasons like nearly all who have it: To mitigate BEAST vulnerability when not having TLSv1.2.

Replying to [comment:3 robert]:

Unfortunately RHEL does not ship any TLSv1.2 capable browser so far. Additionally, most recent browsers still do not handle TLSv1.2 out of the box, see e.g. http://en.wikipedia.org/wiki/Transport_Layer_Security#Web_browsers for an overview (requires FF28+, IE11+, Opera 17+).

Let's not get down in the weeds here. We don't need the browsers to be in RHEL now. As it stands right now, the current browser in Fedora 18 (FF 24.0) supports TLSv1.1 and the current browser in Fedora 19 (FF 25.0) supports TLSv1.2.

If you can not use TLSv1.2, RC4 is unfortunately the only immune cipher AFAIK (see e.g. http://en.wikipedia.org/wiki/RC4#Security). Newer OpenSSL and newer NSS will help for RHEL 6.5+ (which is still not released).

Like I already pointed out, [https://community.qualys.com/blogs/securitylabs/2013/09/10/is-beast-still-a-threat BEAST is already mitigated in clients] such as Firefox and IE so server-side mitigation using a broken cipher (RC4) is no longer necessary.

The currently used/configured cipher suite is IMHO the only result which makes Qualys SSL labs that much happy as it is now...while having old OpenSSL without EC/GCM. Removing RC4 will mark us as vulnerable to BEAST - some time ago not having RC4 even degraded the overall rating (no idea how it is nowadays, did not recheck).

They've come around to new thinking. RC4 is '''so''' broken that they, and many other security experts, are no longer recommend it.

By the way, even Google has RC4 still in their cipher suite - and I guess it's for the same reasons like nearly all who have it: To mitigate BEAST vulnerability when not having TLSv1.2.

We should be doing things correctly and setting the example. If nothing else, at least remove the AES-128 from the second cipher in the list so those of us that disable RC4 can get a good 256-bit cipher.

It should be noted that [https://wiki.mozilla.org/Security/Server_Side_TLS Mozilla recently came out with their recommendation for server-side TLS implementations]. Their recommendations are that AES is preferred to RC4 due to BEAST being difficult to achieve in TLSv1.0 but attacks against RC4 aren't mitigated at all.

Replying to [comment:5 sparks]:

Let's not get down in the weeds here. We don't need the browsers to be in RHEL now. As it stands right now, the current browser in Fedora 18 (FF 24.0) supports TLSv1.1 and the current browser in Fedora 19 (FF 25.0) supports TLSv1.2.

If we care about TLSv1.1 and TLSv1.2 on our website, we maybe should ask our Firefox package maintainers to enable TLSv1.1 and TLSv1.2 by default also in Firefox for Fedora 18 and 19. It at least would make sense from my point of view (if we would like to lead, let's lead on both ends, please).

Replying to [comment:6 sparks]:

It should be noted that [https://wiki.mozilla.org/Security/Server_Side_TLS Mozilla recently came out with their recommendation for server-side TLS implementations]. Their recommendations are that AES is preferred to RC4 due to BEAST being difficult to achieve in TLSv1.0 but attacks against RC4 aren't mitigated at all.

The ciphersuite suggested by Mozilla looks fine to me (especially if BEAST doesn't seem to be an issue anymore even with TLSv1.0 these days) - unfortunately the ciphersuite is very short...at least as long as RHEL 6.5 is not out.

Replying to [comment:7 robert]:

Replying to [comment:6 sparks]:

It should be noted that [https://wiki.mozilla.org/Security/Server_Side_TLS Mozilla recently came out with their recommendation for server-side TLS implementations]. Their recommendations are that AES is preferred to RC4 due to BEAST being difficult to achieve in TLSv1.0 but attacks against RC4 aren't mitigated at all.

The ciphersuite suggested by Mozilla looks fine to me (especially if BEAST doesn't seem to be an issue anymore even with TLSv1.0 these days) - unfortunately the ciphersuite is very short...at least as long as RHEL 6.5 is not out.

I'm not sure what you mean by "unfortunately the ciphersuite is very short". I believe setting the SSLCipherSuite to "HIGH,!LOW,!MD5,..." would be best. By not hard coding ciphers into the servers these settings can move with time as OpenSSL comes out with better choices and removes/downgrades choices that are no longer good.

Replying to [comment:8 sparks]:

I'm not sure what you mean by "unfortunately the ciphersuite is very short". I believe setting the SSLCipherSuite to "HIGH,!LOW,!MD5,..." would be best. By not hard coding ciphers into the servers these settings can move with time as OpenSSL comes out with better choices and removes/downgrades choices that are no longer good.

I simply meant that there are not so many ciphers left without the EC ones.

From what I get by the Mozilla recommendation the order is important and optimized (obviously a mixture out of CPU load vs. security). Not ordering (= not hard coding) the ciphers likely produces a less optimized list that causes more (unnecessary?) load? They also say "If you have to pick them manually for your application, make sure you keep this ordering." - and I guess this is for good reasons without being a crypto expert at all.

So, where are we here?

Should we try a new setting? leave it alone?

Well, since I started this ticket Microsoft has come out against RC4 and SHA-1 in favor of AES and SHA-2. Others, too, have said that no one should be using the broken RC4 cipher.

<farnsworth>Good news everyone</farnsworth>

RHEL 6.5 just came out and has TLS 1.2 support in it.

So, likely we need to reevaluate what we want to do here based on that information. We should probibly decide soon if we want to test a configuration as we go back into freeze next tuesday and I would want a while on a config to confirm it doesn't cause any problems.

Replying to [comment:12 kevin]:

RHEL 6.5 just came out and has TLS 1.2 support in it.

Yeah, I was going to say something about that but the release hadn't occurred yet.

I suspect that most browsers will support TLSv1.2 in the next couple of releases (FF 25.0 already supports it if you turn it on) so we should be able to use the best options available to us without trouble. I'm not sure that we can get statistics on this but it would be nice to see a "survey" of all the clients' crypto capabilities that connect to our servers.

SRT and PST discussed this exact topic today (for OpenShift) and Hubert Kario (PST) came up with the following settings:

Suggested configuration:

SSLHonorCipherOrder on[[BR]]
SSLProtocol +SSLv3 +TLSv1 +TLSv1.1 +TLSv1.2[[BR]]
SSLCipherSuite ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:DHE-RSA-CAMELLIA256-SHA:DHE-RSA-CAMELLIA128-SHA:DHE-RSA-SEED-SHA:AES256-GCM-SHA384:AES256-SHA256:AES128-GCM-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:CAMELLIA256-SHA:CAMELLIA128-SHA:IDEA-CBC-SHA:SEED-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:DES-CBC3-SHA

In depth explanation:

The suites are sorted according to this criteria:
* PFS (ECDHE first, DHE second)
* used MAC: GCM, SHA-2, SHA1
* used cipher: AES256, AES128, CAMELLIA, IDEA, SEED
* 3DES as dead last, both because of security and performance reasons, left only for compatibility with IE on Win XP

Excluded suites:
* ECDSA, DSS - require special certificates
* KRB5 - non useful on Internet
* PSK - requires specific configuration
* export ciphers - completely insecure
* single DES - completely insecure
* RC4 - insecure

We first sort ciphers by PFS capability, then by MAC used, and finally by main cipher used, as some libraries (most notably NSS) don't support SHA384 in TLS while support AES128-GCM with SHA256. Using default openssl cipher ordering the negotiated cipher suite would be AES256 SHA-1 with no PFS.

Since session resumption/tickets are always encrypted with AES128 (and OpenShift does support and use them), using AES256 or other 256 bit ciphers doesn't significantly increase security of the resulting connection.

For OpenSSL v1.0.1e (as shipped in RHE6.5) this results in quoted cipher suite list.

Human readable version:[[BR]]
ECDHE-RSA-AES256-GCM-SHA384[[BR]]
ECDHE-RSA-AES256-SHA384[[BR]]
ECDHE-RSA-AES128-GCM-SHA256[[BR]]
ECDHE-RSA-AES128-SHA256

ECDHE-RSA-AES256-SHA[[BR]]
ECDHE-RSA-AES128-SHA

DHE-RSA-AES256-GCM-SHA384[[BR]]
DHE-RSA-AES256-SHA256[[BR]]
DHE-RSA-AES128-GCM-SHA256[[BR]]
DHE-RSA-AES128-SHA256

DHE-RSA-AES256-SHA[[BR]]
DHE-RSA-AES128-SHA[[BR]]
DHE-RSA-CAMELLIA256-SHA[[BR]]
DHE-RSA-CAMELLIA128-SHA[[BR]]
DHE-RSA-SEED-SHA

AES256-GCM-SHA384[[BR]]
AES256-SHA256[[BR]]
AES128-GCM-SHA256[[BR]]
AES128-SHA256

AES256-SHA[[BR]]
AES128-SHA[[BR]]
CAMELLIA256-SHA[[BR]]
CAMELLIA128-SHA[[BR]]
IDEA-CBC-SHA[[BR]]
SEED-SHA

ECDHE-RSA-DES-CBC3-SHA[[BR]]
EDH-RSA-DES-CBC3-SHA[[BR]]
DES-CBC3-SHA

Replying to [comment:14 sparks]:

SSLProtocol +SSLv3 +TLSv1 +TLSv1.1 +TLSv1.2[[BR]]

For some reason, our Apache doesn't like "+TLSv1.1 +TLSv1.2"...which seems weird to be, given:

{{{
[root@proxy01.stg conf.d]# rpm -q openssl httpd
openssl-1.0.1e-16.el6_5.4.x86_64
httpd-2.2.15-29.el6_4.x86_64

[root@proxy01.stg conf.d]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.5 (Santiago)
}}}

and the fact that the docs (https://httpd.apache.org/docs/2.2/mod/mod_ssl.html#sslprotocol) say:

{{{
TLSv1.1 (when using OpenSSL 1.0.1 and later)
...
TLSv1.2 (when using OpenSSL 1.0.1 and later)
}}}

which we clearly are.

For now, I've enabled this configuration in staging with the SSLProtocol line replaced with:

{{{ SSLProtocol +SSLv3 +TLSv1" }}}

I ran an SSLLabs test on it:

https://www.ssllabs.com/ssltest/analyze.html?d=apps.stg.fedoraproject.org&ignoreMismatch=on

...which scores an A (ignoring certificate name mismatch issues).

Replying to [comment:15 codeblock]:

For some reason, our Apache doesn't like "+TLSv1.1 +TLSv1.2"...which seems weird to be, given:

If I am not mistaken, there was no update of mod_ssl in RHEL 6.5 (Build Date: Fr 02 Aug 2013 14:03:06 CEST). That means that mod_ssl in RHEL 6.5 is still built against the old OpenSSL 1.0.0 (without TLSv1.2 support). As the SSLProtocol options are IIRC built-time relevant a rebuild of mod_ssl using OpenSSL 1.0.1e would be required to make these options available. Right now there is only the OpenSSL 1.0.1e at run-time (which obviously enables TLSv1.2 on the library level). Additionally the mod_ssl in RHEL 6 might be old enough to not contain the required source code changes as well.

From my point of view the relevant code is in modules/ssl/ssl_engine_init.c at ssl_init_ctx_protocol(). While https://svn.apache.org/repos/asf/httpd/httpd/branches/2.2.x/modules/ssl/ssl_engine_init.c has references to TLSv1.1 and TLSv1.2 there are absolutely none in httpd-2.2.15-29.el6_4 - which is the latest RPM for RHEL.

ok, if you look in the report you will see that TLSv1.0, 1.1 and 1.2 are all enabled. Asking around it seems if you do:

+TLSv1

you get all TLS versions. mod_ssl is too old/not built with new openssl, so it doesn't grok the 1.1 and 1.2 directives, but the +TLSv1 gives you TLSv1 and anything else it can.

So, I think dropping the non understood "+TLSv1.1 +TLSv1.2" is the right thing to do and we should be all set.

Can anyone see any problems in staging? Or shall we push this to production and see what happens?

ok, this is live now in production.

Any issues or adjustments left? Or shall we close it out?

Looks a lot more secure, now. SSLLabs is beating us up on the CA's weak certificate but that's nothing we can control right now. Thanks for making these updates!

Cool. I will close this then... please reopen or file a new ticket if any changes are needed.

1. This is still an issue for copr.fedoraproject.org
2. Central documentation about which configuration should be used for publicly available http servers is missing in: http://infrastructure.fedoraproject.org/infra/

If there is a different place this should be documented, please add a pointer here.

Replying to [comment:14 sparks]:

SRT and PST discussed this exact topic today (for OpenShift) and Hubert Kario (PST) came up with the following settings:
In depth explanation:

The suites are sorted according to this criteria:
* PFS (ECDHE first, DHE second)
* used MAC: GCM, SHA-2, SHA1
* used cipher: AES256, AES128, CAMELLIA, IDEA, SEED
* 3DES as dead last, both because of security and performance reasons, left only for compatibility with IE on Win XP

We first sort ciphers by PFS capability, then by MAC used, and finally by main cipher used, as some libraries (most notably NSS) don't support SHA384 in TLS while support AES128-GCM with SHA256. Using default openssl cipher ordering the negotiated cipher suite would be AES256 SHA-1 with no PFS.

Since session resumption/tickets are always encrypted with AES128 (and OpenShift does support and use them), using AES256 or other 256 bit ciphers doesn't significantly increase security of the resulting connection.

From your arguments, GCM mode is prefered to SHA-2/SHA1 and AES256 is not more
important than AES128, but in the actual suggested suites list, AES256 is
prefered over AES128, regardless of the used MAC:

Human readable version:[[BR]]
ECDHE-RSA-AES256-GCM-SHA384[[BR]]
ECDHE-RSA-AES256-SHA384[[BR]]

^- needs to switch with -v

ECDHE-RSA-AES128-GCM-SHA256[[BR]]
ECDHE-RSA-AES128-SHA256

ECDHE-RSA-AES256-SHA[[BR]]
ECDHE-RSA-AES128-SHA

DHE-RSA-AES256-GCM-SHA384[[BR]]
DHE-RSA-AES256-SHA256[[BR]]

^- needs to switch with -v

DHE-RSA-AES128-GCM-SHA256[[BR]]
DHE-RSA-AES128-SHA256

AES256-GCM-SHA384[[BR]]
AES256-SHA256[[BR]]

^- needs to switch with -v

AES128-GCM-SHA256[[BR]]
AES128-SHA256

Also it might make more sense to prefer DHE with GCM mode over ECDHE without
GCM mode, since afaik ECDHE is faster than DHE but not more secure. But this
might not make any difference in practice.

Replying to [comment:23 till]:

I think we've made some strides in the correct direction and can now point to [https://wiki.mozilla.org/Security/Server_Side_TLS Mozilla's SSL recommendations] as the best way forward.

For backwards compatibility:

Ciphersuite: ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128:AES256:AES:DES-CBC3-SHA:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK [[BR]]
Versions: SSLv3, TLSv1, TLSv1.1, TLSv1.2[[BR]]
RSA key size: 2048[[BR]]
DH Parameter size: 1024[[BR]]
Elliptic curves: secp256r1, secp384r1, secp521r1[[BR]]

And for best use:

Ciphersuite: ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!3DES:!MD5:!PSK [[BR]]
Versions: TLSv1.1, TLSv1.2[[BR]]
RSA key size: 2048[[BR]]
DH Parameter size: 2048[[BR]]
Elliptic curves: secp256r1, secp384r1, secp521r1 (at a minimum)[[BR]]

1. copr should be fixed. Can you confirm?

2. We could document it in infra-docs?

Finally is this changed from the recommendation before? Should we change to the 'best use' one?

Replying to [comment:25 kevin]:

1. We could document it in infra-docs?

It would probably be good to document the Mozilla link in the documentation to make it easier to go find the latest settings as these recommendations will likely change.

Finally is this changed from the recommendation before? Should we change to the 'best use' one?

This is ''close'' to the recommendations before. There were lots of discussions between Mozilla, OpenSSL, and other open source and security people and Mozilla came out with this. It's a compromise in some areas but it's sound security advice.

The biggest part of this is that you're not over-doing the level of encryption that isn't supported by the certificates. We're eliminating bad ciphers and anonymous connections.

I think we are all done here, although we are running into a very few people who are having problems.

We can address them in their own tickets.