#2659 Arbitration request: Crypto policy prevents VPN connections
Closed: Rejected 9 months ago by defolos. Opened 9 months ago by dwmw2.

The point in a Linux distribution is to make coherent policy decisions around the full set of packages included in the distro, so that users aren't reduced to downloading tarballs of arbitrary versions of those packages from Sunsite, and praying to the deity of their choice that they work well together.

Fedora normally does this really well, but not always. I'd like to ask for FESCo's help to arbitrate and help set the distro's holistic policy in a case where the package maintainers can't agree. (Disclaimer: one of the offending package maintainers is me).

Background: DTLS ("Datagram TLS") offers security for datagram connections, and it's really important for VPNs. Cisco adopted it very early for their ubiquitous "AnyConnect" VPN product. So early, in fact, that their hardware acceleration predates the publication of the actual DTLS standard in RFC4347. Their "special" version of the protocol doesn't match either the last draft of draft-rescorla-dtls-05 or the final RFC; they cherry-picked some but not all of the last round of changes. When we supported this in OpenConnect, the Free Software client for various proprietary VPNs, it took some experimentation to work out which changes, and get something to interoperate with the Cisco protocol. When we added this to GnuTLS, we called it "DTLS 0.9".

TL;DR: GnuTLS "DTLS 0.9" does not match any version of the standard or preceding draft, and does not match any implementation other than Cisco's. It is ONLY used by OpenConnect and nothing else, ever. (In fact I don't think it even works for establishing a connection the normal way, since the Cisco protocol always "resumes" a simulated session.)

A recent update to the Fedora crypto-policy explicitly disabled the OpenConnect DTLS. In the general case, ideally such a policy wouldn't have gone live in Fedora before it had per-application overrides (https://gitlab.com/gnutls/gnutls/-/issues/1172). But in fact, since this change only affects OpenConnect, we don't even need to wait for that; this change is affecting only one application anyway.

So, a policy change broke VPN users, and their admins are now advising them to switch to the proprietary third-party client which ships its own ancient version of OpenSSL instead of using the system libraries. This is not, overall, an improvement in security. The change has been counter-productive.

In https://bugzilla.redhat.com/show_bug.cgi?id=1960763 I am being asked by one of the affected users to simply bypass the policy by unconditionally setting the GNUTLS_SYSTEM_PRIORITY_FILE environment variable to point to /dev/null in OpenConnect. I have done this in OpenConnect upstream: http://git.infradead.org/users/dwmw2/openconnect.git/commitdiff/7e862f2f0

But to do this in the Fedora package is wrong. It would mean that Fedora, as a whole, is disabling this protocol in the global config and then electing not to use that global config in the one place where the change makes a difference. Which completely negates the effect of the global policy change. Worse, it means that OpenConnect is then opting out of all the other parts of the global crypto policy which actually do make sense. Again, the "security" change ends up being counter-productive and having no beneficial effect at all.

I understand why Cisco DTLS was disabled in the Fedora system policy, and I even understand the reticence about turning it back on again when I asked. Just as everyone will understand why I've explicitly opted OpenConnect out of the system policies which break its functionality. But as a distribution we need to do better than that and define a holistic policy which makes sense.

Please let's re-enable Cisco DTLS in the Fedora crypto-policy package at least until we have the per-application (or prio-string based) way to re-enable them.


Who is the other party here, should be invite them here? Is it @asosedkin?

Who is the other party here, should be invite them here? Is it @asosedkin?

Probably, yes. I just issued a relatively open invitation to comment, in https://bugzilla.redhat.com/show_bug.cgi?id=1960763#c26

My point of view: It would be good if the default crypto policy for server apps would be enforced and for client apps permissive (at each connection, the system says that the cipher <cipher name> is weak it is recommended to inform your sysadmins about the possibility of using cipher <cipher name>. Continue connection?) It very similar to SELinux working modes. But it is hard to implement now.

My point of view: It would be good if the default crypto policy for server apps would be enforced and for client apps permissive (very similar to SELinux working modes). But it is hard to implement now.

Oh $DEITY yes. I consider it a huge mistake that this change was pushed without anyone making it a requirement to allow per-application overrides. If this change had been proposed while I was on FESCo that wouldn't have happened :)

But that isn't the point here; this particular change already is a per-application change anyway, because nothing else uses Cisco DTLS.

(However.... although Cisco AnyConnect was the first protocol that OpenConnect supported, we now support a bunch more types of VPN, some of which do use DTLSv1.0 and don't yet support DTLSv1.2. So we do want that per-application way to re-enable DTLSv1.0 too, it's just not quite as high priority, and not quite as gratuitously broken for us, as the Cisco one)

Please let's re-enable Cisco DTLS in the Fedora crypto-policy package at least until we have the per-application (or prio-string based) way to re-enable them.

If we do this (re-enable Cisco DTLS in the Fedora crypto-policy package) is this guaranteed to only effect OpenConnect, or would it also impact other programs? Or in other words, would it be possible for a different program to use this protocol version accidentally, without explicit opt-in?

I consider it a huge mistake that this change was pushed without anyone making it a requirement to allow per-application overrides. If this change had been proposed while I was on FESCo that wouldn't have happened :)

The way this works is that FESCo members aren't experts in all possible areas and can't be. All proposals are posted on fedora-devel, and this is the place where the proposal can be questioned and evaluated by any interested parties. FESCo members cannot do a comprehensive evaluation of proposals on their own. If the proposal is missing something subtle, as in this case, and nobody raises the issue, it is quite likely to be missed in the FESCo ticket and vote.

Please let's re-enable Cisco DTLS in the Fedora crypto-policy package at least until we have the per-application (or prio-string based) way to re-enable them.

If we do this (re-enable Cisco DTLS in the Fedora crypto-policy package) is this guaranteed to only effect OpenConnect, or would it also impact other programs? Or in other words, would it be possible for a different program to use this protocol version accidentally, without explicit opt-in?

I consider it a huge mistake that this change was pushed without anyone making it a requirement to allow per-application overrides. If this change had been proposed while I was on FESCo that wouldn't have happened :)

The way this works is that FESCo members aren't experts in all possible areas and can't be. All proposals are posted on fedora-devel, and this is the place where the proposal can be questioned and evaluated by any interested parties. FESCo members cannot do a comprehensive evaluation of proposals on their own. If the proposal is missing something subtle, as in this case, and nobody raises the issue, it is quite likely to be missed in the FESCo ticket and vote.

I am inclined to say that Cisco DTLS should just be enabled. Crypto policies should never be configured in such a way that the user must consider a weaker configuration in order to be productive.

If we do this (re-enable Cisco DTLS in the Fedora crypto-policy package) is this guaranteed to only effect OpenConnect, or would it also impact other programs? Or in other words, would it be possible for a different program to use this protocol version accidentally, without explicit opt-in?

This is guaranteed only to affect OpenConnect and perhaps also ocserv, the server for the same protocol. Nothing else would ever use Cisco DTLS, certainly not accidentally.

We should ensure that the ocserv configuration doesn't allow Cisco DTLS by default. Admins might need to enable that if they want to ensure compatibility with older Cisco clients. But even their client does support DTLSv1.2 now, as do the Cisco 'virtual' ASA servers. But the hardware is still stuck with the version they baked into it so many years ago, which is why OpenConnect clients need to support it.

We should ensure that the ocserv configuration doesn't allow Cisco DTLS by default.

How would that look?

/etc/ocserv/ocserv.conf has:

tls-priorities = "NORMAL:%SERVER_PRECEDENCE"

The cisco-client-compat and dtls-legacy options want to be set to false, I believe.

# This option will enable the pre-draft-DTLS version of DTLS, and
# will not require clients to present their certificate on every TLS
# connection. It must be set to true to support legacy CISCO clients
# and openconnect clients < 7.08. When set to true, it implies dtls-legacy = true.
cisco-client-compat = true

# This option allows one to disable the DTLS-PSK negotiation (enabled by default).
# The DTLS-PSK negotiation was introduced in ocserv 0.11.5 to deprecate
# the pre-draft-DTLS negotiation inherited from AnyConnect. It allows the
# DTLS channel to negotiate its ciphers and the DTLS protocol version.
#dtls-psk = false

# This option allows one to disable the legacy DTLS negotiation (enabled by default,
# but that may change in the future).
# The legacy DTLS uses a pre-draft version of the DTLS protocol and was
# from AnyConnect protocol. It has several limitations, that are addressed
# by the dtls-psk protocol supported by openconnect 7.08+.
dtls-legacy = true

Hello. Sorry for being unresponsive, I've been on a vacation.

First of all, I'm really not a fan of apps opting out of crypto-policies, and I can only echo your sentiment about Fedora failing to deliver a coherent experience here.

Blanket-reenabling DTLS 0.9 for everything is a bad idea due to several backends other than gnutls being unable to cherry-pick protocol versions. They can only take ranges, making it an all-or-nothing situation. Fortunately, we don't have to do this.

Currently available granularity for which crypto-policies can re-enable DTLS 0.9 for openconnect purposes would be, since F35, "everything using gnutls".
The subpolicy would be protocol@gnutls = +DTLS0.9. If this is enough for your purposes and you're sure this doesn't compromise the security of other gnutls-dependent applications, then I'm unsure why don't you recommend that workaround instead of switching to a different VPN client.

Custom subpolicy is one way to do it. Another one would be extending crypto-policies to generate a separate configuration file consumed just by openconnect (by pointing gnutls to it), and enabling DTLS 0.9 just for openconnect. Then we could do protocol@openconnect = +DTLS0.9 and make the enablement even narrower. I'm fine with taking this approach, but, unless there actually is a demand for a separate openconnect config, I'd prefer to address the root cause instead.

The root cause is that, in Fedora, crypto-policies controls gnutls in a way that hard-disables algorithms, leaving no way for the application to reenable them back. I wasn't the crypto-policies maintainer when this decision was made and I don't like it. It's not even done this way in RHEL. I have no idea why switching to hard-disablement wasn't the point where complaints would start flowing in, but, somehow, it wasn't. Nevertheless, I've raised the hard-disablement issue [1] before I've even learned of openconnect's existence and woes. Now there's a WIP gnutls PR to add a soft-disabling allowlisting configuration file format for gnutls, as well as some matching WIP code on the crypto-policies side to utilize it instead of the hard-disabling denylisting one. Once this work is finished and lands, this should allow reenabling any algorithm on the application level, through either priority strings or new specially-added API. I hoped that this will come ready in time for Fedora 36, and I'll file that as a system-wide change proposal (since this has the potential to affect everything using gnutls).

So, for F36, I currently think switching to soft-disablement is the proper way forward. For F35, this isn't ready yet. I don't know what to do, honestly. If you insist that reverting the actually-disabling-DTLS-0.9 MR is the way to go for F35, well, guess that's an option too, one that I haven't really considered before. I don't like this as our moving-forward solution, but, since the branch-off should've happened already now, I'm much more OK with it as a temporary F35-only measure though. What would be the downsides? Out of the whole variant spectrum, what would you advise me to do, both for F35 and for F36+?

[1] https://gitlab.com/gnutls/gnutls/-/issues/1172
[2] https://gitlab.com/gnutls/gnutls/-/merge_requests/1427
[3] https://gitlab.com/redhat-crypto/fedora-crypto-policies/-/commit/dd7d273d76b0739fcff5d95c39d7486bdb9b7410
[4] https://gitlab.com/redhat-crypto/fedora-crypto-policies/-/merge_requests/91

Metadata Update from @zbyszek:
- Issue tagged with: meeting

9 months ago

This will be discussed during today's meeting at 21:00 CEST.
@dwmw2, @asosedkin if you could join the meeting that'd be great.

Wow, I thought it was possible for the application to override any policy set in crypto-policies using the GnuTLS priority string, but I see the problem here is that that is not (or no longer true). My $0.02 is that the system crypto policy should be the default, but the application should get final say if it explicitly chooses to set its own policy. I would expect that if OpenConnect sets the priority string @SYSTEM:+VERS-DTLS0.9 then that ought to work. Until that works as expected, the crypto-policies change that broke OpenConnect should be reverted.

This has implications way beyond OpenConnect. I just tested G_TLS_GNUTLS_PRIORITY="@SYSTEM:+VERS-TLS1.0" epiphany -p https://tls-v1-0.badssl.com:1010/ and verified that the environment variable override doesn't work as intended.

I am inclined to say that Cisco DTLS should just be enabled. Crypto policies should never be configured in such a way that the user must consider a weaker configuration in order to be productive.

Problem is "DTLS 0.9" and 1.0 really need to be disabled by default. Applications that need them are broken and should either (a) be fixed, which is impossible in this case because we don't control old Cisco AnyConnect servers, or else (b) explicitly opt-in to insecure settings.

Note that OpenSSL has "DTLS 0.9" as well, but there it is named DTLS1_BAD_VER.

My $0.02 is that the system crypto policy should be the default, but the application should get final say if it explicitly chooses to set its own policy.

No disagreement here whatsoever.

Until that works as expected, the crypto-policies change that broke OpenConnect should be reverted.

It's kinda tangential here... a straw that broke the camel's back, I guess.

OK, if I revert it in F35 but not rawhide, would this make sense?

OK, if I revert it in F35 but not rawhide, would this make sense?

I don't think so, because we don't want to leave OpenConnect broken in rawhide....

While breaking OpenConnect in a way that users can't easily override it wasn't the intention, the goal of Crypto Policies is to provide a distribution with only secure protocols and cryptographic algorithms used by default.

We may disagree if disabling DTLS0.9 was done prematurely, the crux of the matter is that it will need to be disabled at some point in time. And rather sooner than later. Same issue exists with older algorithms too; we can't continue to support MD4, MD5, or soon SHA1 in arbitrary contexts, and claim that Fedora is secure out-of-the-box.

So my question is: what's the plan of OpenConnect maintainers to disable old and insecure algorithms? At what point in time will it be ok to tell users to use a subpolicy that specifies protocol@gnutls = +DTLS0.9 as a workaround, if they need to interoperate with this old hardware/software?

So my question is: what's the plan of OpenConnect maintainers to disable old and insecure algorithms? At what point in time will it be ok to tell users to use a subpolicy that specifies protocol@gnutls = +DTLS0.9 as a workaround, if they need to interoperate with this old hardware/software?

IMO it will never be OK. The GnuTLS priority API documented here no longer works properly, and that needs to be fixed.

I don't think you understood the question.

At some point in time we will have to force users to consciously opt in to re-enable support for obsolete and insecure algorithms and protocols (like DTLS0.9). As there is no other way to tell them "You are using insecure algorithm or protocol, support for it will be removed in the future", without requiring them to perform some special actions to re-enable support for them.

Also, this is not hypothetical, we did remove support for SSLv2 and SSLv3 in the past. DTLS0.9 will be next in line (as when we will remove TLS 1.0, we will also remove TLS 1.1 and its variants like DTLS1.0).

So, I'm sorry but "never" is not a realistic request. We need to have a plan on how do we deprecate and remove support for DTLS0.9.

Sorry, didn't get a change to respond before the meeting, very quick answers... I would be OK with re-enabling DTLS0.9 only in GnuTLS since we only build OpenConnect against DTLS0.9. We don't care about OpenSSL, and NSS never supported it anyway.

It isn't me advocating that people use the third-party Cisco client; that's what admins are falling back to when Fedora breaks.

My plan for disabling DTLS0.9 on the client is simple: We drop it when the servers no longer need it. In the interim we'd ideally accept it much the same as we do invalid server certificates — with a big scary warning and explicit user acceptance. We have an -allow-insecure-crypto option for that purpose in openconnect. But it doesn't work unless we actually have a way to 'opt out'.

My summary of consensus: revert specifically the "hard-disabling DTLS 0.9 in gnutls" crypto-policies change until gnutls switches away from hard-disablement in Fedora.
The impact on policies should effectively be the inverse of https://gitlab.com/redhat-crypto/fedora-crypto-policies/-/merge_requests/91/diffs. Reverting should happen both in rawhide and f35. I plan to work on these reverts this week.

My plan for disabling DTLS0.9 on the client is simple: We drop it when the servers no longer need it. In the interim we'd ideally accept it much the same as we do invalid server certificates — with a big scary warning and explicit user acceptance. We have an -allow-insecure-crypto option for that purpose in openconnect. But it doesn't work unless we actually have a way to 'opt out'.

Would it be possible for OpenConnect to implement DTLS0.9 on its own, perhaps in a memory safe language? Since nobody else will ever use DTLS0.9, it seems to me that the implementation (not the underlying crypto) should live in OpenConnect.

Also, do the servers that require DTLS0.9 have known vulnerabilities? A banner that says “Your VPN server is vulnerable to CVE-ABC-XYZ; see this link for details.” might be a good way to convince admins to upgrade.

Revert was done in crypto-policies-20210819-1.gitd0fdcfb.fc35 / crypto-policies-20210819-1.gitd0fdcfb.fc36, please confirm that this is what you were after.

Thanks. Please could you mark it as fixing e.g. https://bugzilla.redhat.com/show_bug.cgi?id=1960763 or https://bugzilla.redhat.com/show_bug.cgi?id=1990196 and allow the reporter(s) to confirm the fix?

This does also affect F33 onwards, not just F35, doesn't it?

Wait, with this discussion being all about the DTLS 0.9, I have forgotten about 1.0.

DTLS 0.9 hard-disabling done in https://gitlab.com/redhat-crypto/fedora-crypto-policies/-/merge_requests/91 has never reached F33/F34. This has been reverted, as we've recently agreed upon.

I haven't touched upon DTLS 1.0 hard-disabling, which is present in F33 as well. You want it gone too? Notable differences from the 0.9 situation?

The so-called "DTLS0.9" is used by Cisco AnyConnect, which is the protocol that OpenConnect has supported since its inception in 2008. It's the most important one, as far as we're concerned, and Cisco then eventually jumped straight to DTLSv1.2 at least on the virtual appliances, although there's still a lot of hardware with the DTLS0.9 acceleration baked in.

The other protocols we have added in released versions of OpenConnect since then have all been ESP-based, for which OpenConnect builds its own packets.

There are some more protocols we've added in our git tree recently which do use DTLS, some of which have servers that are limited to DTLSv1.0 and can't cope with DTLSv1.2. But those are not yet in a release, although we're planning a release soon.

TLDR: I think we can probably wait for the dynamic prio-string based solution for DTLSv1.0; it was only DTLS0.9 which was urgent.

Thanks.

@dwmw2 @asosedkin Do you want further assistance from FESCo? I'll put it on tomorrow's agenda and you're invited to the meeting if you wish.

I hope we're fine now.

Glad to hear that!

Metadata Update from @defolos:
- Issue close_status updated to: Rejected
- Issue status updated to: Closed (was: Open)

9 months ago

Back-linking to a related Fedora Change proposal that'd allow apps using gnutls to lift the restrictions imposed by crypto-policies: https://fedoraproject.org/wiki/Changes/GnutlsAllowlisting

Login to comment on this ticket.

Metadata