#1783 Firefox 57 and the Updates Policy

Created a month ago by adamwill
Modified a month ago

This is another follow-on to the devel@ discussion, along with #1782 . That one covers the general question of what the updates-testing repo should be used for. This ticket is about whether Firefox 57 is appropriate as an update (per the Updates Policy).

The Updates Policy states, in part:

"Package maintainers MUST:

  • Avoid Major version updates, ABI breakage or API changes if at all possible.
  • Avoid changing the user experience if at all possible.
  • Avoid updates that are trivial or don't affect any Fedora users."

There is, so far as I can tell, no exception to the policy for Firefox.

Most Firefox updates "chang[e] the user experience" at least a little, and thus could be considered in violation of the policy, but the 'if at all possible' clause allows a reasonable counter-argument: upstream does not backport security fixes, except to the ESR release stream we have decided not to follow. It's also the case that, in general, Fedora users seem to prefer getting new Firefox releases rather than being stuck on whichever version was in the release they are running at release time.

However, Firefox 57 seems to be a special case. It is being pushed by upstream as one of the biggest releases for several years. And it comes along with at least one specific change which has the potential to vastly 'change the user experience' for at least some users: it disables all add-on types besides the type upstream refers to as 'WebExtensions'.

I don't know if I'm a representative Firefox user, but as I look at my Add-Ons view, I have 10 add-ons marked as 'Legacy' - i.e. they won't work with Firefox 57 - and just 3 that are apparently compatible. Some of the 'legacy' ones are relatively unimportant, but the group also includes LastPass and HTTPS Everywhere, for instance, which seem fairly significant.

Given the significance of the Firefox 57 release and the clear intent of the updates policy that stable release updates should not disrupt the user experience, I'd say there's a strong case that Firefox 57 should not be considered appropriate as an update for current stable releases - at least not on day 1. I don't think it's incumbent upon us to offer up our user base as day 1 testers of these changes.

So I'm proposing that FESCo should request that the Fedora 26 Firefox 57 update be unpushed, and work together with the maintainer to come up with a viable plan for Firefox updates to Fedora 25 and 26 going forward, which is both manageable for the maintainer and in-line with the Updates Policy, so far as that can be achieved.

The case of Fedora 27 could also be considered, as we are now in the post-Beta phase, and the "Avoid Major version updates" clause does apply between Beta and Final (though "Avoid changing the user experience if at all possible." does not).

Thanks!

As Gerald said in the other ticket, I'd like to note that I don't intend this ticket as any kind of accusation against @stransky , I don't believe he had any bad intentions at all in this case.

@stransky Is maintaining a Firefox 56 branch in F26 viable from a security-update perspective? I see that upstream hasn't made it an ESR release, which is kind of unfortunate.

I'd be in favor of allowing the 57 update into F27 (even post-beta), but keeping F26 at 56 at least initially and as long as possible — possibly updating if there happens to be a security update that can't be avoided

And, it'd sure be nice to get to arbitrary branching and flatpak here sooner rather than later. :)

Adam, I do agree that Fx 57 should be removed from updates-testing for all branches - for reasons outlined in the other ticket.

Regarding post release (Nov 14th) - I completely understand the reasoning - but as you mention, Fx 57 is different. The release date and impact have been well publicised. True that many extensions aren't yet ready - but the deadline is the deadline. It isn't a surprise.

While I have an issue with pushing this change before the release date into the update process (because that in effect moves up the deadline and some developers are still working on their extensions), by the same logic I don't believe it would be correct to hold up the release because some extensions missed the well publicised (like for over a year) deadline.

Additionally, Fx has done a great job slowly tweaking previous releases to introduce the webextensions. As you mention, it is quite simple to determine which extensions may be problematic and plan for it.

"plan for it" is rather vague wording. How exactly is one supposed to "plan for" an extension they rely on not working any more? There isn't really any viable alternative, is there? Besides switching to ESR releases downloaded from upstream, and dealing with any problems in switching from 56 to 52, which - at least from my cursory research - sounds like a possibility?

BTW, to be clear on the proposal here, I'm not saying 'we should definitely say FF 57 never goes into Fedora 26'. I'm saying there's no need to rush into it, and the Updates Policy quite strongly supports that approach. I'm saying let's stick with 56 for at least a week or two after upstream releases 57, and see how things shake out.

BTW, to be clear on the proposal here, I'm not saying 'we should definitely say FF 57 never goes into Fedora 26'. I'm saying there's no need to rush into it, and the Updates Policy quite strongly supports that approach. I'm saying let's stick with 56 for at least a week or two after upstream releases 57, and see how things shake out.

FF57 fixes security issues which will be public at the FF57 release data. Leaving 56 would be potentially risky.

Downgrading from 56 to 52 ESR is possible from RPM perspective but may cause various issues with your Firefox profile - profile downgrading is not supported.

Also consider that Mozilla has hard data about extensions usage from telemetry and that data says that majority of users does not have any extenstion installed or have 1-2 one(s). That's also reason why they're switching to WebExtensions. Don't get an impression that your missing extension is a general problem which hit every Firefox user - because that's not true.

"plan for it" is rather vague wording. How exactly is one supposed to "plan for" an extension they rely on not working any more? There isn't really any viable alternative, is there? Besides switching to ESR releases downloaded from upstream, and dealing with any problems in switching from 56 to 52, which - at least from my cursory research - sounds like a possibility?

What I meant by that was that the move to Fx 57 has been a well publicised and planned migration. Many developers have publicly stated that they have no intention of moving to webextensions; which means that in the new exciting world of Quantum those extensions are gone. Other developers are rushing to meet the deadline. If a particular extension isn't working on November 14th, that shouldn't be a surprise to anyone and I don't view it as a Fedora issue.

BTW, to be clear on the proposal here, I'm not saying 'we should definitely say FF 57 never goes into Fedora 26'. I'm saying there's no need to rush into it, and the Updates Policy quite strongly supports that approach. I'm saying let's stick with 56 for at least a week or two after upstream releases 57, and see how things shake out.

If Mozilla had not communicated what was occurring with Fx 57 I might agree - but in this instance they've done an excellent job in hammering away at the message. Fedora is a leading edge distribution and I believe the vast majority of our users would expect to see Fx 57 in Fedora as close to the release date as possible.

Also consider that Mozilla has hard data about extensions usage from telemetry and that data says that majority of users does not have any extenstion installed or have 1-2 one(s). That's also reason why they're switching to WebExtensions. Don't get an impression that your missing extension is a general problem which hit every Firefox user - because that's not true.

I think the Fedora audience (including the target audience for Workstation in specific) is likely to differ from the general Firefox audience.

Also note that 'vast majority' is a reassuring yet non-specific phrase. I'd say 90% would count as a 'vast majority', but it is acceptable to cause problems for 10% of Fedora Firefox users with a stable update? I'd be more reassured by the reassurances if they came with specific numbers; are these available somewhere?

"If Mozilla had not communicated what was occurring with Fx 57 I might agree - but in this instance they've done an excellent job in hammering away at the message."

I, for one, did not know about this change until the devel@ mailing list thread.

Also note that 'vast majority' is a reassuring yet non-specific phrase. I'd say 90% would count as a 'vast majority', but it is acceptable to cause problems for 10% of Fedora Firefox users with a stable update? I'd be more reassured by the reassurances if they came with specific numbers; are these available somewhere?

I don't have numbers on that... what I can say is that Fedora is known and is advertised as a "Leading Edge" distribution. The logical assumption would be that message has gotten out to our user community so they understand what to expect - and that is part of the reason they chose Fedora as their distribution. As far as issues with Fx 57 - again, that has been well publicised for several years now - and actually I do believe Fx 57 will be one of the best quality releases that Mozilla has had for many years... why, because it is pivotal for their survival. If it crashes and burns the media will pounce, and Mozila is well aware of that fact. That is why the date of November 14th is important.

I, for one, did not know about this change until the devel@ mailing list thread.

And that is fine - my point was that many people do know about this. There are always exceptions. I do agree with @stransky about the webextension situation. Many of the most popular extensions will meet the November 14th deadline. Many of those that won't have been abandoned and a delay won't change that fact.

"what I can say is that Fedora is known and is advertised as a "Leading Edge" distribution."

No, it isn't. Right on fedoraproject.org, for instance, we say "Fedora Workstation is a polished, easy to use operating system for laptop and desktop computers, with a complete set of tools for developers and makers of all kinds." If you click on the Workstation link it says "Fedora Workstation is a reliable, user-friendly, and powerful operating system for your laptop or desktop computer. It supports a wide range of developers, from hobbyists and students to professionals in corporate environments." The random user quote I got on the page says "The plethora of tools provided by Fedora allows me to get the job done. It just works."

AFAICT, the phrase "leading edge" - or anything like it - does not occur anywhere in our prominent, user-facing marketing material.

AFAICT, the phrase "leading edge" - or anything like it - does not occur anywhere in our prominent, user-facing marketing material.

Someone needs to notify Red Hat and Matt Miller: https://twitter.com/RedHatNews/status/886624477206896642

Edited a month ago by gbcox

"Compatible with Firefox 57" counts are misleading. Many extensions are being renamed when ported to webextensions. The new name is very low in popularity counts since it didn't exist till recently, the old one is listed with lots of users and no upgrade path. Other extensions have specific versions which only work starting from 0.57, the web site will hide those if you visit with a pre-0.57 version. Yet other extensions like umatrix are listed as compatible with 0.57 but apparently it's not completely the case, the compatibility checker is not perfect. I don't believe anyone has a clear visibility on the situation except Mozilla via Telemetry.

And Mozilla may choose to ignore the state of some extensions it does not like. Some Mozillans are quite hostile to extensions that "break" their pal's websites by filtering cookies for example. The whole Pocket mess highlighted how Mozilla priorities did not align with part of their userbase. This part is the most likely to have disabled Telemetry. If Mozilla wants 0.57 to succeed it really needs to priorise the port of extensions it does not like and be careful about Telemetry biases.

Mozilla did make a stellar job in announcing the change and promoting ESR for entities that wanted more change time, but its communication on the extension universe state is frankly a mess.

I definitely say leading edge and am probably going to keep doing it. :)

As noted previously, it's bleeding that I'm trying to stay away from as a description. "You get new stuff" is definitely part of our identity in the First foundation. But, it's also not all there is, and we should definitely consider the impact of change on our users.

Sure. No point getting too caught up in the terminology, anyway - the intent is more important than what we call it. I don't think it'd be compromising The Fedora Way to just sit back for a week or two and see how the release plays out in mass usage before we jump on it.

a month ago

Metadata Update from @maxamillion:
- Issue tagged with: meeting

Sure. No point getting too caught up in the terminology, anyway - the intent is more important than what we call it. I don't think it'd be compromising The Fedora Way to just sit back for a week or two and see how the release plays out in mass usage before we jump on it.

The reality is that when the update goes through updates-testing it usually takes a week or two to clear and get pushed into stable. I believe the existing process already accomplishes what you are attempting to do. Problems will be handled by negative karma. Push it to updates testing on November 14th and let the process work as intended. Older releases take longer to clear because typically not as many people are actively testing.

Edited a month ago by gbcox

Firefox is among the group of packages that often get pushed (or unpushed...) much faster than that, due to high user count and high visibility. Of course, it could go to u-t with no autopush threshold, or a very high one.

Firefox is among the group of packages that often get pushed (or unpushed...) much faster than that, due to high user count and high visibility. Of course, it could go to u-t with no autopush threshold, or a very high one.

Well, the last few releases had a few issues getting out of the gate. In any event, my point was no need to reinvent the wheel - the updates-testing process has the features to slow things down if required - especially if we're only talking about a few weeks - but remember, some extensions are gone for good and it isn't practical to delay the push to stable for that reason. Also as Martin pointed out, there are security issues related to delay and profile issues (which, believe me, you don't want to go there) if you try downgrade to a previous Fx version.

Guys, I need to say that there's no alternative to FF57 release, because:

  • we can't update to 52 ESR - it breaks user profiles, saved data for web sites etc.
  • we can't keep 56 and backport patches - it's PITA for maintainers and you need access to Mozilla security bugs (to be at s-g group is not enough since the exploit/0day marked evolved). AFAIK nobody from RH/Fedora has that access. I did the backports years ago for RHEL but the situation is different today.

So only alternative is to ship FF57. Also notice that next ESR (59) will come soon and brings FF57 changes to enterprise.

Also consider that Mozilla has hard data about extensions usage from telemetry and that data says that majority of users does not have any extenstion installed or have 1-2 one(s). That's also reason why they're switching to WebExtensions. Don't get an impression that your missing extension is a general problem which hit every Firefox user - because that's not true.

I think the Fedora audience (including the target audience for Workstation in specific) is likely to differ from the general Firefox audience.

Sorry but I think that's faulty generalization [1]. We hear only few loud people here but I also have good feedback for the FF57 test packages which is not shown here . That's also reason I put the FF57 to testing to get wider audience.

[1] https://en.wikipedia.org/wiki/Faulty_generalization

@stransky I'm not generalizing based on "a few loud people". I'm talking about Fedora's explicit and intentional target audience, which is different from Mozilla's.

Correct me if I'm wrong, but after going back and forth with Adam, I understand that he is suggesting a delay of potentially a few weeks to stable. In the conversation above I thought we both came to an accord that this could be handled with the updates-testing process. Just turn off autopush and watch the karma. Having Fx 57 in updates-testing for a few extra days isn't going to be a big issue. That said, a delay of a month or even skipping Fx 57 I don't believe is warranted or advised because of the issues discussed earlier. It simply isn't appropriate to hold the Fx release hostage because extension xyz isn't available.

@gbcox it's not really up to us to make a decision, though. I mean, I can withdraw the ticket. But I'm perfectly happy with fesco reading this discussion and deciding what they would like to do. I'm not sure we're getting more signal than noise with further discussion at this point, though.

@adamwill - no, don't withdraw the ticket... it is a worthwhile discussion. As I mentioned I was initially confused a bit with what you were requesting... and after a bit of back and forth, I got it. It also brings up the very good issue of communication within the project when major software components are being updated. We should also consider augmenting the upstream communication with our own for the community so everyone is aware something big is coming.

Aside from the webextension part FF57 has a completely different rendering engine and UI.

It would be very unsetling for users to have all those changes in a major desktop app drop on them mid-release.

Since it appears there is no good way to delay FF57 I'd advocate stransky was more right than he thought in pushing it to updates-testing: it should be tested now and it should be part of F27 from day one, even as a prerelease.

At least get some mileage about being first, and avoid a mid-release crisis.

Since it appears there is no good way to delay FF57 I'd advocate stransky was more right than he thought in pushing it to updates-testing: it should be tested now and it should be part of F27 from day one, even as a prerelease.

I agree with the testing part, I disagree hijacking the updates-testing process to accomplish that. That is being discussed here: https://pagure.io/fesco/issue/1782

FESCo Meeting 2017-10-13

firefox 57beta is removed from f25/f26 updates-testing but stays in f27/rawhide. When final it out, maintainer is encouraged to allow extra time in f25/f26 for testing (+1:7, -1:0, +0:0)
a month ago

Metadata Update from @maxamillion:
- Issue untagged with: meeting

FESCo Meeting 2017-10-13
When final it out, maintainer is encouraged to allow extra time in f25/f26 for testing (+1:7, -1:0, +0:0)

Haha, what an attitude of leading edge distribution ;-) Tests are supposed to happen before and not after final release. Also users may ask why other distros (maybe not so super truper leading edge ones ;-)) already have Firefox 57 but we don't...

FESCo Meeting 2017-10-13
firefox 57beta is removed from f25/f26 updates-testing but stays in f27/rawhide. When final it out, maintainer is encouraged to allow extra time in f25/f26 for testing (+1:7, -1:0, +0:0)

I understand the first part, but help me understand the other part. Why do we need extra time for testing? Firefox 57 should be as quality release as any other and yes, there will be a rather significant disruption due to removed "old" plugin support, but how is "extra time for testing" going to change that? It will just surprise people, who don't follow Mozilla news, 2 or so weeks later. The disruption will not be any smaller.

I would also like to point out that it comes with a price: users will not get a newer version of their browser as soon as possible as they're used to and most importantly they will be exposed to security threads because with FF57 there will be announced CVEs it's fixing. That's why we must and have always followed upstream as closely as possible.

Maintaining FF56 and backporting all security fixes there for the time being is out of question. We don't have expertise for that and AFAIK no distribution does that for the very same reason.

As Martin pointed out, introducing FF52 ESR would be possible from the RPM point of view, but it would bring extra maintenance and no one knows how Firefox and its config files would be ready for it. And upgrading users from FF56 to F52 is definitely not what they expect.

So the only sensible solution is to push stable Firefox 57 to F25/26 as soon as it's available and make users ready for the disruptive change by informing them through our popular channels.

"how is "extra time for testing" going to change that?"

It will give us more time to catch any unexpected consequences of the major changes in the update, and it will also give time for addons to get updated, the inevitable bugs in the first attempts to update addons to get fixed, and perhaps workarounds for any significant issues to emerge and be widely discussed.

FESCo Meeting 2017-10-13
firefox 57beta is removed from f25/f26 updates-testing but stays in f27/rawhide. When final it out, maintainer is encouraged to allow extra time in f25/f26 for testing (+1:7, -1:0, +0:0)

I understand the first part, but help me understand the other part. Why do we need extra time for testing? Firefox 57 should be as quality release as any other and yes, there will be a rather significant disruption due to removed "old" plugin support, but how is "extra time for testing" going to change that? It will just surprise people, who don't follow Mozilla news, 2 or so weeks later. The disruption will not be any smaller.

When words are chosen, there is a purpose. There is a great deal of leeway in the FESCo statement for the maintainer to use his best judgement. It's not a edict. I do agree there is nothing that can be done regarding the legacy extensions. There will be breakage by design and replacements may or may not have the previous functionality. People are going to have to adapt. IMO the increased performance and reliability of Fx far outweigh any inconvenience. That is also the judgement of Mozilla - and it's their software.

Edited a month ago by gbcox

"how is "extra time for testing" going to change that?"

I can't speak for all of FESCo, but when I suggested that what I meant was the way the kernel maintainers handle updates. If they simply submitted a kernel update with the default update properties it would go to stable pretty much instantly (since everyone has that package installed and it gets tons of feedback). I would suggest firefox is similar. Lots of people have it installed and a simple +3 karma is likely to push it stable before it spends any time in testing at all. So, instead I was thinking the firefox maintainer could disable autokarma and allow the update to be in updates-testing for a few days to make sure there are not any serious issues. That is of course a trade off on the security issues, but I have no idea how serious those are.

That is exactly what I was thinking as well. When we push a stable kernel update, we leave auto-karma off because we want it to at least hit updates-testing (for current release we will get 3 karma before it ever gets there). After 24 hours in updates-testing, we push provided the karma is good. For a rebase though, we try to give it several days in updates-testing. This gives us more feedback and users more time to adjust.

That is of course a trade off on the security issues, but I have no idea how serious those are.

Just to give you an idea how serious they are:
Firefox 54 - fixing 4 critical, 11 high impact CVEs
Firefox 55 - 6 critical, 11 high
Firefox 56 - 3 critical, 6 high

And countless moderate and low impact CVEs. No one knows how many of them will be fixed and announced with FF57. Mozilla announces that shortly before the release. But based on the past record, there will be some critical and high impact CVEs. Every Firefox upgrade is always a critical security update.

I hope you understand why I feel uneasy holding Firefox updates for longer than absolutely necessary. When there is a critical security issue in another component, no one in Fedora waits for it to be widely discussed etc., it gets fixed ASAP with reasonable measures to make sure it doesn't break things. Waiting until addons are updated and well tested? It may take weeks. There is no way to wait that long with critical CVEs on your neck.

That is of course a trade off on the security issues, but I have no idea how serious those are.

Just to give you an idea how serious they are:
Firefox 54 - fixing 4 critical, 11 high impact CVEs
Firefox 55 - 6 critical, 11 high
Firefox 56 - 3 critical, 6 high

I'm not so concerned about security here - the threat just after release is rather hypothetical as bad guys also need some time to investigate and deploy it.

I'm more concerned that we ship after other distros which is rather ironical with the "FIRST" and "leading edge" proclamations we make....maybe you guys mean "before RHEL" ? :)

That is exactly what I was thinking as well. When we push a stable kernel update, we leave auto-karma off because we want it to at least hit updates-testing (for current release we will get 3 karma before it ever gets there). After 24 hours in updates-testing, we push provided the karma is good. For a rebase though, we try to give it several days in updates-testing. This gives us more feedback and users more time to adjust.

I expect latest beta (which becomes release-candidate and candidate) may be available one or two weeks before final release. Mozilla usually spins that builds for minor/internal issues which do not affect us.

We can have the requested "testing phase" before and test that release candidate...oh, hold on, that's actually prohibited now :) Okay, good, we'll test after the final release!

I'm more concerned that we ship after other distros which is rather ironical with the "FIRST" and "leading edge" proclamations we make.

If we really wanted to be first for everything, we wouldn’t even have stable releases, only Rawhide. On the other hand, if our stated update policy were to be believed, we’d be using only ESR Firefox releases (https://fedoraproject.org/wiki/Updates_Policy#Philosophy).

Everyone wants to receive the newest, shiniest versions of the things that they’re interested in, but the most boring, stable versions of the things that they’re not. Okay, a web browser is a thing that lots of people are interested in, but still, we can’t please everyone.

People want the newest, shiniest version that works for them. If it breaks an addon they use and for which there is not yet an update/replacement, that's new, but it ain't shiny.

@stransky I wouldn't worry too much on the "leading edge" side of things as FF57 is in F27 ...

When we reach GA for F27 the leading edge users will be doing a system update almost immediately and will get that as a result.

The more conservative users will be holding back on F26 to wait for any issues other people report.

As a result, by waiting for a week or two after FF57 reaches GA upstream to push it stable, we'll serve both the leading edge users and the more conservative ones well.

For anyone still on F26 who do want to grab the leading edge they can use my COPR as we published this morning in Fedora Magazine.

On a related note, as this is a breaking change at the F27 release, we need to get it listed in the Release Notes (just that it has happened), Known Issues (some extensions will no longer be compatible) and ideally a, slightly belated, Change entered so that users and admins are fully aware of this when they do the upgrade from F26 to F27.

Let's ensure we get at least Release Notes and Known Issues in place prior to closing this issue off, if not the belated Change as well?

So, QA has asked us for clarification:

1504059 - punt (delay decision) - FESCo's intent is not clear to us, given the wording of their agreement and the lack of context in the meeting logs. we ask FESCo to clarify their intent on what should be done with FF57 in F27, given that it is not in stable but updates-testing and the maintainer would like to push it stable. does FESCo mean for this to be blocker, or FE, and what ff57 build(s) is it expecting when?

Some info:

  • We currently have firefox 57 beta 9 in f27-updates-testing, beta 10 just came out upstream
  • release date for firefox 57 final is Nov 14th.

I'd like to propose we ask QA to accept this as a FESCo blocker and have beta 9 pushed to stable.

If we do not do so, and f27 ships with firefox 56 and then updates to 57 a week or so later it will cause a great deal more disruption to our user base. We should get it pushed in asap so we can have testing with it before release.

I'm in favour of having a beta or RC in the final compose one way or another so F27 ships with FF57 directly, rather than a super temporary FF56 in the updates repo.

There's two reasons I have for this.

For the first I second Kevin's point about the disruption with such a major change to the UX soon after release.

For the second F27 beta users have the updates-testing repo enabled by default so any beta user who does an update will already be on FF57 meaning what is being tested by the beta community would not match what ships as the release.

The only caveat I'd add is to reiterate the need to highlight this as clearly as possible in the release notes and known issues ... and that there really should be a Change filed as admins and users do check the Changesets for a release for any major changes to be aware of... which I feel Firefox is this time round.

Edited a month ago by jhogarth

I kind of hate this choice. I guess we don't really have a lot of options other than "ship the prerelease", but I truly despise it. Unfortunately, the final release is too far out for us to slip the release to catch it.

So +1 to Kevin's proposal under protest.

I have similar feelings as @sgallagh, but I also +1 @kevin's proposal.

a month ago

Metadata Update from @sgallagh:
- Issue tagged with: meeting

AGREED: Fedora 27 will ship a pre-release of Firefox 57 so that we don't hit these woes on first upgrade after install (+6, 0, -0) (sgallagh, 16:51:12)

a month ago

Metadata Update from @sgallagh:
- Issue close_status updated to: Fixed

Login to comment on this ticket.