#1576 Evaluate Workstation graphical upgrade Change status
Closed None Opened 7 years ago by adamwill.

Sorry, I totally forgot to get this in before today's meeting :( My bad.

The Graphical System Upgrade F24 Change:

https://fedoraproject.org/wiki/Changes/GraphicalSystemUpgrades
https://bugzilla.redhat.com/show_bug.cgi?id=1308538

has not been evaluated at either of the two change checkpoints because the tracker bug was placed in the 'this is OK' state at both points. However, we (QA) think this was done incorrectly and the Change needs evaluation.

At checkpoint #1 (which was in February), the requirement is for the Change to "be substantially complete, and testable" and also "if a change is to be enabled by default, it must be so enabled at Change Completion deadline". I don't believe the Change met either of those requirements at the time; it was not testable so far as I know, nor was it 'enabled by default' in the sense that the code was not even in the F23 repositories in any way.

The second checkpoint is called the "100% Code Complete Deadline", and the clue is in the name. That was on April 19. I do not believe this was correctly met either. https://blogs.gnome.org/hughsie/2016/04/20/upgrading-fedora-23-to-24-using-gnome-software/ documents the state of this Change on 2016-04-20: it was testable by enabling a COPR and manually editing a JSON file, and there are notes clearly indicating the Change was not code complete, e.g. "I’ve not backported all the custom CSS for the upgrade banner just yet, but this should be working soon."

We don't see this situation as affecting the release criteria, exactly: as things stand this is still not technically a 'supported upgrade method' for Fedora, and the release criteria only cover those mechanisms. We see it as a case of a Change not meeting the deadlines of the Change process, so we would like FESCo to evaluate it and decide if you are still comfortable with the Change landing in F24 (i.e. for upgrades from F23 (and F22? not sure) to F24) and, if so, to perhaps come up with a broad plan for how this can be implemented between Beta and Final release.

I think what would be needed is for people to be able to enable upgrades to unstable releases via, perhaps, a simple dconf key, without needing an external package repository to get the code. Then we could document this process in a test case and as a sort of 'preview' section of the Upgrading wiki docs.

For QA's part we're also planning to run a Test Day on this topic between Beta and Final.

Thanks!


CCing Richard Hughes and several Workstation WG members.

What's the plan here, folks? Do you think we should bump this to Fedora 25 (well, F24->F25) and offer a COPR for F23->F24 upgrades this time around?

Upgrades are, by their very nature, hard to test before GA. Lack of graphical upgrades has been frequently cited as one of the big things holding back Fedora. Now that it is in reach, you want to defer it because we missed a deadline by a day ?!

Richard is out on a well-deserved vacation after working very hard to get the testability in place for the beta, so you won't hear from him before next week.

Replying to [comment:7 mclasen]:

Upgrades are, by their very nature, hard to test before GA. Lack of graphical upgrades has been frequently cited as one of the big things holding back Fedora. Now that it is in reach, you want to defer it because we missed a deadline by a day ?!

Richard is out on a well-deserved vacation after working very hard to get the testability in place for the beta, so you won't hear from him before next week.

I think the major concern is that we actually have the ability to test it during the run-up to GA. As Matthias pointed out to me on IRC, instructions exist at https://blogs.gnome.org/hughsie/2016/04/20/upgrading-fedora-23-to-24-using-gnome-software/. The confusion I think arose from the fact that Richard doesn't want to push the official packages to Fedora 23 until after GA. That's probably a very reasonable thing, but I think that was read as "This isn't testable before GA", which is not the case.

So I'm in favor of pushing ahead with this and deciding closer to GA if it's actually ready.

On idea I did suggest to Richard is to push the f23 upgrade before GA, but disable the upgrade functionality until we are confident that it will work we can then push another update that just enables the update functionality around or after GA. The disabling could be done in a way that can be overridden locally, for testing purposes (eg with an environment variable).

If that would help, we can look into it when Richard is back, next week.

I don't understand why we can't test upgrades prior to GA. dnf can obviously handle this just fine, or we would have few people using F24 right now.

Replying to [ticket:1576 adamwill]:

The second checkpoint is called the "100% Code Complete Deadline", and the clue is in the name. That was on April 19. I do not believe this was correctly met either. https://blogs.gnome.org/hughsie/2016/04/20/upgrading-fedora-23-to-24-using-gnome-software/ documents the state of this Change on 2016-04-20: it was testable by enabling a COPR and manually editing a JSON file, and there are notes clearly indicating the Change was not code complete, e.g. "I’ve not backported all the custom CSS for the upgrade banner just yet, but this should be working soon."

I agree, this did not go as smoothly as I would have hoped and the deadline was missed. I do not consider the feature testable in its current state; the requirement to enable a COPR is not reasonable at such a late stage in the game.

Still, I would argue this is the single most important feature we've added in... well, as long as I've been using Fedora. Graphical upgrades are essential for Fedora to be accessible to the ordinary user; we do not expect our users to know how to use a terminal, or to so much as know that a system upgrade is even available otherwise. This feature is exceptional and should be treated accordingly. It would be highly undesirable to delay it until F25.

We don't see this situation as affecting the release criteria, exactly: as things stand this is still not technically a 'supported upgrade method' for Fedora, and the release criteria only cover those mechanisms. We see it as a case of a Change not meeting the deadlines of the Change process, so we would like FESCo to evaluate it and decide if you are still comfortable with the Change landing in F24 (i.e. for upgrades from F23 (and F22? not sure) to F24) and, if so, to perhaps come up with a broad plan for how this can be implemented between Beta and Final release.

Hm, on the contrary, we always intended this to be the officially supported upgrade method for Workstation. It's unfortunate that this wasn't coordinated fully with QA, but it is essential for us that graphical upgrade work reliably. If users try to upgrade with dnf, I do hope that works reliably as well, but that is far less important for Workstation IMO.

Anyway, I think Adam's main concern here is that if we have not tested this feature thoroughly, and the clock is running down. If we do not test this properly, odds are very high that it is going to be broken in some serious way; I fear it's rather optimistic to just assume that everything will go well otherwise. Consider the number of complaints Ubuntu used to receive about computers being broken by Ubuntu upgrades. Right now, I suspect this feature has seen minimal testing, and unless it's tested on a large number of computers, I'm afraid it could break badly on release day and we will have some horrible PR mess on our hands. So while I don't want to delay the feature, we ''really'' ought to if there's no time available to test it thoroughly.

Replying to [comment:9 mclasen]:

On idea I did suggest to Richard is to push the f23 upgrade before GA, but disable the upgrade functionality until we are confident that it will work we can then push another update that just enables the update functionality around or after GA. The disabling could be done in a way that can be overridden locally, for testing purposes (eg with an environment variable).

If that would help, we can look into it when Richard is back, next week.

This is exactly what I was expecting all along. Better late than never. I propose we proceed with this plan. When that's ready, we should ask developers on our mailing lists to test the upgrade and report back on it ''regardless of whether it succeeded or failed.'' Then we can make a decision as to whether to proceed with this for F24 based on the feedback we get.

I'd love some really simple, straightforward instructions for testing which I could link to in the release announcement and blog posts next Tuesday.

I have mixed feelings about using a Copr for testing here. On the one hand, that limits the testing pool. On the other hand, if something does go wrong for a user without the level of command-line familiarity to do that, will they be able to recover (or even provide useful feedback)?

I think so too that we should put it in updates-testing for F23 to make testing easier; after all it makes sense to test it as thoroughly as possible, regardless if it's going to be a recommended method or not. In any case, I think we should definitely push it to F23, even if it's going to be an experimental method and DNF stays officially supported.

As for adding a gsettings key for making testing easier, I'm all for it -- I can work on this to add a gsetting such as "org.gnome.software show-upgrade-to-unstable" to enable upgrades to F24 before it's officially out, like adamw suggested above.

We can also completely disable autokarma so that it stays in updates-testing for a month or so, and only push it to stable right before F24 is getting out of the door. This also gives us more time to do testing; if it turns out it's not working as well as it should, we can leave the upgrade feature disabled.

I will be unable to attend the meeting today and so is hughsie, so it might be better to put it on next week's agenda instead.

Replying to [comment:10 catanzaro]:

I don't understand why we can't test upgrades prior to GA. dnf can obviously handle this just fine, or we would have few people using F24 right now.

Did dnf notify you that an F24 branch is now available ? Did it present you with a link to the release notes draped on the f24 background ? Did it warn you that some of your third party rpms might break during the upgrade ?

Of course it didn't.

And the reason it didn't is simple: Many of those things are simply not available yet. And that is why testing the graphical update feature before GA is hard: it relies on many pieces of data that we only produce shortly before GA time.

I agree, this did not go as smoothly as I would have hoped and the deadline was missed. I do not consider the feature testable in its current state; the requirement to enable a COPR is not reasonable at such a late stage in the game.

Fact is, there are testing instructions. Yes, they require enabling a copr. But if QA wants to test this feature, they can.

Anyway, I think Adam's main concern here is that if we have not tested this feature thoroughly, and the clock is running down. If we do not test this properly, odds are very high that it is going to be broken in some serious way; I fear it's rather optimistic to just assume that everything will go well otherwise. Consider the number of complaints Ubuntu used to receive about computers being broken by Ubuntu upgrades. Right now, I suspect this feature has seen minimal testing, and unless it's tested on a large number of computers, I'm afraid it could break badly on release day and we will have some horrible PR mess on our hands. So while I don't want to delay the feature, we ''really'' ought to if there's no time available to test it thoroughly.

This fear is exactly why Richard doesn't want to push the update into F23 now and have it springloaded to offer the graphical upgrade to all our users on release day.

Change should move forward. QA and FESCo and developers will work to find a acceptable testing process. (+5,0,0)

Replying to [comment:13 mclasen]:

Fact is, there are testing instructions. Yes, they require enabling a copr. But if QA wants to test this feature, they can.

I hope that just sounded more divisive than you meant. QA should be part of the Fedora Workstation Working Group and invested in this feature. Fedora should never have a "throw it over the wall and the other team can pick it up" structure.

In this case, I think we really want more testing than is possible from a handful of dedicated people in Workstation or on the QA team as a whole — that's why there's a push to get something testable into the wider beta user community.

This fear is exactly why Richard doesn't want to push the update into F23 now and have it springloaded to offer the graphical upgrade to all our users on release day.

So, if the decision comes down to something like:

"Fedora 23 is released today. New users can download install media and advanced users can use dnf system-upgrade today. In two weeks, we'll be rolling out a new feature which allows easy full-system upgrades from existing Fedora 23 Workstation systems — stay tuned."

that seems perfectly fine. I just need the story so I (and Fedora Marketing / Docs) can work on the messaging.

This ticket is still having meeting keyword and I found no further action plan was given. I saw that https://fedoraproject.org/wiki/Test_Day:2016-05-16_Workstation_Graphical_Upgrade has been done and it got a good response and bugs are getting tracked at https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=1308538

Shall we close this ticket now?

Well, it was discussed at a meeting with a fairly milquetoast agreement: "Change should move forward. QA and FESCo and developers will work to find a acceptable testing process." I would like us to have somewhere to track that, though. We ran a test day and found a bunch of bugs and I believe the current plan is to 'activate' graphical upgrades for regular users some time after F24 release (not at release time), but I don't know if there've been any details agreed on this plan.

AGREED: tenative plan is to have this change implemented on release day and update it monday with further details and revisit this ticket next meeting (6, 0, 0) (paragan, 16:31:25)

OK, here's the plan hughsie and I came up with:

1) Put it in F23 updates-testing this week with autokarma disabled (https://bodhi.fedoraproject.org/updates/FEDORA-2016-fad11727bf)

2) Keep fixing things until next week, do another release on Tuesday and put it in F23 updates-testing again

3) Evaluate how this is going at F24 go/no-go time and decide if it's ready to go to stable

4) If yes, submit to stable at F24 GA time and let the marketing team know that it's ready; otherwise keep it in updates-testing and keep iterating until it works well and submit it to stable a week or two after GA.

We discussed this in the FESCo meeting today and adamw and kparal are going to determine a list of blockersbugs for the graphical system upgrade feature; then we'll decide based on that next week whether to ship it at GA time or later.

Here's the latest status:

QA discussed this in their weekly meeting and their take was that they'd consider two more bugs as blockers for shipping this, https://bugzilla.redhat.com/show_bug.cgi?id=1335463 and https://bugzilla.redhat.com/show_bug.cgi?id=1336530

We're still figuring out the second one and would need a bit more time. However, coincidentally, it looks very much like the whole F24 release is about to be slipped as there's still no RC build, so I guess we are still shooting for having graphical updates ready by F24 GA time, just one week later.

Replying to [comment:21 kalev]:

Here's the latest status:

QA discussed this in their weekly meeting and their take was that they'd consider two more bugs as blockers for shipping this, https://bugzilla.redhat.com/show_bug.cgi?id=1335463 and https://bugzilla.redhat.com/show_bug.cgi?id=1336530

We're still figuring out the second one and would need a bit more time. However, coincidentally, it looks very much like the whole F24 release is about to be slipped as there's still no RC build, so I guess we are still shooting for having graphical updates ready by F24 GA time, just one week later.

I am fine with this plan, particularly since the release is slipping.

Does FESCo really need to do anything further at this point? It seems squarely in the hands of QA and the Gnome Software developer.

This was discussed into today's FESCo meeting, and we'll continue to keep an eye on things and hope that QA and the Gnome Software developers are able to make good progress on the outstanding issues.

I'll leave the "meeting" keyword on the ticket so that we can revisit next week.

Latest status update: we've got fixes for both of the blocker bugs and I'll do new F23 builds and put them to updates-testing tomorrow. I think we're a bit too late to ship it together with F24 GA though as I'd prefer for the new builds to sit in updates-testing for a few days before pushing them to stable to make sure they don't regress anything.

@kalev: Let me know when you're ready to push, since we have a Magazine article waiting in the wings to promote and explain the process to users.

How about the Magazine article goes forward noting that the tool will be in Fedora 23 updates-testing at GA, and if all goes well it will appear in stable by the following week?

Replying to [comment:27 chrismurphy]:

How about the Magazine article goes forward noting that the tool will be in Fedora 23 updates-testing at GA, and if all goes well it will appear in stable by the following week?
We're in the Magazine meeting now, and just scheduled this one to go out tomorrow as a "here's what's coming!". The word should start spreading tomorrow. :)

  • 1576 - Evaluate Workstation graphical upgrade Change status


    (sgallagh, 16:04:09)
  • AGREED: FESCo requires a GO decision from two established members of
    the QA team. (+6, 0, -0) (sgallagh, 16:20:10)

Quick update: we're still waiting for the next builds - the ones in https://bodhi.fedoraproject.org/updates/FEDORA-2016-fad11727bf at present do not address all the 'blocking' bugs. So this is not yet ready to go.

Beyond the two previously identified 'blocking' bugs we're also a bit worried about the 'PackageKit downloads are very slow' bug - that's https://bugzilla.redhat.com/show_bug.cgi?id=1336404 - for this, because it gives you a bit of a bad experience; unlike offline updates (where you're notified after package download is complete, so you don't notice that it's slow) with graphical upgrade you're notified before package download is done, so you see that the upgrade is available, get excited, click the button...then wait for hours for the download process to run. But we don't think it's an outright blocker, more a judgement call about whether we think it's OK to send the feature out with that bug still in place.

We'll let QA wrap up with maintainers on this topic.

All the serious bugs have been resolved and successfully verified in our testing. I and pschindl are giving GO for the feature. It's now up to the developers how long they want to wait for additional feedback (Bodhi, Bugzilla) before pushing this to F23 stable.

Thanks! I've submitted it to stable just now. I'm travelling and on PTO this week, but I'll be back next week and can fix things if anything new comes up.

Login to comment on this ticket.

Metadata