#107 Reconsider updates policy
Opened 5 months ago by catanzaro. Modified 9 days ago

I'm getting tired of being prompted by GNOME Software to install updates every day after turning on my computer.

Some thoughts:

  • We should prompt to install updates when shutting down the computer, but this is broken.
  • We shouldn't prompt to install updates first thing in the morning.
  • If we're going to prompt the ser to install updates, it should only come after a few days of not having installed the updates with the power-off dialog.
  • We should check for updates less-frequently, once every week or so. Current logic is to check daily and notify daily if there is a security update, or weekly otherwise. But it seems there are too many security updates. I would change this to always check weekly at first, maybe with an eye towards even less-frequent checks.
  • By checking for updates less often, we should wind up with dnf users testing updates before they reach other users. In case of severe regressions, dnf users should notice immediately and hopefully the package maintainer can get a corrected update out before non-dnf users even receive the bad update. I think of this as "poor man's update packs" since update batching was unfortunately removed from bodhi.

We are thinking of reworking the update cadence for Silverblue as well. I've talked to Fedora QA here in Brno and they're not opposed, but first we really have to define everything - https://teams.fedoraproject.org/project/silverblue/epic/95

This is how it works right now:

Every hour, gnome-software wakes up. It looks at the system clock and if it's past 6 am, goes and downloads new update metadata. It rate limits itself to doing it once per day to avoid excessive downloads.

Once metadata is downloaded (doesn't matter if there was any change or not), it then proceeds to download any available updates.

Once updates are downloaded, it then goes and installs any flatpaks it downloaded.

Regular rpms are immediately prepared as an offline update and made available. Once that is done, gnome-software then looks at the set of available updates; if there are any security updates, then it pops up a notification to install them. If not, then it waits until a week has passed since the last update and only then pops up a notification to install offline updates.

@kalev
Every time I login (including merely logging out and immediately logging back in), PackageKit consumes 100% CPU for a minute, laptop fans kick in, and Software uses about 1/3 of that for a slightly shorter time. If Software could check once per day, rather than once per day per login, that'd be neat.

Could the downloading of packages could be delayed by ~15 minutes; when I do testing, very often they're throw away VMs. I can't change this behavior via gsettings fast enough. And it also blocks reboot/shutdown.

The last paragraph describes downloading RPMs, and possibly waiting a week to install them. On Fedora that likely means downloading RPMs that will become stale in a week's time. I take it the RPMs themselves contain the metadata to distinguish them as security updates vs regular updates, and that this information isn't in repo metadata?

The last paragraph describes downloading RPMs, and possibly waiting a week to install them. On Fedora that likely means downloading RPMs that will become stale in a week's time. I take it the RPMs themselves contain the metadata to distinguish them as security updates vs regular updates, and that this information isn't in repo metadata?

No, that information comes from the repo metadata. gnome-software downloads them just in case so that they are ready if the user should want to install them earlier before a week has passed.

The last paragraph describes downloading RPMs, and possibly waiting a week to install them. On Fedora that likely means downloading RPMs that will become stale in a week's time. I take it the RPMs themselves contain the metadata to distinguish them as security updates vs regular updates, and that this information isn't in repo metadata?

We are talking about a released Fedora though. How many rpms are revving on a more-than-weekly basis post GA? Should not be that common, really

We are talking about a released Fedora though. How many rpms are revving on a more-than-weekly basis post GA? Should not be that common, really

Should not be that common, indeed.

Still, I think it doesn't really matter. We don't need to preemptively download updates so that the user can update quicker if she manually decides to apply updates. For most users, manual checking for updates should be quite rare, and it's OK to wait a bit for downloads when that happens. I suspect we really just want to ensure we download all updates before proposing a prepared update, and that only needs to happen weekly. Downloading the updates early annoys frequent dnf users, and the benefit of doing so is very limited, so might as well wait until we are actually going to prepare and propose an update before doing so. Fair enough?

Regular rpms are immediately prepared as an offline update and made available. Once that is done, gnome-software then looks at the set of available updates; if there are any security updates, then it pops up a notification to install them. If not, then it waits until a week has passed since the last update and only then pops up a notification to install offline updates.

Thanks for this summary.

We've previously established that we have security updates almost every day, and this is much too fast. It is extraordinarily rare for security updates to fix vulnerabilities that are being actively exploited, and such fixes should be marked urgent.

Proposal 1: Wait until a week has passed to prepare updates (including downloading updates), unless the updates are urgent priority. Do not treat security updates specially. Urgent updates should be marked urgent.

Proposal 1a): As proposal 1, except two weeks instead of one week. (I split this out separately since I expect it will be controversial, but I really do want to slow things down a bit more.)

Proposal 2: No notifications until one day after an update has been prepared. The user should have the opportunity to apply updates at power off or reboot time, when it's less inconvenient, before being pestered with notifications.

Proposal 3: Figure out a technical solution to the bug preventing the power off dialog from displaying updates reliably.

We don't need to preemptively download updates so that the user can update quicker if she manually decides to apply updates. For most users, manual checking for updates should be quite rare, and it's OK to wait a bit for downloads when that happens. I suspect we really just want to ensure we download all updates before proposing a prepared update, and that only needs to happen weekly. Downloading the updates early annoys frequent dnf users, and the benefit of doing so is very limited, so might as well wait until we are actually going to prepare and propose an update before doing so. Fair enough?

No.

How do you suggest doing that ? We propose an offline update on the logout dialog. That is not the time to go off downloading the updates. We may be offline, etc, the user may want to reboot quickly, etc etc.

Not making the user wait, ever, has been one of the core design principles behind g-s, from the very beginning. We haven't been entirely successful at that, but I don't think we should give that principle up now.

I think our current behavior is already basically following what you want to achieve - we prepare an offline update early, so users get a chance to install it at the 'less inconvenient' time every day (the poweroff dialog), and we don't pester them with notifications until a week has passed.

The one proposal that seems worth discussing is recalibrating what we consider security updates that are worth notifying about.

We are talking about a released Fedora though. How many rpms are revving on a more-than-weekly basis post GA? Should not be that common, really

If I install Fedora 31 today, reboot, go through g-i-s, immediately over 1GiB's worth of updates start downloading and I cannot stop it without a forced reboot. A normal reboot is held up with a systemd wait job on PK (downloading metadata or RPMs or both, not sure). And 4 out of 5 times I'm doing throw away installs, I don't need those updates. And also during beta, it's similar because u-t is enabled but we're also in freeze, so u-t is accumulating updates that need to be applied by default.

These are edge cases, and QA folks are used to it, but I also do throw away Windows and macOS installs and don't run into such an aggressive need to immediately download updates. I'd say UX wise, GNOME Software updates are almost as nice as macOS except for the CPU usage and inhibited reboot. It's way better than Windows updates, which I find exceptionally irritating.

Perhaps instead of a delay downloading the RPMs, if PK could self-throttle CPU and bandwidth usage for this task? Making my laptop hot on every login is just not a good UX. And if it could be interruptible so I can reboot now without having to do a force power off that'd be nice also. I don't think updates need to be available so quickly. Windows and macOS, the first update after a clean install isn't available for maybe an hour (not least of which is that it takes that long to download their incredibly massive updates.)

No.
How do you suggest doing that ? We propose an offline update on the logout dialog. That is not the time to go off downloading the updates. We may be offline, etc, the user may want to reboot quickly, etc etc.

No no, that's not what I meant. Of course I agree we should never propose an update before all packages have been downloaded.

According to Kalev's explanation, Software currently downloads updates daily even if it's not scheduled to propose an update for another week. There's not much benefit to doing that as it only saves time for the user if the user manually runs a check for updates in GNOME Software at a time when the user has not been prompted to install a prepared update. Correct? That's really just a tweak to our current behavior, not a major change and not even an essential change. The benefit would be to hopefully reduce the number of complaints we receive from 'dnf update' users. Fairly or not, it's GNOME Software that gets blamed for performing duplicate downloads and not sharing a package cache with dnf; for whatever reason, nobody seems to blame dnf for not sharing with GNOME Software....

Then maybe I misunderstood this point: "Regular rpms are immediately prepared as an offline update and made available." Does "made available" mean the update is immediately presented on the power off dialog, without first waiting for a week since the previous update? In that case, users will still wind up with daily proposed updates when powering off, just not daily notifications. My suggestion is to propose updates no more often than weekly (or biweekly). Then wait some additional time (suggestion: one day, could be even more) before nagging the user with a notification.

The benefit would be to hopefully reduce the number of complaints we receive from 'dnf update' users. Fairly or not, it's GNOME Software that gets blamed for performing duplicate downloads and not sharing a package cache with dnf; for whatever reason, nobody seems to blame dnf for not sharing with GNOME Software....

The cache is not shared because dnf doesn't have a supported api for doing so. I don't think we should accept the blame.

Does "made available" mean the update is immediately presented on the power off dialog,

That is my understanding (modulo the current gnome-shell bug). Kalev, correct me if thats wrong.

Metadata Update from @chrismurphy:
- Issue untagged with: meeting

2 months ago

@churchyard has started: RFC: Security policy adjustments to make it easier to implement and more friendly to maintainers

While that's not directly related to this issue, if there's anything policy, infra, or RPM related that we want or need to better handle this, now might be a good time to take it into account.

The benefit would be to hopefully reduce the number of complaints we receive from 'dnf update' users. Fairly or not, it's GNOME Software that gets blamed for performing duplicate downloads and not sharing a package cache with dnf; for whatever reason, nobody seems to blame dnf for not sharing with GNOME Software....

The cache is not shared because dnf doesn't have a supported api for doing so. I don't think we should accept the blame.

Actually, you should accept the blame, as the fault mostly lies with the PackageKit developers and the DNF team not working together specifically to fix this problem. As the GNOME Software developers are also the PackageKit developers, this is partly their fault. There is also blame to be had with the DNF team, as they have not tried to rectify this either.

As for why nobody blames DNF, it's because it's not visibly DNF's fault. DNF is the package manager. Things are supposed to integrate with it, not the other way around. And all other interfaces for DNF use the shared cache in /var/cache/dnf: dnfdragora, simple-dnf, dnf-automatic, and of course dnf itself. If every frontend to libdnf actually used their own caches, then it might be a different story.

But technically the PackageKit backend could be configured to use the same cache as every other libdnf frontend. The main problem is that we don't have an API to "lock" the package manager to a single instance, as far as I know. @jmracek or @dmach might know more there.

@mclasen the DNF API for sharing the cache exists - it's the cachedir option.
PackageKit never used it. The required change needs to be made in PackageKit, libdnf or both. If you're interested in switching PackageKit to using cachedir, let us know. Please note that we'll need a commitment from your team to test PackageKit properly, debug any new issues and help us fix them in libdnf. As @ngompa suggested, it will also require proper locking and we need to make sure that running dnf clean all doesn't crash PackageKit etc.

To be honest, we are not planning any PackageKit improvements because it's a dead project for about a year.

The DNF team is planning to stop supporting the libdnf API (in upstream, support in Fedora stays for a while) used by PackageKit shortly and we'll work on a new daemon replacing PackageKit. This new daemon will share cache with DNF. I'm going to send an announcement with more details to fedora-devel in about a week.

I made a note for myself to review this issue and design an update policy for the new daemon.

Does GNOME Software have a way of acting on severity=urgent differently from any of the other severity levels? (high, medium, low)
https://github.com/fedora-infra/bodhi/blob/develop/bodhi/client/bindings.py#L262

And are package severity levels being reliably set? Are the updates tagged urgent really urgent, or are they rather high severity?

Does GNOME Software have a way of acting on severity=urgent differently from any of the other severity levels? (high, medium, low)
https://github.com/fedora-infra/bodhi/blob/develop/bodhi/client/bindings.py#L262

Yes, the data is there, but PackageKit doesn't support different security levels in its current API, so gnome-software doesn't know anything about them.

Metadata Update from @chrismurphy:
- Issue tagged with: meeting

11 days ago

@mclasen the DNF API for sharing the cache exists - it's the cachedir option.
PackageKit never used it. The required change needs to be made in PackageKit, libdnf or both. If you're interested in switching PackageKit to using cachedir, let us know. Please note that we'll need a commitment from your team to test PackageKit properly, debug any new issues and help us fix them in libdnf. As @ngompa suggested, it will also require proper locking and we need to make sure that running dnf clean all doesn't crash PackageKit etc.
To be honest, we are not planning any PackageKit improvements because it's a dead project for about a year.

Yes, too late now for PackageKit. Back when PackageKit was actively developed, there was no api to share the cache, afaik.

The DNF team is planning to stop supporting the libdnf API (in upstream, support in Fedora stays for a while) used by PackageKit shortly and we'll work on a new daemon replacing PackageKit. This new daemon will share cache with DNF. I'm going to send an announcement with more details to fedora-devel in about a week.
I made a note for myself to review this issue and design an update policy for the new daemon.

It would be good to make sure that the new daemon supports all the PackageKit use cases. In particular, the offline update dance is complicated and will requires attention to avoid regressions.

General agreement to reduce the frequency of non-urgent updates to not more than once per week (TBD); and there needs to be a reliable mechanism to indicate truly "urgent" severity updates (possibly entailing automation such that Bodhi requires a bugzilla with CVE reference, to consider an update urgent, otherwise it's demoted to "medium" severity).

ack/nack/patch?

@mattdm is on vaca this week, so tentatively push this issue back to 03 March to give him some time to dig out upon return.

Metadata Update from @chrismurphy:
- Issue untagged with: meeting

9 days ago

My vote: all non-urgent updates get batched every two weeks. (Weekly is still a bit much!) Start with client-level batching in gnome-software, because that's what we can do today. Ideally in a better future, bodhi would learn to batch updates itself once again, so we can have lightly-tested "update packs" that get sanity-checked by OpenQA.

The definition of what makes a security update urgent is a bit hazy. In general, I'd say only critical-severity CVEs are urgent, or important CVEs actively exploited in the wild (very rare), or CVEs that are receiving special media attention (such that the CVE is an urgent PR issue). I'd also say the update is not urgent if mitigated by a sandbox (so e.g. critical CVEs in Firefox are not urgent, unless the CVE is a sandbox escape). I suggest we trust maintainers to set the priority level of their updates appropriately, using their best judgment, and in accordance with considerations that we document in the Fedora updates policy (which might look like what I just wrote). Abuse can be reported to FESCo to be handled on a case-by-case basis.

None of the above applies to flatpaks. Flatpaks can be updated automatically at any time. In the future, we might want to selectively begin shipping some default apps as flatpaks, e.g. Firefox. (But probably not before we have debuginfo for Fedora flatpaks, and ABRT integration.)

Is there a useful way to informally parse repo metadata over the next 3-7 days, for "urgent" severity updates? Would this help estimate whether this designation might be useful or problematic?

None of the above applies to flatpaks. Flatpaks can be updated automatically at any time. In the future, we might want to selectively begin shipping some default apps as flatpaks, e.g. Firefox. (But probably not before we have debuginfo for Fedora flatpaks, and ABRT integration.)

That's probably not smart to make that assumption. Flatpaks are just as dangerous given that they can easily destroy user data with the sandbox. That's what people care about, and you're suggesting that doesn't matter, when it probably matters more to people.

General agreement to reduce the frequency of non-urgent updates to not more than once per week (TBD); and there needs to be a reliable mechanism to indicate truly "urgent" severity updates (possibly entailing automation such that Bodhi requires a bugzilla with CVE reference, to consider an update urgent, otherwise it's demoted to "medium" severity).
ack/nack/patch?

One point that was raised yesterday: urgent doesn't necessarily mean "security issue of a certain severity". That could potentially invalidate the "requires a bugzilla with a CVE ref" approach.

The main difficulty in all this is the word "reliable". We need to be realistic in our expectations for what can be achieved while individual maintainers are in control of what gets updated when.

If updates that genuinely need to be pushed out urgently are rare, wouldn't it be possible to have some kind of centralised process for that? If the update needs to go out urgently, the maintainer has to ask person foo and they push a special button.

There is a proposal for updates policy on Fedora Silverblue, most of it can be adapted even for "vanilla" Workstation.

https://docs.google.com/document/d/1gz1inA8e_LQP-dO751bkpS_ZVVLFAqYwFiRlqKJ3G9g/edit?usp=sharing

Comments welcome :)

Login to comment on this ticket.

Metadata