#258 Offline updates: install updates after the user triggers the shutdown/restart command, instead of installing them in the next boot
Opened 6 months ago by limao-doce. Modified 4 months ago

There are a few problems with the currently implementation:
- For those who use disk encryption, the system needs to be started two times and this is a bit annoying.
- When pressing the shutdown button from the GNOME Shell's top bar or from KDE's application launcher, the system shuts down without installing the updates. In the next startup, the computer won't be ready to use because the updates installation will be mandatory. Is way better to start the computer without waiting for it to be ready. GNOME Software does have an option to skip updates when clicking in the "Restart OS & Update" button, but it's too weird to open GNOME Software just to shutdown the computer. By the way, KDE Discover doesn't have this option and possibly others software centers too.

In a possible new implementation, the updates would be installed after the user initiated the shutdown command. Is possible to trigger Plymouth and then the package manager's update command? This would solve the problems above. Also, in order to not force the user to install updates at every shutdown, some option could be shown to ask the user if the updates installation should be initiated or not.

Restart

It would be from:
- Download updates > click in the "restart" button > type LUKS password in the next boot > install updates > restart > type LUKS password in the next boot > PC ready for use.

To:
- Download updates > click in the "restart" button > trigger Plymouth > install updates > restart > type LUKS password in the next boot > PC ready for use.

Shutdown

From:
- Download updates > check the "install pending software updates" and press on "shutdown" > restart > type LUKS password in the next boot > install updates > shutdown.

To:
- Download updates > check the "install pending software updates" and press on "shutdown" > trigger Plymouth > install updates > shutdown.


Since Fedora uses btrfs by default, why not take advantage of btrfs's snapshotting capabilities to provide zero downtime updates like what Silverblue already does?

@rhughes , are you able to comment? I don't remember how we ended up with the current behaviour.

We should be able to skip the initial reboot by using systemctl isolate, though it won't be as safe as if we reboot (imagine if PackageKit or the kernel is somehow in a bad state before updates begin).

IMO it's more important to get rid of the LUKS encryption password prompt by fixing #82 than to avoid one extra reboot. Reboots are cheap and harmless if we get rid of that.

Wouldn't excluding the system from encryption allow system files to be modified by using a Linux live USB?

Metadata Update from @aday:
- Issue tagged with: meeting

5 months ago

Wouldn't excluding the system from encryption allow system files to be modified by using a Linux live USB?

Yes, but things like authenticated filesystems with fsverity would resolve that problem. That's what @dcavalca, @salimma, and @borisb are working on.

Correction:

It would be from:

  • Download updates > restart > type LUKS password > install updates > shutdown.
  • Download updates > restart > type LUKS password > shutdown > next boot > install updates > restart > PC ready for use (when not using GNOME Software option)

Frankly I don't understand this corrected proposal either. LUKS password > shutdown, what...?

WG is interested in: download updates -> isolate, install updates -> shutdown or reboot. Action item: Michael to file an upstream GNOME Software ticket for this after the holidays.

Metadata Update from @catanzaro:
- Issue untagged with: meeting
- Issue tagged with: pending-action

5 months ago

I think the pending action here is @catanzaro to open a ticket upstream , to see what can be done there.

Correction:

It would be from:

  • Download updates > restart > type LUKS password > install updates > shutdown.
  • Download updates > restart > type LUKS password > shutdown > next boot > install updates > restart > PC ready for use (when not using GNOME Software option)

Frankly I don't understand this corrected proposal either. LUKS password > shutdown, what...?

WG is interested in: download updates -> isolate, install updates -> shutdown or reboot. Action item: Michael to file an upstream GNOME Software ticket for this after the holidays.

@catanzaro

Whoops. I edited the post now with the correct proposal. Thanks!

Frankly I don't understand this corrected proposal either. LUKS password > shutdown, what...?

WG is interested in: download updates -> isolate, install updates -> shutdown or reboot. Action item: Michael to file an upstream GNOME Software ticket for this after the holidays.

I'm not sure that GNOME Software is the right place for a ticket, actually. Maybe this should be reported against PackageKit? Hi @mcrha, do you know? The plan is to reboot only once rather than twice when installing updates. Instead of:

Reboot -> Enter LUKS passphrase -> Install updates -> Reboot again -> enter LUKS passphrase again

We would instead do:

systemctl isolate -> install updates -> Reboot -> enter LUKS passphrase

Should this change be proposed to GNOME Software or to PackageKit?

GNOME Software does not do offline updates, it only tells PackageKit to download them (and to prepare them) and then issues Restart, which the GNOME Shell can catch on and offer to "Install Updates" (which, when using "Turn Off", then does Restart->Install->Shutdown, which is really odd, but it does not belong here). What precisely the GNOME Shell does I do not know, but I suppose it also sets somehow the offline updates to happen, when the check is checked. There might be involved pk_offline_trigger_upgrade, I guess, which corresponds to the org.freedesktop.PackageKit.Offline TriggerUpgrade method.

I can't tell if it can be isolated from graphical.target. It might need systemd enhancement. man systemd.offline-updates logic item 2 suggests reboot is the way it currently works. @zbyszek ?

We should be able to skip the initial reboot by using systemctl isolate

Please don't base anything new on isolate. In the original design, systemd was conceived as mostly a dependency-based system. This meant that concepts like snapshots (returning the system to some previous version of the unit tree) and isolation (pruning the unit tree to exclude everything apart from some predetermined set) made sense. Very soon the inadequacy of those abstractions was apparent, and unit attributes like IgnoreOnIsolate were added. After a few years I managed to get rid of snapshots. Snapshots were completely unusable, and thus unused, and it was fairly easy to kill them. Unfortunately isolate is used to some extent, and it is still available.

The reason why isolation and snapshotting doesn't work for the unit tree is that we not only start units through dependency logic, but also in response to events (including user commands, user logins, failures of units, hardware events, timers). And to make things even murkier, we also start units as dependencies of external state: e.g. if a bluetooth device is plugged in, we start a whole dependency tree through bluetooth.target. When we move to a different top-level target, we don't want to stop those "extra" units. But isolation/snapshotting does not take this into account. There were various proposals to enhance how isolation was done (search for "softer isolate"), but it's just messy, and in some cases you want to stop various units, and in others not. In the end, we started using a different approach: we declare Conflicts between many different units as appropriate. So for example, anything that should be shut down before we switch back to the initramfs before shutdown has Conflicts=shutdown.target. Anything that should be stopped before we enter rescue mode has Conflicts=rescue.service. This is more granular and allows us to effectively have multiple overlapping replacements of isolate.

--

Overall, the way that we do shutdown is somewhat messy. Unfortunately KillUserProcess=yes is still not enabled by default, and there are also system services which (usually very unreasonably) insist on excluding themselves from systemd process management. This means that right now we can't properly guarantee that everything has been stopped cleanly even though the units have been stopped. Though when some process stays behind, this is really a bug, and we should just fix it. Doing upgrades during boot is easier because we mostly avoid the issue this way. If updates were done during shutdown, we should tighten the handling of stray processes during shutdown.

I think installing updates before shutdown is something worth exploring… Instead of using isolation, the upgrade job would have to be inserted somewhere into the dependency tree of shutdown.target. (Maybe this could even be done without touching any files on disk: just start an appropriate job with infinite JobTimeoutSec and JobRunningTimeoutSec and appropriate After/Before values?)

I think installing updates before shutdown is something worth exploring… Instead of using isolation, the upgrade job would have to be inserted somewhere into the dependency tree of shutdown.target.

Years ago, I had a proof-of-concept version of a PackageKit enhancement to do exactly what @limao-doce was asking for, but when I discussed it with Lennart at the time, he pointed out that part of the reason for requiring the reboot is to ensure the state of the mounted filesystems. If the user has modified their filesystem mount points in any place that the upgrade might touch, then it's no longer safe to do so.

Similarly, if any live kernel patching has occurred during this boot, it may lead to any number of subtle and difficult-to-diagnose issues.

So while I find the current approach to be really annoying with requiring me to re-enter my LUKS password twice, I think I have to agree with @catanzaro that the approach described in Lennart's blog from September is by far a better long-term solution.

part of the reason for requiring the reboot is to ensure the state of the mounted filesystems. If the user has modified their filesystem mount points in any place that the upgrade might touch, then it's no longer safe to do so.

I think this is a bit contrived. A user who has admin privileges can make any number of modifications to the system that cannot be fixed by a mere reboot. A user who overmounts part of the file system gets to keep the pieces if they try to do an upgrade on such a modified system. But I just don't think that's a very likely problem: a user who is smart enough to overmount parts of the filesystem in a way that let's the system continue without error is usually smart enough to understand the consequences.

Similarly, if any live kernel patching has occurred during this boot, it may lead to any number of subtle and difficult-to-diagnose issues.

Hmm, so the whole point of the live kernel patching is that you continue with a running kernel. "Subtle issues" would mean that the patching went wrong… In such a case, you're again more likely to have problems with other parts of the system. The upgrade itself uses a small subset of IO syscalls and is unlikely to be affected by those "subtle issues".

We discussed this ticket at yesterday's working group meeting. While we agree that the behaviour requested in this ticket would be a good thing, we don't currently see how it could be safely implemented. If anyone has any ideas or recommendations, we'd love to hear them!

Metadata Update from @aday:
- Issue untagged with: pending-action

4 months ago

My take is that it would be safe to do updates before shutdown, but I can't estimate how much systemd work is needed to get there. I agree with @zbyszek that it's worth exploring updates before shutdown. But equally worth exploring transactional updates.

The gist of it is similar to rpm-ostree, but with traditional rpm package management workflow. Snapshot the system, in a container run the update on the snapshot instead of the currently running system. If it fail or is interrupted for any reason, throw away the snapshot. If it succeeds, boot the snapshot. (It doesn't have to be Btrfs snapshots, could also use XFS reflink copies, et al.)

That's safer than either updating before or after shutdown, because we're not modifying the single copy of root we know is working, and risking failure in the middle of the update. Right now it means the user is left holding all the pieces.

The use case for this is broad. It benefits desktop, server, cloud with significantly less downtime, by permitting the update to happen concurrent to normal operation, isolated by cgroups resource control to ensure it doesn't negatively impact the primary workload. It can even be paused and resumed, as well as unceremoniously killed.

Unrelated, a few ideas have been floated for switching read-only /usr at runtime without a reboot. So we could possibly get to a point where reboots aren't needed at all, except for kernel updates - and maybe one day that too will be old news.

Login to comment on this ticket.

Metadata