#97 always update the bootloader on major upgrades
Closed: Deferred to upstream 2 years ago by chrismurphy. Opened 2 years ago by chrismurphy.

Background:

On x86 BIOS, the bootloader is initially installed by anaconda with grub2-install and that bootloader "blob" is never automatically updated again. Thus, over time, it becomes stale even as the GRUB package is updated through the normal mechanism, since grub2-install isn't ever run as part of any update or upgrade process.

Meanwhile on x86 UEFI, the bootloader binaries: shim$(arch).efi grub$(arch).efi are built by Fedora and included in their RPMs. When the package is updated, even within a release, the bootloader files are copied to /boot/efi/EFI/fedora/ and therefore it is updated.

(Currently on Silverblue, the bootloader is not updated on either x86 BIOS or UEFI.)

The problem:

On x86 BIOS systems, bug and security fixes related to the bootloader aren't really deployed until 'grub2-install' is manually executed by the user. This resulted in problems during the Fedora 30 release cycle with users having a bootloader at least as old as Fedora 20.
https://fedoraproject.org/wiki/Common_F30_bugs#blscfg-fail

And actually there were other problems due to stale bootloader, but Javier was able to work around them. Nevertheless, we really need to take responsibility for updating the bootloader regularly, and not allow it to go stale like this.

The other problem:

At least on x86 BIOS, it's ambiguous whether the bootloader is Fedora's, or something custom. If a Fedora upgrade just does grub2-install for everyone, users with custom bootloaders will have their bootloader replaced with Fedora's, and it'll cause them some grief. I can't estimate how many users might be affected, but it's reasonable the overwhelming majority use and depend on the Fedora bootloader installed by default by anaconda.

Implementation:

Option A: offer an opt out from an automatic grub2-install to update the bootloader. Perhaps a key/value pair in /etc/default/grub? Or maybe suggest they uninstall the Fedora grub2 package?

Option B: only do this bootloader update at major upgrade time (when the release version changes), not just any time the GRUB package is updated within a release cycle.

Option C: both an opt out and only update the bootloader when upgrading.

Other ideas?

Arguments:

The Workstation PRD says two things about upgrades:
should give a result that is the same as an original install
should be a safe...never leaves the system needing manual intervention

That's not a policy or a requirement, but rather it's an idealized goal. And also, it's really hard to make certain no one ever needs to manually intervene, when there are competing interests. If a Fedora upgrade steps on a custom bootloader, that person will need to manually intervene to restore their custom setup. But the alternative, leaving the bootloader untouched, means the vast majority have an increasingly stale bootloader, and to avoid it today they need to manually update the bootloader by calling grub2-install.

My position is, the WG should vote that the Fedora bootloader is part of the product Fedora delivers, and that Fedora is responsible for keeping it reasonably up-to-date.

And then after that commitment happens with a vote, we can discuss the implementation options with GRUB developers, and also get more feedback from devel@ on as yet unseen liabilities for such a change.


Metadata Update from @chrismurphy:
- Issue set to the milestone: Future Release

2 years ago

Does anyone know if there is way to trigger a script to run only during dnf system-upgrade or a GNOME Software upgrade, not have that script dragged in during a regular dnf update or the GNOME Software equivalent?

javierm says on IRC

javier__ for every grub2 pkg upgrade we can do it in a %posttran scriptlet but I don't know how it can be done only for major upgrades

Background:
On x86 BIOS, the bootloader is initially installed by anaconda with grub2-install and that bootloader "blob" is never automatically updated again. Thus, over time, it becomes stale even as the GRUB package is updated through the normal mechanism, since grub2-install isn't ever run as part of any update or upgrade process.

Yes, this is also a problem on ppc64le with OpenFirmware since grub2 is also used there. For ppc64le with OPAL isn't a problem because the bootloader is part of the firmware and not owned by Fedora.

Meanwhile on x86 UEFI, the bootloader binaries: shim$(arch).efi grub$(arch).efi are built by Fedora and included in their RPMs. When the package is updated, even within a release, the bootloader files are copied to /boot/efi/EFI/fedora/ and therefore it is updated.
(Currently on Silverblue, the bootloader is not updated on either x86 BIOS or UEFI.)

Indeed, that's because 'ostree' doesn't update the ESP as a part of the deployment transaction. I've created a pull-request to get the ball rolling. But that's a separate topic since has to be handled by ostree / rpm-ostree.

The problem:
On x86 BIOS systems, bug and security fixes related to the bootloader aren't really deployed until 'grub2-install' is manually executed by the user. This resulted in problems during the

I looked at the dist-git log to understand why grub2-install isn't executed on a package update. When the package was originally imported in commit:

0e50f1f9a3c Initial import.

grub2-install was executed in a %post scriptlet. But the primary goal of that seemed to be to upgrade from legacy grub to grub2 rather than to update to a newer grub2 version on package update.

There was also %preun scriptlet that was meant to remove all the modules from the legacy grub installation but it seems that it also removed the ones from the grub2 install. And it was that scriptlet that seemed to cause most issues for users when grub2 was made the default bootloader in Fedora 16. The filed bug was:

735259 - after updating grub2, booting drops me to grub rescue prompt.

Another problem seemed to be that grub2-install was called with --grub-setup=/bin/true, so only the core.img was updated without updating the MBR which also cause issues due incompatibilities. The mentioned workaround for both issues was to run 'grub2-install /dev/sda' after booting the machine.

The issue about files being removed was fixed by @kalev in commit:

eb5089f407d Fix upgrades from grub2 < 1.99-4 (#735259).

And the second issue of grub2 core.img being updated without updating the MBR was fixed by @pjones in commit:

5c9195d6908 Don't do sysadminny things in %preun or %post ever. (#735259)

The %preun and %post scriptlets were removed and since then grub2-install is never called anymore on package updates.

There were some users in that bug complaining that they had installed grub in a partition instead of the MBR and that this didn't work anymore and others saying that the pkg should never update the bootloader but most of the complains are about the two mentioned bugs.

Fedora 30 release cycle with users having a bootloader at least as old as Fedora 20.
https://fedoraproject.org/wiki/Common_F30_bugs#blscfg-fail
And actually there were other problems due to stale bootloader, but Javier was able to work around them. Nevertheless, we really need to take responsibility for updating the bootloader regularly, and not allow it to go stale like this.

Agreed.

The other problem:
At least on x86 BIOS, it's ambiguous whether the bootloader is Fedora's, or something custom. If a Fedora upgrade just does grub2-install for everyone, users with custom bootloaders will have their bootloader replaced with Fedora's, and it'll cause them some grief. I can't estimate how many users might be affected, but it's reasonable the overwhelming majority use and depend on the Fedora bootloader installed by default by anaconda.

I think that's the crux of the issue. As you said we can't know if the bootloader is owned by the OS that's updating the grub2 package or any other installed OS.

We should investigate if a version could be added to the core.img so it can be used to figure out if the installed grub version belongs to the OS updating grub or not. That version should also contain the machine ID so the OS known that owns the bootloader and ignore if that's not the case.

This means that only new installations that have a grub that added this version field will be able to update grub, but I think that's OK and users that want to opt-in could do it with an explicit grub2-install.

Implementation:
Option A: offer an opt out from an automatic grub2-install to update the bootloader. Perhaps a key/value pair in /etc/default/grub? Or maybe suggest they uninstall the Fedora grub2 package?

Agreed, rather than abusing /etc/default/grub I think that we should add a /etc/sysconfig/bootloader similar to the one we have to control how kernels are installed (/etc/sysconfig/kernel).

That file could also be used be used to explicitly define what bootloader is used (i.e: bootctl install could set the bootloader to sd-boot and installing grub2-efi set it to grub).

Option B: only do this bootloader update at major upgrade time (when the release version changes), not just any time the GRUB package is updated within a release cycle.

I like this option, but as you mentioned I don't know if there is a way to hook that action to a system-upgrade.

Option C: both an opt out and only update the bootloader when upgrading.

I think this option is also OK. We already do that for EFI anyways so legacy BIOS will be consistent with that.

Metadata Update from @chrismurphy:
- Issue tagged with: meeting

2 years ago

Metadata Update from @chrismurphy:
- Issue untagged with: meeting

2 years ago

This was discussed at this week's meeting.
https://meetbot.fedoraproject.org/teams/workstation/workstation.2019-07-15-13.00.log.html

Re: the question about how to make grub2-install only happen on major upgrades rather than same release minor updates, there's more discussion in the full long but here are a few excerpts:

13:41:01 <kalev> something like a drop in directory, /usr/lib/upgrades.d with scripts to run
...
13:42:02 <sgallagh> Doesn't need to be a dnf/pk change; we just need to have triggers in place.
13:42:08 <sgallagh> RPM would handle that fine

Part A is the trigger mechanism, and Part B is the when to trigger. And Part B has two parts:
- Fedora policy, i.e. major upgrades rather than updates. But also bootloader devs plausibly might want to "push" a bootloader reinstall on demand for urgent bug fixes even
- User opt in/out

Working group probably doesn't need to track this: (a) it'll happen when it happens (b) we'll need to defer to experts in this area anyway, as the issue and solution pertains to low-level function, and security related issues.

The most recent concept being explored is being tracked here: Design tooling to ship bootloader updates #510

Metadata Update from @chrismurphy:
- Issue close_status updated to: Deferred to upstream
- Issue status updated to: Closed (was: Open)

2 years ago

Login to comment on this ticket.

Metadata