This is unreasonable. Without /var and without /usr, we are left with almost nothing. We use /var for persistent local and shared state in scriptlets. You should reconsider making /var RO during composes.

Hi Neal, thanks for looking!

We use /var for persistent local and shared state in scriptlets.

This is accounted for in the proposal (see lines 7 and 8 of the diff).

This looks good to me overall, but it could probably use some examples. https://bugzilla.redhat.com/show_bug.cgi?id=1657041 is one.

Hi Neal, thanks for looking!

We use /var for persistent local and shared state in scriptlets.

This is accounted for in the proposal (see lines 7 and 8 of the diff).

I'm saying it's unreasonable to restrict it to just those locations. If an application has state that needs to be modified for any reason anywhere in /var, it needs to be permitted. For example, if display manager "state" needs to be modified on upgrade (to handle desktop session configuration transitions, for example), then it has to be possible. Fedora KDE did this in F34 and other desktops will do this over time for various reasons (mostly around Wayland transitions).

If an application has state that needs to be modified for any reason anywhere in /var, it needs to be permitted.

Of course it's permitted, it's just done by executing code as part of the service/app, not via %post.

If an application has state that needs to be modified for any reason anywhere in /var, it needs to be permitted.

Of course it's permitted, it's just done by executing code as part of the service/app, not via %post.

We do not have the ability to demand applications to change their architectures. So, no, this is not happening.

We do not have the ability to demand applications to change their architectures.

The proposal mentions this: if the application/service itself cannot carry out the migration, then a systemd unit can be shipped to do so.

Speaking at a very general level, migration code run in a %post script could just as well run in a systemd unit. On traditional systems, the systemd unit would run when the service is restarted. On an image-based system, it would run on the next boot. Obviously, the best course of action depends on the particular case. The guideline can only present the issue and provide some common approaches to solve it.

To provide more context on this: this proposal is essentially documenting the status quo, at least for any package currently shipping in any OSTree-based variant today. Scriptlets in all those packages must already account for these limitations. The goal of having it in the packaging guidelines is to raise awareness and in the process reduce the likelihood of discovering compatibility issues for packagers, composers, and end-users.

You realize that by forcing systemd units, we are forcing people to have to go through a whole secondary process to deal with things like presets, right? That's a lot of extra steps for stuff like this. We do not allow as a matter policy people forcibly enabling systemd units in scripts unless specifically approved by FPC or FESCo as an exception.

That's why your solution is unacceptable and you need to rethink the limitations you're imposing on your system.

We do not allow as a matter policy people forcibly enabling systemd units in scripts unless specifically approved by FPC or FESCo as an exception.

There's a lot of irony though in having zero approval process for arbitrary code run as root on the build servers and client machines in %post.

AFAIK the rationale behind presets is mainly to control socket/network accessible services; I don't think we should have similar constraints for "operating system implementation details".

If it comes down to it I could imagine adding something to rpm which is like
%clientpost
which is explicitly defined to run on the end machine. But I dunno, it still seems like a bad pattern to encourage versus having transition code run as part of the existing codebase.

In the case you mentioned

For example, if display manager "state" needs to be modified on upgrade (to handle desktop session configuration transitions, for example),

Isn't there already a systemd unit in kdm.service or whatever that could contain the relevant code?

Honestly, this committee's role in things is to say whether these guidelines look good assuming that FESCo approves the feature request. Whether we agree with the feature itself isn't really the point here, and I don't think this is the right place for that discussion. FPC doesn't get a veto if FESCo approves something.

We do not allow as a matter policy people forcibly enabling systemd units in scripts unless specifically approved by FPC or FESCo as an exception.

I think exceptions should be allowed for migration cases since they are not long-running services and are purely in service of another system service (pun intended). Another way to model this actually is to put the migration service unit into the .requires/ of the target service. That's better even than presets since it only runs the migration code if the service was actually in use in the first place.

We do not have the ability to demand applications to change their architectures. So, no, this is not happening.

It's not about demanding. The role of this issue is leading the larger "we" in improving the FOSS ecosystem. See for example https://github.com/linux-nvme/nvme-stas/issues/130 where we successfully converted the nvme tooling to use a systemd unit instead of a package script.

We do not have the ability to demand applications to change their architectures.

The proposal mentions this: if the application/service itself cannot carry out the migration, then a systemd unit can be shipped to do so.

Speaking at a very general level, migration code run in a %post script could just as well run in a systemd unit. On traditional systems, the systemd unit would run when the service is restarted. On an image-based system, it would run on the next boot. Obviously, the best course of action depends on the particular case. The guideline can only present the issue and provide some common approaches to solve it.

Would this not be burdensome for applications that might not even be meant to run as a service, but do put files in /var ?

Also this seems to assume the runtime environment has systemd, which IIRC won't be true for containers.

Also this seems to assume the runtime environment has systemd, which IIRC won't be true for containers.

Some containers do run systemd, but yeah most don't. But actually the larger issue with containers is that one usually does not update a long-living container. Instead it's nuked and replaced by a new built image (Flatpaks even more so). So %post scripts don't have a chance to run at all there to migrate data volumes mounted in.

In those cases, the service/app kinda already has to be able to deal with these upgrade situations seamlessly and not rely on distros handling it in scriptlets (or of course, they're free to e.g. request users to manually run some tool or to nuke the data volumes, etc... -- the main point is that this is already an issue today and not specific to OSTree).

This is only true for OCI "app" containers, and not true for OCI "toolbox" containers. nor is it true for LXC or nspawn containers, nor is it true for VM workloads or regular user workloads.

This issue is specific to RPM-OSTree workloads, just like how RPM-OSTree still to this day cannot handle Lua scriptlets.

This is only true for OCI "app" containers, and not true for OCI "toolbox" containers.

It's not common for pet containers to run long-running services that manage persistent state.

Also, many people (including me) do in fact nuke and recreate pet containers routinely on top of newer images. Both toolbox and distrobox document this approach.

nor is it true for VM workloads or regular user workloads.

I suppose this assumes those systems don't do transactional updates, which brings us back to this ticket. :)

Note openSUSE packaging guidelines have similar wording to what is being proposed here:
- https://en.opensuse.org/openSUSE:Packaging_guidelines#Migration_.2F_Upgrades
- https://en.opensuse.org/openSUSE:Packaging_for_transactional-updates#Files_in_/var

The MariaDB example there have an analogue in their upstream container docs: see the MARIADB_AUTO_UPGRADE variable to automatically run migration code on container startup: https://mariadb.com/kb/en/mariadb-server-docker-official-image-environment-variables/#mariadb_auto_upgrade-mariadb_disable_upgrade_backup.

Both of these are examples in transactional update systems that are not OSTree that must also deal with state in /var.

I'm in two minds on this.

FPC is generally more about saying what is best/common practice and having words that explain it to people who aren't familiar. In some cases "new" things happen and we have to provide guidance, in very rare cases things change and we have to tell people to stop doing certain things how they did (Eg. /usr move).

This doesn't feel like any of that.

On the other hand, I personally would love it if we moved closer to a world where %post didn't exist and I think long term that's where we should be preparing to go ... and I know I'm not alone in both thoughts.

Four thoughts come to mind, in rough order of joy:

1. As part of ostree/bifrost/whatever projects create a systemd "post upgrade running environment" service. Basically it would run things "immediately" after the rpm transaction in a traditional environment, but otherwise it would run at first boot in a container/ostree/etc. environment. With only a small amount of thought this service would have a way to drop a script into a directory somewhere, and that script/binary only runs once ... but also some mechanism to refresh (so it will run once again, even if the script didn't change). Having the migration from using %post to using this service should be as nice as possible.

2. Change the wording so that it only applies to changes within a release. Also maybe don't just outright ban reading from /var, make it obvious what reading will work and what will have problems.

3. "not really a solution": Change the wording so that if a package needed to do this then they'd had to mark the package in some way (Provides: upgrade-dirty(/var)). Again, it doesn't solve it, but it hopefully helps makes it easier to see where the problems are.

4. "political dumpster fire": Go argue with FESCO and the community that we need to do this.

On the other hand, I personally would love it if we moved closer to a world where %post didn't exist and I think long term that's where we should be preparing to go ... and I know I'm not alone in both thoughts.

:heart:

1. As part of ostree/bifrost/whatever projects create a systemd "post upgrade running environment" service. Basically it would run things "immediately" after the rpm transaction in a traditional environment, but otherwise it would run at first boot in a container/ostree/etc. environment. With only a small amount of thought this service would have a way to drop a script into a directory somewhere, and that script/binary only runs once ... but also some mechanism to refresh (so it will run once again, even if the script didn't change). Having the migration from using %post to using this service should be as nice as possible.

This is similar to Colin's idea in https://pagure.io/packaging-committee/pull-request/1333#comment-198470 I think (i.e. trying to provide some abstraction to make it just work in both cases).

I think we could do that if it helps with adoption. OTOH, I do worry that the major difference in implementation details might add confusion (vs. if it's a bona fide systemd unit, then it's easier to explain: it gets run immediately on traditional systems, but on reboot in transactional systems). For services that could also be containerized, it would still be preferable if the service itself can take care of this sort of migration.

Is the next step for this now to create a Fedora 41 Change Proposal? We'll definitely want to make sure to capture some of the discussions we've had here.

I noticed this was tagged with the meeting label. Has this been discussed yet? Interested in making progress on this.

I would love to revive this thread for Fedora bootc. Anything I can do to help move things forward?

Sorry - it looks like the meeting label was missed on this ticket - I'll try to remember to bring it up in the next meeting (likely in the new year).

That said, I think the entire linux ecosystem might just not be ready for this yet. The alternatives mentioned in the proposal (migration via one-shot systemd services et al.) would probably be better in the long run, but require code changes that I don't think are reasonable (yet) to put on package maintainers. I think it would make sense to make this an aspirational goal, but making the current approach a "SHOULD NOT" seems to be a bit too early IMO.

Yes, this is just about having guidelines and recommendations.

Thanks, Fabio! It's really about guidelines at the moment. We can improve incrementally and address issues as they pop up.

We discussed this during today's meeting. But we'll need to come back to it next week.

We talked about it this week (not sure we had before, although a bunch of FPC had read/commented).

https://meetbot.fedoraproject.org/meeting-1_matrix_fedoraproject-org/2025-01-09/fpc.2025-01-09-17.00.html

If you have time to read that and make PRs that you think we'll merge, please do.

Direct link to the meeting log: https://meetbot.fedoraproject.org/meeting-1_matrix_fedoraproject-org/2025-01-09/fpc.2025-01-09-17.00.log.html

I think completely removing scriptlets is a good aspiration. The Council would like to see Fedora move towards image-based deliverables as our primary outputs, and this is one of many things we'll need to solve along the way.

I also agree with tibbs' comment, and Colin's about improving the whole ecosystem. I'd like to see a change proposal with a plan to progress from recommendation to strong suggestion to requirement over a series of releases.

Thanks all for discussing this! Seems like we have overall consensus on this, so let's try to land something to start. :)

What needs to change in this PR? It currently uses SHOULD already which I had assumed was appropriate. Is that not the case?

Also, anything else in the PR that should be changed?

Trying to sumarize the conversation (without trying to speak for everyone) ... I'd say for the first pass:

1. Make a note that most packages don't need to access /var ... but explicitly say that for those that do have a good reason, it's still allowed. Basically explicitly saying it's a weaker version of SHOULD NOT, where you aren't required to go to a huge effort to fix it but if you don't really need it then don't make the problem worse.

2. Put some SIG/group/matrix-channel/etc contact details with some text like "if you are confused, feel free to contact blah to help you".

3. List more examples of things packagers think they need to do, and don't actually due to macros or dropping files in places etc.

4. Having some kind of long term plan somewhere (as Matthew talked about), with the known problems (https://github.com/james-antill/scriptlets-monitor/ -- still works) etc. would probably also help alleviate fears that we won't just pass this PR and then do a GoT "shame" walk for anyone that doesn't comply. Linking to this data esp. from the text in point 1 might be useful, as a way to say "this is the final destination we're trying to get to, mostly what you need to do is not intentionally move away from it if you don't need to"

I think it's fine to use "SHOULD NOT". Packaging Guidelines don't apply retroactively to existing packages, so "should not" just means that the rule applies to new packages, and existing packages should be updated at some point, but without a clear requirement. And "should not" still allows packages that have a good reason to violate the rule. So splitting hairs to make an even weaker version of "should not" doesn't bring any additional benefit.

The text mostly seems fine. In "Alternatives", please add links to the parts that describes tmpfiles and the other solutions. And some more examples, here or in the linked places, would be great.