/cc @tibbs @james

I'm not sure I'll be around in your meeting if you want to discuss this there, but I can preemptively add details here if needed.

Also, while I don't personally see any issue with this approach, it would be good to get some feedback from the assignee/maintainer @jrybar.

Some feedback from the assignee/maintainer:
Despite the fact that I don't see a hard reason why polkit couldn't have dynamically assigned UID/GID, there is a long-term request (which gets amplified lately) to move polkit to systemd-sysusers logic in order to remove any explicit "useradd" queries from specfile. I'm more into that, as it conforms diskless/stateless systems way better than hardcoding values.

That's just some feedback from the assignee/maintainer, hopefully it's good enough.

@jrybar ack, thanks. Let me then put this on hold for a figure while we figure out further details on bugzilla.

We further discussed this in https://bugzilla.redhat.com/show_bug.cgi?id=2104615 and it turned out that there was some mixup between the sysusers.d move and the actual IDs to use.

The two things are not entangled, and there is still the problem with directories/files shipped as part of the package and owned by a custom user/group, for example the following:

d00750 0 999 /etc/polkit-1/localauthority
d00700 999 0 /etc/polkit-1/rules.d
d00700 999 0 /usr/share/polkit-1/rules.d

@jrybar agreed in BZ that PRs to fix this would be welcome.

Given the above, I'd like to proceed with the allocation of a pair of static GID/UID for the polkitd user and group.
I've checked the uidgid table, and the pair 114:114 seems to be suitable for this.

FPC folks, please let me know if/when I can go ahead with recording this in the setup source and then update the polkit specfile accordingly.

You have my +1 for this. Looks like a pretty clear-cut case, given that polkit is definitely installed and used almost everywhere. Not sure if we need a "formal" vote here ...

If it isn't controversial and doesn't need a formal vote, I'll just wait one more day for any additional feedback or pushback, and then I'll proceed with the PRs for setup and polkit.

I did start assembling PRs for this:
* https://pagure.io/setup/pull-request/42
* https://src.fedoraproject.org/rpms/polkit/pull-request/8

I see there is a "meeting" label on this ticket now, so I'm leaving them in a closed state for the moment. I can re-open them and ask for merges if there are no blockers.

+1 no objections.

We alked about this ticket and the general problem of ostree installed packages in this weeks meeting:

https://meetbot.fedoraproject.org/fedora-meeting-1/2022-08-11/fpc.2022-08-11-16.00.html

...we didn't have quorum, but the general sentiment was that if a package is installed in ostree we might as well just allocate soft-static uid/gids for packages.

So if you can just open one more ticket and include any remaining packages with uid/gids that need them we can +1 that one (again, there are some assumptions that it's not 100s).

Thanks. For context, this was coming from a planned cleanup for Fedora CoreOS, see https://github.com/coreos/fedora-coreos-config/issues/1826.
The list was pretty small (4 packages in total) and we further trimmed it down to 2 users/groups. To my knowledge, this was the last entry missing static IDs.

Hmm, I'm surprised by this. I don't think that any reasons for static allocation apply.

The rationale behind this static UID/GID request is that the polkit package is currently shipping several files and directories under both /usr and /etc which are owned by the user or group polkitd.

That is true for pretty much any package…

• improve OS content reproducibility (by having stable UID/GID in file metadata)

For OS content reproducibility, it is necessary to install the same packages in the same versions. Without that, there won't be reproducibility. And if you have identical package sets, then the uids that will be assigned will be the always identical; the installation process it reproducible.

• avoid possible security mixups (due to the mixing of restricted permission bits and dynamic UID/GID)

That doesn't seem to be a valid reason. You could argue the same for each and every package. We refer to users by name in configuration, and there is no reason to think that having the same uid on different systems is helpful in any way. It is true that permissions masks and owernship have to match, but this is unrelated to specific uid/gid values chosen in a local installation.

• improve deterministic consistency for systems performing out-of-band composes (e.g. ostree and other image-based technologies)

See the previous-previous point above.

Overall, this is strange. The issue described in the original bug report would apply to pretty much any package. The reasons for static uid allocations are:

• sharing of files between initrd and host systemd (e.g. state in /run)
• sharing of files in network fs installations between different machines

… neither of which applies?

Hmm, I'm surprised by this. I don't think that any reasons for static allocation apply.

The rationale behind this static UID/GID request is that the polkit package is currently shipping several files and directories under both /usr and /etc which are owned by the user or group polkitd.

That is true for pretty much any package…

Wait, what? I think you misunderstood something here - or I am misunderstanding your comment. Almost no packages create users / groups and have files in /usr or /etc owned by that user / group and not by root. And for packages like polkit, which apparently do this, and which are installed by default in most (all?) Fedora deliverables, having those users / groups have a consistent UID / GID makes sense, I think.

(Additionally, even if the installation process might be reproducible - and I doubt that it really is - can you guarantee that the polkit UID / GID are consistent across i.e. Fedora Workstation ISO image and Fedora Azure container image, to take two extreme examples?)

Almost no packages create users / groups and have files in /usr or /etc owned by that user / group and not by root.

OK, "any package" is obviously wrong, I misspoke. But quite a lot of packages do, e.g. many packages that create a non-root-owned directory for storing state under /var. I don't know how to search for this cleanly, but rg '%attr.*[^t-]\)' *.spec shows ~400 packages that have non-root-owned files. (Some are %ghost, so more filtering would be required…)

And for packages like polkit, which apparently do this, and which are installed by default in most (all?) Fedora deliverables, having those users / groups have a consistent UID / GID makes sense, I think.

No, not really. If static allocation was totally free, this would be less of an issue, more or a preference/style question and we might assign such numbers more often. This is what we did for some time in the past. But at some point people realized that:

• the space of uids to assign from is severely limited and it's very hard to extend it. Raising the limit for new installations requires a lot of coordination. Raising it for existing installations is essentially impossible. And we have thousand of packages that require thousands of system users if installed in parallel, so we can't assign static uids to all of them. And once this space is exhausted, we're in deep trouble.
• soft-static allocation requires coordination (at least between setup and FPC and the package itself) so it's an overhead.
• soft-static allocation is not at all useful for most of the packages where we had it.
• soft-static allocation creates poor UX, when the user has some local UID/GID already assigned. Even if the space is not exhausted, with the birthday paradox, probability of conflicting uid assignments grows a lot if it there are many assignment. So we want to keep it sparse.

The guidelines say:

Don’t use soft static allocation unnecessarily as the number of available values is limited. Soft static allocation is only appropriate for packages where the UID or GID values are shared between computers. For instance, if the package creates files with the assigned UID or GID that are likely to be shared over NFS.

(Strictly speaking, this text is missing an important case that I mentioned above: initrds. We need static allocation of various gids for ownership of files in /dev because they are created by udev in the initrd, and then immediately available after the transition to the real host. In this regard, the initrd and the host system are like "different computers". In fact, udev uses more static allocations than any other package, and we should describe it better in the Guidelines.)

Additionally, even if the installation process might be reproducible - and I doubt that it really is - can you guarantee that the polkit UID / GID are consistent across i.e. Fedora Workstation ISO image and Fedora Azure container image, to take two extreme examples?

That is not what I meant. It'll most likely be different between two different installations. That is expected and OK. I only said that if you want to reproduce some build artifact, and if you install everything in exactly the same way, you'll get the same uids/gids in that image. (Uids/gids are assigned serially, and since dnf/rpm would use the same installation order, the scriptlets to create users/groups would be executed in the same order, and thus we end up with the same numbers.)

--

I think that the original misunderstanding here is that it's somehow desirable to have those uids the same between different installations. It is not. It would be like trying to ensure that files in directories have a certain ordering or that the on-disk sector layout of files is fixed. All of those things, like the uid/gid assignment, are local implementation details that we shouldn't try to fixate.

But quite a lot of packages do, e.g. many packages that create a non-root-owned directory for storing state under /var.

This is totally true, I agree, but it wasn't discussed here.

The specific scope here is on the packaged content itself which is part of the OS, i.e. mostly the /usr hierarchy (and maybe /etc, but I won't argue for that here).

The specific scope here is on the packaged content itself which is part of the OS

Yes. That is all fine. All the discussion above also covers that case.

(Longer explanation: whether a package has packaged content owned by non-root is important for how packaging is done. If the files owned by non-root are created "at runtime", the package has more flexibility, because it can create the user after the package has been installed. If the files are part of the package, the user must be created before, so in a %pre scriptlet, or possibly in some other package that must be installed earlier. But all this has no relevance for the question whether soft-static or dynamic allocation is used.)

I'm (and have always been) inclined to what @zbyszek says. I don't think it's negative about polkit that it creates its user in %pre section. More disturbing is that some other systems are not robust enough to cope with the natural feature of UIDs/GIDs added dynamically to the system. What's more inconvenient for me at this time is that there are two static UIDs to be reserved for the "same" package. One for polkit, one as an artifact of PolicyKit (which is the same project, actually).

I fear that this isn't a good medium for this kind of discussion (too high latency) and at the same time that the scope of this is growing past the real practical concern at hand (polkit packaging is suboptimal for hermetic systems). I can leave a further comment here, but I'm also happy to have a longer chat somewhere if it helps.

First, on the scoping part: I'd like to focus on /usr content (i.e. not covering /var in this) exactly because it can't be created "at runtime" (firstboot) on hermetic systems. The rest can be somehow covered by the combo of systemd-sysusers and systemd-tmpfiles, thus it isn't my concern here.
This scopes the topic to a manageable set with small cardinality, hopefully mitigating @zbyszek concerns about numbers. Looking at the current content in Fedora CoreOS, it's a total of filepaths belonging to 5 UIDs/GIDs (and 3 of those were already static IDs since long time).

Second, on the deterministic side: scoped only to the hermetic OS content above, yes I think it is desiderable for this specific subset of IDs to be deterministically stable.
I'm not disputing the fact that they are already reproducible on a single fixed compose.
I'm also not disputing that across different installs there will be anyway dynamic UIDs/GIDs (likely, a majority) and local data belonging to them.
I'm stopping one step earlier, when the content of an hermetic /usr is assembled and then distributed.
I think it would be beneficial for those (few) IDs appearing there to be stable across composes, independent of the package set as a whole.

I'm stopping one step earlier, when the content of an hermetic /usr is assembled and then distributed.
I think it would be beneficial for those (few) IDs appearing there to be stable across composes, independent of the package set as a whole.

This is the what. But the why seems to be missing: what is the problem that this is designed to solve?

I'm already happy that we narrowed down the "what" part :)

On the "why" side, the main reason is to have consistent IDs in /usr across subsequent composes of hermetic images (wherever feasible).
That one is not a problem to solve in itself, but it is a useful system property which in turn has several positive ramifications. Non-exhaustive example list:
* composing-machine and installing-machine can agree that the setgid binary with GID xyz belong to a given expected group
* new images/composes stay consistent with the local IDs in the /etc databases of installed machines, independent of newly introduced packages

I'd like to note that this isn't a completely new property. It's another variation of the "shared over NFS" guideline, and most packages already respect that. Example are the set-gid binaries /usr/bin/write (group tty, static GID 5) and /usr/libexec/utempter/utempter (group utmp, static GID 22), just looking at the ~400 packages in the Fedora CoreOS image above.

Overall, this kind of property helps migrating hermetic images (e.g. ostree ones) from an nss-altfiles approach (where most IDs needs to be fixated at compose-time on the composing-machine) to a systemd-sysusers approach (where most IDs can be dynamically picked at run-time on the installing-machine).
The minor friction in this is ensuring that the shared IDs between compose-time and run-time (i.e. those for /usr content above) are consistent.

On the "why" side, the main reason is to have consistent IDs in /usr across subsequent composes of hermetic images (wherever feasible).

In general, the system that handles containers (for a loose definition of container, incl. package build systems and such), must be prepared for the uid/gids in the chroot using a different scheme. In other words, unless it's certain that the container is using compatible uid/gid assignments, isolation must be maintained by the container manager. In effect, it's not particularly useful to have some subset of uid/gid assignments in some images consistent with the host, because you still need to isolation in place. And it's completely infeasible to provide consistency in all cases, because you have newer and older systems, other distros, custom images, etc, etc, etc.

Even at the lever of one version of one distro, for this to work, all packages involved would need to be changed to soft-static allocation. And then we quickly reach the limits of static allocation themes: the systemd package alone defines 7 users that would need to be converted, and we have oodles and oodles of packages.

I'd like to note that this isn't a completely new property. It's another variation of the "shared over NFS" guideline, and most packages already respect that. Example are the set-gid binaries /usr/bin/write (group tty, static GID 5) and /usr/libexec/utempter/utempter (group utmp, static GID 22), just looking at the ~400 packages in the Fedora CoreOS image above.

Those are "special cases". In particular, tty==5 is statically assigned because it value needs to be passed (as a number) to the kernel when mounting /dev/pts. And similarly, various gids that udev uses are static, because, as mentioned above, /run and /dev fiilesystems are shared between the initrd and the host, and the initrd and the host may be built separately, so we need a distro-wide synchronization of those numbers.

There are some other cases where it's reasonable to assign a static number. In particular services that want to create some state files in the initrd and transition to the host system, e.g. systemd-networkd which writes lease information to /run/systemd/netif.

There are also static allocations for various services where it's pointless, but was done before the current policy was put in place, e.g. systemd-journal-gateway, rtkit, wallaby, aelous, ricci, luci,stap*. (Note that many of those packages aren't even part of the distro anymore.)

polkit doesn't fall into any of categories where it'd make sense to assign a static uid.

--

So overall, soft-static assignment of a few uids would provide a partial solution, and since this partial solution still needs to be combined with a real solution that covers all cases, it's just not useful.

On the "why" side, the main reason is to have consistent IDs in /usr across subsequent composes of hermetic images (wherever feasible).

Could you elaborate here?

It sounds like you'd want to have consistent UID/GID of various files in /usr across different composes (i.e. even if the package set / installation order changes, you still want polkit to get the same IDs).

I assume this affects OSTree based deliverables, i.e. you wouldn't want an updated compose for the base image cause incompatible assignment of UID/GID for OS components like polkit. Am I understanding this correctly?

@decathorpe your summary is correct (I wouldn't frame it as specific to ostree artifacts but fair, that is my current context).

I assume this affects OSTree based deliverables, i.e. you wouldn't want an updated compose for the base image cause incompatible assignment of UID/GID for OS components like polkit.

If that is the issue, then some ostree-specific solution should be used. E.g. pre-populate /etc/{passwd,group} with a copy from a previous compose. This will actually work comprehensively for all assignments, not just the few that were converted to static allocations.

The approach with static allocations is impacts the whole distro and creates long-term issues with uid/gid availability.

Thanks everybody for all the comments, I'll personally stop here.
I've effectively just gone full-circle back to the initial starting point (nss-altfiles and workarounds in ostree), and I've been worn out by multi-weeks discussions.