Ugh, what a mess.

I'm +1 to requiring the removal of the provides from the package.

This was discussed today at the EPEL steering committee meeting. We will leave this open for discussion for EPEL Steering Committee members for one week. At that time if there are no -1s and at least 3 +1s, this is approved. Please note that voting is only for EPEL Steering Committee members, not the general public.

It was also suggested at the meeting that we should consider removing the obsoletes as well. The upgrade is disruptive enough that it's worth considering preventing that disruption for users that have yet to upgrade from the singularity package. I'll open a separate issue for that to hold a separate vote.

+1

See #215 to discuss and vote on removing the obsoletes as well.

It was unfair of Carl to take only one side of this story to the EPEL Steering Committee meeting, without inviting an opposing view. Since the meeting was yesterday I will have to try to do my best to write up the defense here, when I have a chance, later today.

This proposal, to remove the "Provides: singularity" from apptainer package in EPEL, rests on two grounds.

The first, the config directory change, is easy to justify. Of course it had to change, because the package name changed. The configuration changes were not automatically imported, but there is a warning every time the tool is run if an old changed configuration still exists, so a system administrator will immediately know what to do and fix the situation.

Before addressing the second grounds, I want to say that I have apologized and I apologize again for not properly following the EPEL Incompatible Upgrades policy when I updated the package. I have been providing only this one package for so long without encountering this issue that it did not occur to me to check the rules. Now that I know the rules, I will not make the same mistake again. I am convinced, however, that if I had gone through the proper process the EPEL Steering Committee would have permitted me to go ahead with it without removing the "Provides: singularity". It is justified for security reasons (detailed below), which the policy allows, and the previous behavior can still be easily achieved by the system administrator action of installing the additional package apptainer-suid.

The second grounds, the switch to unprivileged user namespaces by default, is a significant improvement in security. The previous behavior used setuid-root to do something that kernel developers say is inherently unsafe and can be used to attack the kernel: it mounts filesystems with squashfs and ext2/3/4 kernel drivers while still allowing the user write access to the underlying data. Apptainer 1.1 added the ability to do that all with unprivileged user namespaces using squashfuse, fuse2fs, and (when needed) fuse-overlayfs.

The proposal refers to the fact that an alternative implementation called singularity-ce was recently released in EPEL. That is a package from the SingularityCE project which is based on a fork of the community-based Singularity project. The existence of two so similarly-named projects caused much confusion in the community. When the Singularity project later joined the Linux Foundation, the Linux Foundation required the project to select a new name. Apptainer is not a fork of the Singularity project, it is the same project, renamed. So far Apptainer has been importing most of the improvements that SingularityCE develops, but SingularityCE has indicated that they do not intend to import these security enhancements from Apptainer, even as a non-default option.

Since unprivileged user namespaces are enabled by default in el8 and later, this change is not much of an incompatibility at all there. (It is also not a drop in performance). It's only el7 where unprivileged user namespaces are not enabled by default. It is possible by packaging alone in epel7 to maintain the compatibility by default, by having the apptainer package Require apptainer-suid. The decision was made by the Apptainer project leadership to go ahead and force a system administrator on el7 to make a choice as to which type of security they want. This could still be changed if the EPEL Steering Committee recommends it, although this would decrease the security by default especially for those that have already upgraded.

I believe that for these reasons that this proposal is unwarranted and I request that it be rejected.

It was unfair of Carl to take only one side of this story to the EPEL Steering Committee meeting, without inviting an opposing view.

There are no sides here, just the technical details. The incompatibilities I cited were directly from your own (late) announcement email. I can't think of anything more fair than neither package providing singularity.

It is justified for security reasons (detailed below), which the policy allows, and the previous behavior can still be easily achieved by the system administrator action of installing the additional package apptainer-suid.

The regular updates policy does not allow this. Security implications are a factor that is considered when the incompatible update process is initiated. That process is not a guaranteed approval. It may result in further discussion that changes the plan. For example, if the old behavior is achieved by installing apptainer-suid, why didn't apptainer-suid obsolete singularity? Security is important but so is compatibility.

The decision was made by the Apptainer project leadership to go ahead and force a system administrator on el7 to make a choice as to which type of security they want.

The way this was executed was not forcing a choice, it was making the choice for users which broke the functionality for a significant number of those users. Upstream project desires do not overrule EPEL policy.

Hello. As the package and upstream maintainer for singularity-ce I'd like to address a couple of points that are troubling. I won't wade into the packaging specifics of provides / obsoletes as I don't feel I really have any authority as a new packager.

Apptainer is not a fork of the Singularity project, it is the same project, renamed.

The original singularity lineage has moved a lot over the years, including through Sylabs, but is now in the retired repository below... note that it is dormant, and redirects people to another repository, a fork, github.com://apptainer/apptainer.

https://github.com/apptainer/singularity/

So far Apptainer has been importing most of the improvements that SingularityCE develops, but SingularityCE has indicated that they do not intend to import these security enhancements from Apptainer, even as a non-default option.

This is somewhat misleading. The linked GitHub thread has more nuanced discussion. SingularityCE is working towards unprivileged by default, but we are taking a conservative staged approach which is not compatible with the way Apptainer has implemented it... due to the needs of the rest of our roadmap. We also have considerable concern over maintaining full compatibility for users on EL7, or who are otherwise adverse to enabling user namespaces. At this stage we do, in fact, have equivalents for portions of the user namespace / fuse related changes in apptainer, but they are optional.

I would also like to point out that there is further incompatibility from the singularity 3.8 package to apptainer 1.1, beyond the issues brought up so far around the user namespace by default change.

• Existing encrypted SIF containers cannot be run with apptainer 1.1 (unless apptainer-suid is installed).

• Some MPI workflows may require different workarounds with apptainer 1.1 (https://github.com/apptainer/apptainer/issues/769)

• Support for build --remote (using Sylabs remote build service, which has a free tier, and open source client side code) was removed completely at Apptainer 1.0.

• The default remote endpoint for library:// URIs was changed from cloud.sylabs.io to cloud.apptainer.org which does not exist. This breaks existing workflows for people who have been pulling containers from Sylabs cloud library via library:// URIs, unless they manually reconfigure.

• In non-suid mode, now default for apptainer, users can specify a custom --config file that overrides restrictions the sysadmin specifies on which containers are allowed to run:

http://apptainer.org/docs/admin/latest/configfiles.html#limiting-container-execution
http://apptainer.org/docs/admin/latest/configfiles.html#execution-control-list

Note that the documentation above does note the limits only apply in setuid mode, but it is a significant change vs the singularity package.

Does anyone have any links / transcripts / emails / video on why Fedora didn't allow Provides or Obsoletes in apptainer for Fedora?
I know where the policy is, I'm talking about apptainer specifically.

In it's Package Review,[1] it was discussed and they figured out the correct format for Provides and Obsoletes. And so it was initially built in Fedora with a Provides and Obsoletes. But I can't find anything else.

[1] - https://bugzilla.redhat.com/show_bug.cgi?id=2034758

It wasn't discussed by Fedora. It's just that in the direct negotiation between Carl and me, I offered to drop the Provides for Fedora, because it's EPEL that I care more about.

If this wasn't brought up to Fesco or even the Fedora mailling list, then let's just drop the whole Fedora part of this discussion. It's irrelevant.

First point. Is Apptainer a rename of Singularity?
From everything I can see, legally, yes it is.

Second point. Does it provide a similar functionality?
From everything I can see, yes, in the overall sense.

Third point. Is it backwards compatible?
Without modifications on the user machine. No.
With modification on the user machine. Yes

So the real question here is, how incompatible does something have to be before we do not allow it to have a "Provides" and/or "Obsoletes"?

For that we should go to the Fedora Renaming guidelines.
https://docs.fedoraproject.org/en-US/packaging-guidelines/#renaming-or-replacing-existing-packages

Reading those guidelines we see

If a package is being renamed without any functional changes, or is a compatible enough replacement to an existing package (where "enough" means that it includes only changes of magnitude that are commonly found in version upgrade changes), provide clean upgrade paths and compatibility with:

Provides: oldpackagename = $provEVR
Obsoletes: oldpackagename < $obsEVR

If a package supersedes/replaces an existing package without being a sufficiently compatible replacement as defined above, use only the Obsoletes: line from the above example.

To me, it looks like this rename is not a sufficiently compatible replacement. Thus it should NOT have a Provides, but it SHOULD have an Obsoletes.

I think this is a change of magnitude commonly found in version upgrade changes. It was done for improved security, as many packages commonly do.

Say, it just occurred to me based on Carl's recent comment in #215 that on epel7 we could move the Provides to apptainer-suid instead of apptainer. That should only affect people that have not yet updated singularity, right? Wouldn't that make it compatible enough? It is not something I thought of before. I will test it.

To me, it looks like this rename is not a sufficiently compatible replacement. Thus it should NOT have a Provides, but it SHOULD have an Obsoletes.

This was my original feeling as well. Someone else suggested at the last meeting that we should at least consider dropping the obsoletes as well. I don't necessarily agree (or disagree) with that idea, but I do agree it merits discussion. That discussion is taking place in #215.

I think this is a change of magnitude commonly found in version upgrade changes.

Renaming the configuration directory or path is not a change commonly found in version upgrade changes.

I think this is a change of magnitude commonly found in version upgrade changes.

Renaming the configuration directory or path is not a change commonly found in version upgrade changes.

Fedora did not object to that. It would apply to nearly every package rename except for those that have no configuration.

Fedora did not object to that.

If by Fedora you mean the package rename review, it doesn't look like backwards compatibility was discussed in the review at all. All the scratch builds were against rawhide at the time, and the only mention of stable Fedora branches is in the last comment stating that it was built there too. This is the sort of change that would be acceptable in rawhide (preferably with a change proposal) but not in released branches. As Troy stated, Fedora is not really relevant to this discussion, but I encourage you to read the Fedora updates policy.

Yes! Putting the Provides on apptainer-suid on el7 works well. A yum install singularity installs apptainer-suid and apptainer there, and if only apptainer is installed then a yum install singularity says "No package singularity available". On el8, a dnf install singularity installs only apptainer.

In addition, in order to address the concern about a change in configuration file, I could add an automatic conversion process so that a system administrator would not need to it themselves. The configuration is nearly identical, and it will make the few changes needed.

So, I have this counter proposal:
1. Keep the Provides: singularity and Obsoletes: singularity, but on el7 move them to apptainer-suid.
2. Add a configuration file importer from singularity to apptainer to run whenever apptainer replaces singularity.

Will that be enough? It would fix both factors addressed in the proposal of this issue.

How does this affect singularity-ce? Because by @dctrud's claim, it's also a sufficiently compatible implementation.

If both are compatible but aren't co-installable, could we have them both Provides+Conflicts the singularity name too?

How does this affect singularity-ce? Because by @dctrud's claim, it's also a sufficiently compatible implementation.

If both are compatible but aren't co-installable, could we have them both Provides+Conflicts the singularity name too?

I'm not too fussed about singularity-ce having a Provides: singularity tbh. To my mind, Apptainer and SingularityCE are both going to diverge further from what was singularity as a package up to 3.8. This divergence should be via changes only at major versions, packaged where appropriate per EPEL policy, but I don't want to add a Provides: singularity now, only for it to cause future arguments, or a need to remove it, later.

The idea of Provides: sif-runtime is more attractive to me.... the situation is similar to crun and runc with Provides: oci-runtime - xxx-runtime is a more abstract concept than saying we provide the prior name of the software.

My bigger concern is that even if we have user namespaces enabled by default (EPEL8 / EPEL9), or apptainer-suid installed (dwd's EPEL7 specific counter proposal), there are remaining incompatible changes from the singularity -> apptainer packages that I've mentioned above. I don't think the Provides: singularity level of compatibility for apptainer that others have set out is satisfied here?

• Existing encrypted SIF containers cannot be run with apptainer 1.1 (unless apptainer-suid is installed). Would not be an issue on EL7 under dwd's counter proposal, but still under EL8 and EL9

• Some MPI workflows may require different workarounds with apptainer 1.1 (https://github.com/apptainer/apptainer/issues/769) Would not be an issue on EL7 under dwd's counter proposal, but still under EL8 and EL9

• Support for build --remote (using Sylabs remote build service, which has a free tier, and open source client side code) was removed completely at Apptainer 1.0.

• The default remote endpoint for library:// URIs was changed from cloud.sylabs.io to cloud.apptainer.org which does not exist. This breaks existing workflows for people who have been pulling containers from Sylabs cloud library via library:// URIs, unless they manually reconfigure.

• In non-suid mode, now default for apptainer (except EPEL7 in dwd's counter proposal), users can specify a custom --config file that overrides restrictions the sysadmin specifies on which containers are allowed to run.

The removal of build --remote support in apptainer, and the change such that it doesn't pull container images from cloud.sylabs.io by default, cause real confusion to users. This is what common flows that worked with the old singularity package now look like with apptainer:

$ apptainer build --remote bob.sif bob.def
Error for command "build": unknown flag: --remote

$ apptainer pull library://ubuntu
FATAL:   Unable to get library client configuration: remote has no library client

There is limited automatic conversion of old settings for the pull from library:// case... but it doesn't cover all cases.

We've never had very satisfactory explanations on why these features were removed / changed.

The pull library:// one is particularly odd to me. Apptainer ships configured to use a URI cloud.apptainer.org which doesn't even exist.

$ apptainer remote list
Cloud Services Endpoints
========================

NAME           URI                  ACTIVE  GLOBAL  EXCLUSIVE  INSECURE
DefaultRemote  cloud.apptainer.org  YES     YES     NO         NO

$ ping cloud.apptainer.org
ping: cloud.apptainer.org: Name or service not known

Because Sylabs (my employer) runs cloud.sylabs.io, we have to periodically field inquiries as to why apptainer pull library://ubuntu doesn't work, or whyapptainer build --remote doesn't work for users on systems that have upgraded singularity -> apptainer.

Based on these additional incompatibilities, it still seems to me that apptainer is not "a sufficiently compatible replacement". I agree with @tdawson that the provides should be removed from apptainer to follow the packaging guidelines.

Based on these additional incompatibilities, it still seems to me that apptainer is not "a sufficiently compatible replacement". I agree with @tdawson that the provides should be removed from apptainer to follow the packaging guidelines.

Those additional incompatibilities mentioned by @dctrud are all minor and I believe they fall under the rule exception of things that could be expected in normal version upgrades. Especially with my counter proposal changes, apptainer is indeed sufficiently compatible.

Apptainer no longer needs the --remote option because container building can now be done locally by unprivileged users, even without the system administrator setting up /etc/subuid and /etc/subgid mappings, by automated use of the fakeroot command. Apptainer probably should have a more helpful error message telling people to just try leaving out the --remote option; I add that to my counter proposal too. The Linux Foundation required that the --remote option be removed because the service was only run by a for-profit company and implemented with a closed-source product. The default setting for the library command was removed also because it was only for a service for a for-profit company, something the Linux Foundation didn't want (in fact they wanted Apptainer to remove the library:// URL entirely, but I led a campaign to keep it because of alternative open source implementations of the server side). The documentation tells people how to restore it to the previous behavior if they want it. These changes would have been introduced even if the name didn't change, once Sylabs left the project, so they're part of a normal upgrade.

The things that still only work with suid flow (assuming unprivileged user namespaces are available) are not very much used, and they would have been introduced regardless of the name change. The encryption feature is the only one that has somewhat significant usage, and that is expected to be fixed for non-suid use in version 1.2. MPI has significant use, but there's an easy documented workaround and the problem doesn't affect all use cases.

I think it is especially important to keep the "Provides: singularity" in epel7, because with only an Obsoletes then even though "yum update singularity" installs apptainer when singularity is already installed, a "yum install singularity" does not. Normally people expect both do the same thing. dnf in el8 and later does not have this problem. epel7 is also the place where my counter proposal will be keeping the suid mode by default for upgrades.

@tdawson what do you think about my counter proposal? Doesn't it make Apptainer compatible enough to the former Singularity?

At this point, I am going to abstain from responding and/or thinking much about this until Monday. I'm still sick from the same illness that caused me to miss the weekly meeting on Wednesday. I tell ya'll this just so you now I'm not ignoring things, but that I'm going to be quite for a bit.

Those additional incompatibilities mentioned by @dctrud are all minor and I believe they fall under the rule exception of things that could be expected in normal version upgrades. Especially with my counter proposal changes, apptainer is indeed sufficiently compatible.

Apptainer no longer needs the --remote option because container building can now be done locally by unprivileged users, even without the system administrator setting up /etc/subuid and /etc/subgid mappings, by automated use of the fakeroot command.

This is not true for a significant proportion of container builds. Because of how apptainer uses fakeroot, If the glibc in the container being built is older than the host, it will not work.

This is a very common workflow - e.g. building a container that holds an older distro, to support older software, after upgrading the host environment to a newer release.

Issue noting this: https://github.com/apptainer/apptainer/issues/865

If I have no subuid/subgid mappings, I cannot build an EL7 container from newer systems, without --remote. This means that even on EL8, and EL9, apptainer is significantly incompatible with singularity.

$ cat el7.def 
Bootstrap: docker
From: centos:7

%post
  echo Hello

$ apptainer build el7.sif el7.def
INFO:    User not listed in /etc/subuid, trying root-mapped namespace
INFO:    The %post section will be run under fakeroot
INFO:    Starting build...
Getting image source signatures
Copying blob 2d473b07cdd5 done  
Copying config 6d42e44a9f done  
Writing manifest to image destination
Storing signatures
2022/12/19 17:23:58  info unpack layer: sha256:2d473b07cdd5f0912cd6f1a703352c82b512407db6b05b43f2553732b55df3bc
INFO:    Running post scriptlet
/.singularity.d/libs/faked: /lib64/libc.so.6: version `GLIBC_2.33' not found (required by /.singularity.d/libs/faked)
/.singularity.d/libs/faked: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by /.singularity.d/libs/faked)
fakeroot: error while starting the `faked' daemon.
kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
FATAL:   While performing build: while running engine: exit status 1

This is not true for a significant proportion of container builds. Because of how apptainer uses fakeroot, If the glibc in the container being built is older than the host, it will not work.

This is a very common workflow - e.g. building a container that holds an older distro, to support older software, after upgrading the host environment to a newer release.

That's not nearly as common a use case as the cases that work. Most of the time, it works. It also has a big advantage over "--remote" by making it possible to have access to the host's compilers and libraries.

Now that I am feeling better, I think I can reply politely and sensibly.

Now that I've had time to think of it, I like having two issues, one for provides and one for obsoletes. Since there are different criteria for each, it allows us to talk about each independently.

Before I give my opinion, I just want to point out that the EPEL Steering Committee is not here to decide which program is better, or anything like that. This discussion is about the interpretation of policy.

First, is this a rename or a fork. After looking through everything, I still say this is a rename.
Going through all the legal stuff, singularity went to the Linux Foundation which required it to change it's name. Was the code then forked to create apptainer, yes, because that is one of the main ways of doing that.
@dwd was also the Fedora/EPEL singularity maintainer and went through the process for a package rename. Some steps might have gotten glossed over, but he has fixed, or working on fixing those, which is where this discussion came from.

Should apptainer have "Provides: singularity" in it?
Currently, no. It is too much of a transition for your average user/admin.

What if @dwd or the apptainer community do the fixes he proposed:
Put the Provides on apptainer-suid
Provide a config conversion script / utility so users/admins get their configs moved over automatically or manually.
* Do a wrapper, or something, so that functions that were removed don't just fail, but give the user some idea what is going on and a way to move forward.

If that get's done, then in my opinion, yes, go ahead and put "Provides: singularity" in apptainer-suid.
But until that get's done, you should take the "Provides: singularity" out of apptainer.
As things stand right now, apptainer is basically creating a Denial of Service to all the current singularity users that do updates.

I've thought all night about my last sentence, because it sounds harsh. But I still haven't been able to think of better phrase. So, I apologize that it sounds harsh, but I think it conveys the feeling that an average user/admin that currently does an update from singularity to apptainer.

Ok. We discussed it in the Apptainer Technical Steering Committee and agreed to go ahead and move the Obsoletes/Provides from apptainer to apptainer-suid everywhere, not just on epel7. Even though people who have singularity will not by default get the benefits of improved security available with the unprivileged user namespaces available by default on el8 and later, it will be more compatible. They will have to remove apptainer-suid manually (or set an apptainer.conf option) to get the improved security.

I propose to apply apptainer packaging PR 2 to all EPEL & Fedora branches tomorrow, after 1.1.4-1 is promoted to stable, and then implement a singularity.conf import function and --remote error message for the 1.1.5 release which will be targeted for 3 weeks from today, after the holidays.

Ok?

As a member of both FESCo and EPSCo, I'm rather annoyed at how all of our policies have been subverted/ignored. I'm also confused why the Apptainer TSC has anything to do with this.

But the evidence indicates that Apptainer doesn't qualify to retain Provides on anything at all. And frankly, the acrimony demonstrated in this ticket makes me weary of this whole thing.

Regardless, I cannot support Apptainer retaining Provides: singularity unless there's a demonstrated commitment to compatibility with what historic Singularity users expect.

Apptainer has already demonstrated that this not a goal (which is fine), and @dctrud has also indicated that Singularity-CE isn't likely to do it either, so I think Provides: singularity should just go away for now.

I'm also confused why the Apptainer TSC has anything to do with this.

I was just just trying to say that it wasn't me alone that was agreeing to assign apptainer-suid the replacement role instead of only apptainer as @tdawson suggested, the whole project leadership agreed.

Regardless, I cannot support Apptainer retaining Provides: singularity unless there's a demonstrated commitment to compatibility with what historic Singularity users expect.

With apptainer-suid and the other changes I referred to, the compatibility with historic Singularity is very strong.

With apptainer-suid and the other changes I referred to, the compatibility with historic Singularity is very strong.

Clear error messages for removed functionality are important, but they are not the same thing as being compatible.

I also think that dropping Provides: singularity is the right call here -- Apptainer and Singuarity-CE should carry Provides: sif-runtime, but neither is a drop-in replacement for the "old" singularity, and neither should have Provides: singularity. This is true for Fedora, and it is especially true for EPEL, where policy requires avoiding major functionality changes within releases.

And for voting purposes: I'm +1 on the proposal to drop Provides: singularity

I'd like to extend the offer in my last comment a little further.

In addition to the mitigation for the --remote option we will also mitigate the missing default library:// URL. That takes care of the last remaining incompatibility with old Singularity.

@tdawson said

If that get's done, then in my opinion, yes, go ahead and put "Provides: singularity" in apptainer-suid.
But until that get's done, you should take the "Provides: singularity" out of apptainer.

I would move the Obsoletes, and Provides if the final decision here allows, to apptainer-suid as soon as the decision here is finalized. We will do the other things as soon as possible after the holidays, in 3 weeks. So if we need to drop the "Provides: singularity" in the immediate update, could we put it back on apptainer-suid with the next release in 3 weeks?

I'm somewhat new to rpm style packaging being more of an apt guy. I don't want to derail the adult conversation in the room, but my understanding of the Provides and Obsoletes labels are that they are a courtesy to an existing user base, I guess in this case singularity, and less of a contract. If they're left up to the judgement and good taste of the packager generally, why not here? Maintainers are free to update package versions for example if they think it's warranted. If two separate packages are laying claim to the singularity legacy, is there an easy way to compute that and present it somewhere?

I don't think I've ever installed a package (or not) on the basis of Provides but perhaps dnf does something nice for me so I don't have to "Where's Waldo" a rename.

I'm also not voting because I don't know enough to cast an informed vote. And the new guys should watch and learn, usually in silence. As Abe Lincoln is reputed to have said: "Better to be silent and thought a fool than to open your mouth and remove all doubt."

So if we need to drop the "Provides: singularity" in the immediate update, could we put it back on apptainer-suid with the next release in 3 weeks?

I am agreeable to that. Just remove the "Provides:" line is removed from your pull request [1] for the time being.

At some point, we (the EPEL Steering Committee) have to look at the security issue, and I think that @dwd plan makes the most sense for that.
If nobody provides singularity, and security issues arise, then the end users/admins are left not even knowing there is an issue.
If the apptainer developers are willing and able to do the changes I asked above (configs and function wrappers) then I think they should put Provides in their spec file.
The fact is, singularity is gone. The users/admins have to make a choice at some point.
As things stand, with apptainer leaving updated users in the dark, I don't like that.
But if apptainer makes it easier for them to see the path forward, one way or another, that is much better than leaving them unaware that a change has happened.

[1] - https://src.fedoraproject.org/rpms/apptainer/pull-request/2#_1__50

I have updated the PR with that.

I'm a +1 for removing the Provides now, though given the issues raised here, I think we should revisit this (maybe in a new ticket) before apptainer-suid acquires the Provides: singularity.

Landing the removal now is more important, to mitigate the situation of users being upgraded to an insufficiently compatible configuration.

In addition to the mitigation for the --remote option we will also mitigate the missing default library:// URL. That takes care of the last remaining incompatibility with old Singularity.

Again, adding friendly warnings for removed functionality is a great thing that should happen, but that is not the same thing as actually being compatible enough to retain the provides for the former name.

If they're left up to the judgement and good taste of the packager generally, why not here? Maintainers are free to update package versions for example if they think it's warranted.

It's not entirely up to the packager. There are guidelines governing renaming packages and what version updates are appropriate in each branch (EPEL branchs, Fedora branches).

If two separate packages are laying claim to the singularity legacy, is there an easy way to compute that and present it somewhere?

That's not what this issue is about. @dwd has agreed to remove the provides in apptainer in Fedora to follow the packaging guidelines, specifically the part that states replacement packages which are not sufficiently compatible should only obsolete and not provide the previous package name. He wants to diverge from the policy and do something different for EPEL (keep the provides). I advised him that this was not justified, and he asked to escalate to the EPEL Steering Committee.

I'm also not voting because I don't know enough to cast an informed vote.

Per my previous comment, only EPEL Steering Committee members vote.

At some point, we (the EPEL Steering Committee) have to look at the security issue, and I think that @dwd plan makes the most sense for that.

I'll note that this is not a security issue like a CVE. Everyone seems to agree that switching to unprivileged user namespaces is more secure, but it's not something that we necessarily need to force, especially on EL7 where manual action is needed to enable them.

If nobody provides singularity, and security issues arise, then the end users/admins are left not even knowing there is an issue.

Providing singularity doesn't change this. If apptainer-suid obsoletes singularity, no one is left behind on an unmaintained package. Apptainer can communicate within their community that uninstalling apptainer-suid (after manually enabling unprivileged user namespaces on EL7) will opt users into better security.

This was discussed in the EPEL Steering Committee meeting.
Please go ahead and merge and build
https://src.fedoraproject.org/rpms/apptainer/pull-request/2
or
https://src.fedoraproject.org/rpms/apptainer/pull-request/1

When the time comes that you have the changes discussed above, please open a new issue.

The final vote was (+3, 0, -0) to remove Provides: singularity from apptainer in EPEL branches to match what is planned for Fedora branches.

https://meetbot.fedoraproject.org/fedora-meeting/2022-12-21/epel.2022-12-21-21.00.log.html

As we don't see a justification for EPEL to diverge from Fedora in this instance, a future request to add the provides back should go to FESCo, not EPSCo.