#2146 Temporarily revert breaking module changes
Closed: Accepted 2 months ago by ignatenkobrain. Opened 4 months ago by psabata.

The tooling is currently not ready for modules that change runtime module-level dependencies within the same stream, e.g. switch from depending on module:a to module:b. However, such changes recently happened in the silver and bat modules which is now breaking rawhide.

We do not permit automatic stream switches. Once you enable a stream, you follow it. If an update previously enabled module:a, DNF can't automatically enable module:b even if a new version of your module requires it. Furthermore, DNF currently looks at all versions of each module when considering dependencies and RPM names for filtering so that all RPM versions within a stream are available for cherry-picking.

We need to discuss/design how to handle these situations, potentially by making DNF only consider the latest selected version of each stream. That would resolve the situation where you're deploying a new system. We also need to design some behavior for the upgrade case.

I'd like to propose dropping the Rawhide builds and the updates until this is resolved.


Note that this breaks all Rawhide network installs at present:

https://bugzilla.redhat.com/show_bug.cgi?id=1719976

due to that I'm very much +1.

I don't understand why this requires a fesco vote, but +1.

  1. What is the ETA for fixing these?
    1.1. If it is more than a few weeks / month, you might want to drop any Rust modules which depend on libgit2 because I can't ship updates for them because of this.
    1.2. If it is less, I think I am fine with just dropping latest builds and wait for a resolution. And then tagging them back.
  2. Something should be done in stable releases as well. Be that stopping from updating any rust apps depending on libgit2 or even dropping all of them entirely is up to you.
  3. If resolution for this DNF bug will take more than a half year (as with other DNF-modularity bugs), feel free to retire all Rust modules and remove defaults for them because I am not going to do any modularity stuff then.

Btw, it is really depressing when you try to use main features which modularity brings and world breaks. Modularity is about shipping multiple versions of content, however you can't really use that because DNF is just broken by design.

Well, given that there's no actual defined behavior for modularity that DNF (or anything else) can conform to, I would wager that this is not fully DNF's fault.

But this is a very serious problem, because once again, we're screwed on shipping Rust software because of issues like this.

Oh yeah, and I forgot one more thing (which came up from our discussion on #fedora-rust with @ngompa and @jistone). We've upgraded non-modular libgit2 to 0.28 in rawhide. We also changed a default stream... And people who had those old modules dependent on 0.27 would have to explicitly disable all of them and enable 0.28 one. So we are in a loop here, if we untag new builds, people who use those modules will have to stay on 0.27 while KDE would be built against 0.28 so they simply can not.. And we can't ship updates for those modules because switching does not work.

So you might want to consider opening some discussion on devel@ about how to handle this.

We do not permit automatic stream switches. Once you enable a stream, you follow it.

I suspect you can switch manually?

Can we at least have a rawhide workaround for the update (aka, switch screams, run dnf update in one transaction?).

we package libgit without modules as proposed by @carlgeorge in https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/P4XUEWEK33UQESIUJL3OLNGZLJYYXX7R/

That is just temporary workaround. Also it doesn't say that libgit2_0.28 will be added into f30. And also, it does not show how Obsoletes are going to be handled.

But since that is just workaround, we need to decide what to do with Modularity in general. We need to get some commitment from DNF and/or Modularity WG on providing a fix for these bugs.

Alternatively, we need to develop a plan if there is no such commitment. I have something in mind, but I don't dare to speak it.

Alternatively, we need to develop a plan if there is no such commitment. I have something in mind, but I don't dare to speak it.

I definitely don't disagree. My point is that we need to either get commitment or just stop using technology which just breaks distribution instead of helping to fix problems we have.

That is just temporary workaround.

Care to elaborate? What stops this solution from being permanent?

Also it doesn't say that libgit2_0.28 will be added into f30.

If we also need to solve for needing 0.28 in F30, there are two ways to go about it.

  • Add libgit2_0.27 to F30 and upgrade libgit2 to 0.28.x
  • Keep libgit2 at 0.27.x and add libgit2_0.28

Both are valid, but I would prefer the unversioned package to be the latest. If you would like to see it in action I'd be happy to make that change in the copr for experimentation.

And also, it does not show how Obsoletes are going to be handled.

With multiversioned library packages obsoletes are not necessary. Unless you explicitly install one of these, they will be removed by dnf autoremove or by the default clean_requirements_on_remove switch.

Care to elaborate? What stops this solution from being permanent?

Sure. Modularity is designed to allow us to ship multiple versions of software. We tried to use it and it breaks distro. Going back to packages with mangled names is just workaround until DNF will be fixed.

Add libgit2_0.27 to F30 and upgrade libgit2 to 0.28.x

You need to do it for F29 also then. And also it means, you need to change all packages which depend on libgit2 to select proper version.

With multiversioned library packages obsoletes are not necessary. Unless you explicitly install one of these, they will be removed by dnf autoremove or by the default clean_requirements_on_remove switch.

If you add libgit2_0.28, then upgradepath will be broken.

We tried to use it and it breaks distro.

It seems to work fine for some software, but using it for libraries is where the problems arise. Maybe the solution is not letting modules depend on other modules? Even if modules are limited in that way, they would still offers real benefits for end user applications.

You need to do it for F29 also then.

Obviously the same approach can be take for all active Fedora releases. Again, if you'd like to see this in action I can implement it in that copr to demonstrate.

And also it means, you need to change all packages which depend on libgit2 to select proper version.

False. Packages should build require pkgconfig(libgit2) (optionally with constraints to force not the latest one), then they will require the soname of the library (libgit2.so.27, libgit2.so.28, etc.) and will resolve the the package that provides it. Me and @remi did this with libsodium in EPEL and it worked well.

If you add libgit2_0.28, then upgradepath will be broken.

Good point, so keeping libgit2 on the latest version and adding previous versions as compat packages is the best choice.

It seems to work fine for some software, but using it for libraries is where the problems arise. Maybe the solution is not letting modules depend on other modules? Even if modules are limited in that way, they would still offers real benefits for end user applications.

This is kinda my point, just dealing with libgit2 does not help anything. We need to find out what to do in future to prevent such things from happening, whether it will be prohibiting dependencies of modules or entirely stop modularizing or something else needs to be figured out.

False. Packages should build require pkgconfig(libgit2) (optionally with constraints to force not the latest one), then they will require the soname of the library (libgit2.so.27, libgit2.so.28, etc.) and will resolve the the package that provides it. Me and @remi did this with libsodium in EPEL and it worked well.

They are, but if package is not compatible with 0.28, dnf will select libgit2-devel (0.28.x) by default for pkgconfig(libgit2) which will break build of those applications.

They are, but if package is not compatible with 0.28, dnf will select libgit2-devel (0.28.x) by default for pkgconfig(libgit2) which will break build of those applications.

We already have an established way to solve that problem that isn't unique to the modularity scenario.

BuildRequires: (pkgconfig(libgit2) >= 0.27 with pkgconfig(libgit2) < 0.28)

We already have an established way to solve that problem that isn't unique to the modularity scenario.

I'm not saying there is no existing way to solve the problem. I'm just saying that somebody will need to fix existing packages to use these constraints. It's not many, but somebody has to do it.

I believe that there is a common agreement that the last changes must be reverted +1. Sooner is better. Any prolongation of the present situation (broken update testing/rawhide) is harmful
to Fedora project itself.

I'm pretty sure the libgit2 breakage will get reverted, but how that will take place is still not decided.

We still have to actually solve the problem. And if it gets reverted with no valid solution, this will be swept under the rug.

I agree. Any temporary solution must have a deadline. Otherwise it is not temporary.

We are at the situation that maintainer of silver, bat, tokei is pushing updates to bodhi even with knowledge that it breaks the distribution. See 10 days old update https://bodhi.fedoraproject.org/updates/FEDORA-MODULAR-2019-7e30cc5c88.

But this is not against any rule or packaging guidelines or modularity recommendations. And there is no mechanism like a gating that would prevent it.

According to last event mentioned above it is getting to be a general issue therefore FESCO should agree at least on statement, plan, or guidance.

Do we have an ETA from the DNF team and/or the modularity team to fix this mess or should we look into "ban this temporarily forever" kind of thing?

What I know is that there is a document in preparation about what is the modularity.

According to discussions the modularity provides an alternative but without a harm. Changing module defaults, or module requirements during the life-cycle is against the philosophy.

Also DNF team cannot change the behaviour in this particular case because it is against other requirements like switching module stream must be forbidden in all cases. The changing streams is similar to system upgrade or system downgrade. It means painful or unsupported or impossible operation. We have to also consider non-modular content provided by third parties that could be negatively affected by switching streams.

Of course DNF team will be happy to provide an improvements.

After thinking about this a bit, I think @churchyard's / @carlwgeorge's proposal is the appropriate solution. The biggest critique was that it's not clear if this is just a temporary solution for this one problem, or if it is supposed to be general. Let's make it general.

Proposal: modularity SHOULD NOT be used for libraries and other packages which form building blocks for other packages and may be installed in parallell on the target systems (i.e. -devel headers and such do not need to be co-installable). Instead, the "traditional" solution with compat packages should be implemented following normal guidelines. Packages which depend on those libraries should declare versioned BuildRequires and Requires, if necessary. Note that package review is not required for compat packages.
Link: https://docs.fedoraproject.org/en-US/packaging-guidelines/Naming/#multiple
Link: https://docs.fedoraproject.org/en-US/packaging-guidelines/Conflicts/#_compat_package_conflicts

@zbyszek what about python and ruby bindings for libgit2?

@zbyszek also, this is no a solution since the solution would be to ban changing dependencies within one stream.

@zbyszek also, this is no a solution since the solution would be to ban changing dependencies within one stream.

Changing module dependencies within a stream is banned. It always has been, but the documentation had missed this note. It is fundamentally unchangeable at this time. This may change in the nebulous future, but as of now it should be treated as "not going to happen".

After thinking about this a bit, I think @churchyard's / @carlwgeorge's proposal is the appropriate solution. The biggest critique was that it's not clear if this is just a temporary solution for this one problem, or if it is supposed to be general. Let's make it general.
Proposal: modularity SHOULD NOT be used for libraries and other packages which form building blocks for other packages and may be installed in parallell on the target systems (i.e. -devel headers and such do not need to be co-installable). Instead, the "traditional" solution with compat packages should be implemented following normal guidelines. Packages which depend on those libraries should declare versioned BuildRequires and Requires, if necessary. Note that package review is not required for compat packages.
Link: https://docs.fedoraproject.org/en-US/packaging-guidelines/Naming/#multiple
Link: https://docs.fedoraproject.org/en-US/packaging-guidelines/Conflicts/#_compat_package_conflicts

Just to rephrase this so I know I understand what it means: it is acceptable for modularity to be used for libraries and packages for which exactly one version is permitted to be on the system, correct? So, for example, if something owns a well-known DBUS service name, it's okay to modularize that. Similarly something that owns a well-known network port.

So, the problem with libgit2 (which I only learned later) is that it actually is one of these cases. The low-level C library can coexist with a compat package, but many (most?) of its clients actually use one of the language bindings. These bindings are not parallel-installable (e.g. the python binding owns the import libgit2 statement). So this specific example would not be appropriate for the above guideline.

I'll use silver as the example here. The silver module provides a rolling stream. Upstream does do releases, but they have been known to bump major dependencies (libgit2) in point releases. As a result, Igor wants to be able to bump the module-level dependency for the silver:rolling stream from libgit2:0.27 to libgit2:0.28 as part of a basic update stream.

This, however, flies in the face of a core tenet of Modularity: "updates may not change enabled module streams". Now, there is some talk for the future (read: F32, maybe) of recording which module streams were selected by the user's direct choice and which were installed implicitly as dependencies or because they were the default stream and consider allowing the latter to be adjusted on updates/distro-upgrades. This is not implemented currently (nor scoped for inclusion in any Fedora release yet).

Part of the problem with the above plans is that updates run into issues when silver updates to depending on libgit2:0.28 and bat lags behind, still requiring libgit2:0.27. Module-level dependencies would mean that if this situation exists in the repositories and you have both silver and bat enabled locally, you can't update silver (possibly failing the transaction, depending on the --best flag).

So what do we do here?

Given that libgit2 and its bindings own a well-known name on the system, we could just not modularize libgit2 at all and treat it as part of the platform. This would mean that silver and bat would be limited to waiting for a major Fedora release to move to the new version.

Another option is to choose between a variety of bundling approaches. silver and bat could rename the libgit2 shared object and link against them in a private location. If it's reasonable to assume that they would rarely be installed together, we could keep libgit2 modularized and use module-inclusion instead of module-dependency to pull them in. (Essentially, all of the output contents of the libgit2:<stream> would become part of the output of the silver:rolling stream. You just wouldn't be able to enable silver:rolling, bat:rolling or libgit2:<anything> at the same time.

Frankly (as much as I dislike bundling), I think silver and bat carrying a private copy of the library and having libgit2 cease to be a module dependency is the easiest solution here.

Do bat and silver use Python/Perl/etc. bindings or the C library?

Do bat and silver use Python/Perl/etc. bindings or the C library?

I'm reasonably certain they just use the C library, but @ignatenkobrain is the authority there.

This, however, flies in the face of a core tenet of Modularity: "updates may not change enabled module streams". Now, there is some talk for the future (read: F32, maybe) of recording which module streams were selected by the user's direct choice and which were installed implicitly as dependencies or because they were the default stream and consider allowing the latter to be adjusted on updates/distro-upgrades. This is not implemented currently (nor scoped for inclusion in any Fedora release yet).

I'll be frank here, why did we let modularity get implemented in Fedora? Modules in Fedora are a terrible idea if dependencies can literally never change. That's fundamentally broken behavior. I also recall a previous ticket where this question was asked. It sounds like the answer here is "let everyone have dependency hell".

This is a terrible regression and I'm disappointed (but not surprised) that this wound up happening.

I think I'm going to have to suggest that we don't allow modules for most applications and all libraries unless the entire stack is being built and switched. That fundamentally makes modules prohibitively expensive for people to use, though. And it makes it impossible for most people to test building modules locally, because they will be stuck waiting multiple days for builds. At this point, I don't know if we've kept any of the efficiency gains we had from being a binary distribution rather than a source one.

Please tell me I'm wrong, because my analysis of this situation seems to indicate we've royally screwed up the Fedora case for modules.

I think I'm going to have to suggest that we don't allow modules for most applications and all libraries unless the entire stack is being built and switched.

This means I'm asking that unless modules are being used to create things like flatpaks or snaps, it should probably require FESCo approval before creating it. The user experience with modules is noticeably worse than regular RPMs, and the developer experience is equally terrible.

I'm reasonably certain they just use the C library

Then I don't see the point in bundling, when we an regularly package them and use them packaged.

why did we let modularity get implemented in Fedora

AFAIR the whole idea was presented as "multiple available versions for users to pick from" not as "we'll fundamentally screw the entire distribution" which brings us back to https://pagure.io/fesco/issue/2114 - we allowed modularity but we haven't got an idea what we want to use it for, we let terrible things happen and now everybody is afraid to pull the breaks.

why did we let modularity get implemented in Fedora

AFAIR the whole idea was presented as "multiple available versions for users to pick from" not as "we'll fundamentally screw the entire distribution" which brings us back to https://pagure.io/fesco/issue/2114 - we allowed modularity but we haven't got an idea what we want to use it for, we let terrible things happen and now everybody is afraid to pull the breaks.

I think you're vastly overstating your case here. We didn't set out to "fundamentally screw the entire distribution", and I think that such inflammatory language doesn't help the situation.

Yes, Modularity has warts. There are many situations we didn't account for when we developed it. It is absolutely impossible for any group of humans to predict all of the possible situations they might have to face. We are dealing with the natural growing pains of innovation: we try something, see what breaks, fix it and repeat. Asserting that we have to abandon things entirely because they don't work perfectly on the first try is unreasonable.

You're probably right about the libgit2 situation. Thinking about it more, it probably makes the most sense to pull the C libraries (and compat versions) out of the modules and back into the main distro and then make the modules solely for selecting which bindings are owning the well-known name. That's a good, balanced approach to the specific problem.

I think I'm going to have to suggest that we don't allow modules for most applications and all libraries unless the entire stack is being built and switched.

You're probably right that we opened the flood-gates too early, before we really knew what all of the edge-cases look like. I'm sure we still don't. I think this proposal is overstating the problem though; there are still cases (Node.js, for one) that doesn't fit this mold.

I'm open to the idea that we restore the requirement for some level of module review (which would have to include explaining each stream's compatibility guarantee and upgrade plan). We originally decided to drop module review on the grounds that we felt that the primary purpose of review was to ensure license compatibility, but it seems we really need to have a process for ensuring that packagers understand the lifecycle requirements.

I'm going to ping @psabata @langdon @asamalik and @imcleod here as interested parties in such a review process.

This means I'm asking that unless modules are being used to create things like flatpaks or snaps, it should probably require FESCo approval before creating it. The user experience with modules is noticeably worse than regular RPMs, and the developer experience is equally terrible.

Saying something is "terrible" or "noticeably worse" without explaining what that means is not very helpful. I absolutely agree there is room for improvement, but we need to clearly identify the pain-points and see what we can do about them.

Do bat and silver use Python/Perl/etc. bindings or the C library?

They use the rust-git2 crates which use the c library, no python or perl deps.

So, the problem with libgit2 (which I only learned later) is that it actually is one of these cases. The low-level C library can coexist with a compat package, but many (most?) of its clients actually use one of the language bindings. These bindings are not parallel-installable (e.g. the python binding owns the import libgit2 statement).

Yes, the language binding are not parallel installable. But I don't think we need more than one version of the bindings, because the language bindings don't change the interface in backwards incompatible ways as much. For R-git2r, there are no packaged users. For python2-pygit2 likewise. For python3-pygit2 there's just one: pagure, and it doesn't specify a versioned dependency.

When upgrading libgit2, we have two choices wrt. python bindings:
1. not rebuild python-pygit2, so that it continues to depend on libgit2.so.27(), provided by the old libgit2 version. If it needs to be rebuild, make sure that there's a version constraint to keep using the old library.
2. update python-pygit2 and release in the same update as pygit2.
Which choice is appropriate depends on how compatible python3-pygit2 is, and it should be made following the normal Update Guidelines.

As a result, Igor wants to be able to bump the module-level dependency for the silver:rolling stream from libgit2:0.27 to libgit2:0.28 as part of a basic update stream.

... and this works great with the compat packages. It really becomes a non-issue, as long as somebody has done the work to provide both libgit2 and libgit2_0.27, but somebody already has.

Now, there is some talk for the future (read: F32, maybe) of recording which module streams were selected by the user's direct choice and which were installed implicitly as dependencies or because they were the default stream and consider allowing the latter to be adjusted on updates/distro-upgrades.

Yes. We need a solution quickly, and this kind of thing is at least months in the future. I think we should ignore any such possible changes for the purposes of current discussion. Instead, any proposals for future enhancements should be informed by our discussion and problems here.

Sorry if my language is inflammatory but we repeatedly see that issues arise and nobody seems to actually fix those things. Every issue is swept under the rug with temporary workarounds and nothing ever gets resolved properly. It seems that we all agree that things are "raw" and need to be resolved, yet do we actually have the engineering power to do so?

it probably makes the most sense to pull the C libraries (and compat versions) out of the modules and back into the main distro and then make the modules solely for selecting which bindings are owning the well-known name.

I don't see why the bindings should be in modules, as a general rule. It would be only useful if the interfaces provided by the bindings in different versions are very incompatible. Normally, I'd expect bindings to hide most of the incompatibilities in the underlying C library. So this would have to be considered on a case-by-case basis: mostly compatible — great, let's just have a normal package; incompatible, but limited set of users we can adjust at the same time — let's make a normal package and schedule coordinated updates; badly incompatible and many users — too bad, we have to make a module.

Saying something is "terrible" or "noticeably worse" without explaining what that means is not very helpful. I absolutely agree there is room for improvement, but we need to clearly identify the pain-points and see what we can do about them.

This whole ticket is both a packager and user pain point!

Modulemds still require people to figure out the whole dep chain for encapsulation, which most packagers have no good way to figure out. Yes, @ignatenkobrain automated that problem away specifically for Rust, but we need a generalized version of that for everyone.

The user experience dealing with upgrades is completely unintuitive. The wrong kind of interface is exposed for resolving module problems, making it difficult and unfriendly for dealing with stream swaps, upgrades, etc.

After thinking about this a bit, I think @churchyard's / @carlwgeorge's proposal is the appropriate solution.

Should I proceed with submitting a libgit2_0.27 package? Do we need libgit2_0.26 as well?

Should I proceed with submitting a libgit2_0.27 package? Do we need libgit2_0.26 as well?

Please wait for the discussions to finish. At this point a few more days of delay won't make things significantly worse.

You're probably right that we opened the flood-gates too early, before we really knew what all of the edge-cases look like. I'm sure we still don't. I think this proposal is overstating the problem though; there are still cases (Node.js, for one) that doesn't fit this mold.
I'm open to the idea that we restore the requirement for some level of module review (which would have to include explaining each stream's compatibility guarantee and upgrade plan). We originally decided to drop module review on the grounds that we felt that the primary purpose of review was to ensure license compatibility, but it seems we really need to have a process for ensuring that packagers understand the lifecycle requirements.
I'm going to ping @psabata @langdon @asamalik and @imcleod here as interested parties in such a review process.

@sgallagh I believe that providing guidance in our docs about module architecture, update requirements, and possibly an explicit list of things that modularity does solve and doesn't (including things that we're learning about here) could be a good first step.

I get that an additional review could help us prevent this in the future, but to be honest, I'm personally not in a huge favour of adding more steps to the process. Would we be even able to predict this with such review before we knew what would actually happen? But if similar things keep happening even after we have the docs, sure, let's start doing reviews. But I wouldn't do it just because of this one case (even though it's a very visible one).

Metadata Update from @churchyard:
- Issue tagged with: meeting

3 months ago

We have discussed this today on FESCo meeting:

AGREED: Accept the proposal to use compat packages for libgit2 and put 'dnf module reset …' work-around in Common Bugs. (+8, 0, -0) (ignatenkobrain, 16:09:56)

@carlwgeorge feel free to create those compat packages. Or I'll create them this weekend. Also add me as maintainer to them, please. Once those are built, I'll change bat/silver/exa/… modules to not depend on libgit2 module.

Metadata Update from @ignatenkobrain:
- Issue assigned to ignatenkobrain

3 months ago

Metadata Update from @ignatenkobrain:
- Issue untagged with: meeting

3 months ago

@ignatenkobrain Could you provide a brief status update in this ticket?

Oh right.

In F29/F30:

  • libgit2:0.27 is default stream
  • Non-modular libgit2 is 0.27.x
  • libgti2_0.28 package has been created

F31:

  • libgit2:0.28 is default stream
  • Non-modular libgit2 is 0.28.x
  • libgit2_0.27 package has been created

libgit2-0.28.x Obsoletes: libgit2_0.28 < … so that upgradepath works between F29/F30 and F31. But dnf module reset was not documented, so most likely usual distro-sync won't work. This problem is not specific to libgit2, but to any module which changes default stream between releases.

Most of the modules are rebuilt without modular dependency. There are few which are not done yet, but I hopefully will get some time to fix this in couple of weeks.

I guess we are good with this :)

Metadata Update from @ignatenkobrain:
- Issue close_status updated to: Accepted
- Issue status updated to: Closed (was: Open)

2 months ago

Login to comment on this ticket.

Metadata