#108 How do we keep rawhide sane? (re: forcing people to latest modules)
Closed: Fixed 7 days ago by ignatenkobrain. Opened 2 months ago by ignatenkobrain.

Today in rawhide libgit2:0.27 is default, but in F30 or F31 I will make a switch to libgit2:0.28, but rawhide users won't switch because they already have 0.27 enabled.

What do we do with this?


Metadata Update from @ignatenkobrain:
- Issue tagged with: Meeting

2 months ago

This is intentional behavior. The Module Prime Directive is that the users' enabled streams are never changed for them. If you need a "rolling" stream for libgit2, it would be preferable for the default stream to be named something like "stable" which could upgrade at need and then also provide the 0.27 and 0.28 streams for people who have a reason to lock onto those versions.

The other option, of course, is for libgit2 to have no default stream in the module repositories, but only to provide each of the non-default streams. Then the non-modular RPM can continue to follow the Fedora Stable Updates Policy, as would be traditional and the modules supplement this by having an opt-in way to pick the preferred version.

While it is intentional, it complicates things for maintainers who want to move their content solely to modules, especially later on with Ursa Major available and non-leaf packages. Rawhide users (and anyone during distribution upgrade) would stick with stream-provided packages that are incompatible with the ursine set, leading to broken dependencies or hidden bugs.

I suppose having rolling streams for all of these things would help, even though I find kinda "meh".

While it is intentional, it complicates things for maintainers who want to move their content solely to modules, especially later on with Ursa Major available and non-leaf packages. Rawhide users (and anyone during distribution upgrade) would stick with stream-provided packages that are incompatible with the ursine set, leading to broken dependencies or hidden bugs.

Right, which is why I suggested that this particular case is better solved by NOT using a module for the "default" version.

I suppose having rolling streams for all of these things would help, even though I find kinda "meh".

The problem with that is that you either have to roll the stream for all releases or else carry a different "view" of what the "stable" branch looks like on F28, F29, F30, etc. instead of simply using MSE to build the same one for all of them.

I think my suggestion above of only using modules for "locking" versions like this is probably the best approach for this specific use-case.

but how do you update distro? imagine that F29 has libgit2:0.27 default while F31 will have 0.28, how are users are supposed to switch to new default?

but how do you update distro? imagine that F29 has libgit2:0.27 default while F31 will have 0.28, how are users are supposed to switch to new default?

They'll need to switch streams either before or after the upgrade process (one or the other state must support both streams).

I've been thinking about this for a while, and I do think we need to add an option on distro-upgrades to move between changed default streams, but we need to design this and figure out what the appropriate behaviors will be.

Another workaround I can think of (and which I've seen in modules) is to have a latest stream which tracks latest upstream versions.

I have some thoughts on this and I'm going to try to organize them and then make a first-draft of a proposal that we can discuss.

First, I'll restate the problem so I'm sure we're on the same page.

Let's assume that we have a module "coolmodule" that has three streams, each of which provides an RPM called "coolframework". I'll call them "LTS", "stable-2.0" and "stable-3.0" (don't sweat the names too much, they're just for illustration). During the development of Fedora 30, the "LTS" and "stable-1.0" streams exist. The LTS branch is an older, long-term-stable version of coolmodule. We want to offer it for users who are running older applications that rely on it, but for Fedora 30 we want users to get the "stable-2.0" stream by default. So we make coolmodule:stable-2.0 the default stream for Fedora 30 and make coolmodule:LTS available as a non-default stream.

During the development phase of Fedora 31, the stable-3.0 module is created and added to both Fedora 30 and Fedora 31. In Fedora 31, we make the default stream coolmodule:stable-3.0. On new installations on Fedora 31, uses will get coolframework from the coolmodule:stable-3.0 stream and on Fedora 30 they will continue to get coolframework from the coolmodule:stable-2.0 stream, though the stable-3.0 stream will be available if they want to switch.

Concerns with upgrades:

  1. Traditional users of Fedora expect to get the latest packages on update. They would probably expect that the default behavior on an upgrade from Fedora 30 to Fedora 31 would be to update to the stable-3.0 stream and get coolframework upgraded from that stream.
  2. What do we do when a currently-selected stream does not exist on the upgrade target platform? (The stream has gone EOL).
  3. What do we do if the user continues to need the currently-selected stream because their application relies on it? (Two cases: one where they have explicitly selected coolmodule:LTS, the other where their application uses the default coolmodule:stable-2.0 but is incompatible with coolmodule:stable-3.0)

So here's the first-draft of a new proposal (suggestions encouraged):

  1. (Technical) We add a new state to module streams: "locked". On upgrade, if an enabled stream is marked as "locked", the upgrade must remain on that stream after the upgrade or fail before making changes if the target platform no longer offers that stream.
  2. (Technical) On upgrade, if a stream is the default stream and the stream is not marked as "locked", then the user will be transitioned to the default stream of the target platform.
    • The case of a stream that is not the default stream and not marked as "locked" is special and discussed in the "Open Questions" section below.
  3. (Policy) We set the "locked" state on a module stream when one of the following occurs:
    a. The user performs a dnf module operation that includes the stream explicitly on the command-line (e.g. dnf module enable coolmodule:LTS or dnf install @coolmodule:stable-1.0. Note that explicitly selecting the same stream that would be the default stream is also locking it.
    b. The user calls dnf module lock module:stream explicitly. (New DNF command to be added.)

Open Questions:

  • What about modules enabled as a dependency? E.g. coolmodule:stable-2.0 depends on languagemodule:1.0, but coolmodule:stable-3.0 depends on languagemodule:2.0. The default in both F30 and F31 is languagemodule:LTS (so in neither case is it depending on a default stream).
    • I think DNF should attempt to resolve this like other dependencies and refuse to perform the distro upgrade if it cannot be resolved manually. Put another way, if a stream was enabled implicitly it will not be marked as "locked" and thus should be resolved automatically.

Thanks for the writeup.

I'll just note three things:

  • The proposal doesn't address Rawhide; would new defaults appearing in the Rawhide repo trigger the distro upgrade scenario or would you suggest something different?

  • You only mention dnf module install but nothing for dnf module enable. Are you suggesting the same behavior for enable? (I would think so)

  • Even the current state with "somewhat enabled" defaults and "explicitly enabled" modules is rather confusing; I worry this will make it worse. It we decide more states are really necessary, let's ensure it's as intuitive as possible. And let's make sure the same commands always do the same thing, regardless of what the current defaults are.

Thanks for the writeup.
I'll just note three things:

The proposal doesn't address Rawhide; would new defaults appearing in the Rawhide repo trigger the distro upgrade scenario or would you suggest something different?

Yeah, thinking more on it, this would be a way to resolve the general case of module defaults changing. So I guess what we'd want to do is make sure DNF knows which modules are default before and after any transaction and treat it accordingly.

You only mention dnf module install but nothing for dnf module enable. Are you suggesting the same behavior for enable? (I would think so)

Whoops. I rephrased that sentence a few times and meant to end with "The user performs a dnf module operation". So yes, any action that results in a module getting enabled, if the stream was specified manually it should be treated as locked.

Even the current state with "somewhat enabled" defaults and "explicitly enabled" modules is rather confusing; I worry this will make it worse. It we decide more states are really necessary, let's ensure it's as intuitive as possible. And let's make sure the same commands always do the same thing, regardless of what the current defaults are.

I agree and I think that once we get this plan to general agreement, we should involve a user-experience designer to figure out the exact syntax.

  1. (Technical) We add a new state to module streams: "locked". On upgrade, if a stream is marked as "locked", the upgrade must remain on that stream after the upgrade or fail before making changes if the target platform no longer offers that stream.

Just for completeness sake, this concerns the module stream as present on users' systems and is a result of user activity (as opposed to repo or build system side and decisions by the module maintainer).

For the record, I just edited my post above to clarify two points raised by @psabata and @nphilipp.

So it sounds like the proposal is: if the user doesn't specify a specific stream, they will get the default and any upgrades will follow the same experience they are used to today. But, if they specify a stream or explicitly leverage a lock command, they will stay on that stream across upgrades until they explicitly switch streams.

From a user experience perspective, this seems like a good solution to the issue and keeps things simple. I especially like the idea that specifying a stream implicitly locks to that stream until another user action is taken to change it, even across major updates. This doesn't require the user to learn or discover a new command to achieve the result they are looking for by choosing a specific stream. Do we even need an explicit command to lock into a stream or is specifying a stream enough. If we do, what should the command be? Is lock going to be confusing to users or imply something other than what we are describing?

In addition, I think that at the time of upgrade, we should be providing them with a list of the modules that are 'locked' so that they can decide to make changes to those if needed. And, of course, if their locked stream isn't supported in the upgrade, we should be asking them which new stream they want to move to.

So it sounds like the proposal is: if the user doesn't specify a specific stream, they will get the default and any upgrades will follow the same experience they are used to today. But, if they specify a stream or explicitly leverage a lock command, they will stay on that stream across upgrades until they explicitly switch streams.

Yes

From a user experience perspective, this seems like a good solution to the issue and keeps things simple. I especially like the idea that specifying a stream implicitly locks to that stream until another user action is taken to change it, even across major updates. This doesn't require the user to learn or discover a new command to achieve the result they are looking for by choosing a specific stream. Do we even need an explicit command to lock into a stream or is specifying a stream enough. If we do, what should the command be? Is lock going to be confusing to users or imply something other than what we are describing?

That's a fair point; we need the "lock" under the hood as a technical implementation detail, but it doesn't necessarily need to filter up to the user commands. It might be enough to say that it's set if you use a stream-specific command (e.g. dnf module install modulename:stream) and that we have a separate command like dnf module default-stream modulename (i'm not married to that command...) that turns it off.

In addition, I think that at the time of upgrade, we should be providing them with a list of the modules that are 'locked' so that they can decide to make changes to those if needed. And, of course, if their locked stream isn't supported in the upgrade, we should be asking them which new stream they want to move to.

If the experience is not too intrusive, maybe we can display the locked streams, but I worry that it's giving them information they don't need if there isn't a conflict to resolve.

I think we certainly want to provide advice if we are in a situation where the target platform no longer offers the locked stream, but I don't know if we necessarily want to try to have UX in the upgrade itself for that. Though I suppose we can't always guarantee overlap between streams on upgraded platforms. Suggestions on how to display this would be nice.

I think it might be helpful to outline a number of specific use-cases that we want to solve, so here they are:

Use Case 1: Traditional Fedora Workflow

I, as a user of Fedora since the dawn of time, want to continue operating exactly as I always have, largely unaware of modules. I accept that modules exist and that I may see mention of them in the package manager (such as those coming from default streams), but I do not want to interact with them ever.

Expected Release Upgrade Behavior

When moving from Fedora N to Fedora N+1 (or N+2), I expect upgrades to work the same as they traditionally have. If there are packages on my system that were installed from default module streams, I expect that on a release upgrade they will be automatically updated to the version belonging to the default stream in the target system.

Use Case 2: The Important Application

I, as an IT administrator for my business, have a critical application that must not break between Fedora releases. I want to take advantage of performance, security and hardware-enablement features of a system upgrade, but I do not want my application to stop working because the target system has a too-new framework.

Expected Release Upgrade Behavior

In this case, the IT administrator has run dnf enable framework:1.0. On upgrade to Fedora N+1, this must not change to a new stream of framework. It must either stay on the existing stream on the new target release or else return a useful error notification prior to executing the upgrade that the stream is unavailable on the target system and provide some useful feedback on how to proceed if an upgrade is still desired (provide documentation on how to uninstall the module that cannot be upgraded or suggest switching it to a newer stream if possible).

Use Case 3: The Complex App

As an IT administrator using a complex application (I'll use FreeIPA as an example) that depends on other module streams, I want to be able to handle upgrades in a reasonable manner.

For example, let's say that the default stream for the tomcat module in Fedora 29 is '9'. However, FreeIPA is not yet compatible with version 9.x, so the freeipa module has a dependency on tomcat:8. The administrator installs FreeIPA with dnf module install freeipa:4.7 (knowing in advance that they will prefer to stay on this release as in Case 2).

On Fedora 30, the version of freeipa:4.7 has been certified to work with tomcat:9 and so its dependency has changed on that version.

Expected Release Upgrade Behavior

In this case, the expected behavior of the user is that upgrading to Fedora 30 while remaining on freeipa:4.7 would be for the implicit dependencies such as tomcat to be moved to the appropriate version during the upgrade. This implies that we must differentiate between modules installed explicitly and implicitly, since they will need to have different upgrade semantics.

I believe that my proposal above covers each of these cases, but if I have missed any important cases or details, please provide further information.

CC @dmach and @jmracek to make sure they are following this thread.

So, after reading the proposals and discussing it over jit.si today, I think this is generally good and makes sense. To sum it up:

  • we get three states, sort of: default (not truly enabled), enabled implicitly (via package installation or being pulled in as a dependency), and enabled explicitly (via module commands)
  • defaults are just about making packages available, no modules are enabled; these are updated automatically as defaults change in the repodata
  • implicitly enabled modules would follow the defaults unless they cannot, for instance if another module depends on the currently used stream
  • explicitly enabled modules are what we have today

This proposal would impact DNF -- it would need to distinguish between the two levels of enablement in its database (/etc/dnf/modules.d) and potentially in its depsolving mechanism. I would like to understand the impact here before giving my +1. We might also want to make the behavior configurable -- let's say a DNF configuration switch that chooses between the one proposed here (the potential future default) and the current, strict enablement only, provided there turns out to be a use case for it.

Furthermore we have at least one module that might change its stream outside of DNF's control, and that is platform. This happens both during distribution upgrade and in certain Rawhide scenarios. Here again I would like to understand what the impact of that would be and how we could ensure it fails reasonably, if at all.

We might also want to make the behavior configurable -- let's say a DNF configuration switch that chooses between the one proposed here (the potential future default) and the current, strict enablement only, provided there turns out to be a use case for it.

I agree with everything but this part. Adding a knob like this means an increase in QA effort as well as considerable ongoing maintenance to make sure we don't break either set of expectations. I think it's far better if we decide to use the proposed approach across the board. At worst, we add a command to DNF that says "lock all currently-enabled modules" and then those are thereafter just treated as "explicitly enabled". Then at least it's the same codepath to deal with.

I've read all of this, I've watched the meeting recording (I couldn't attend the meeting) and here's my take:

I agree with the general concept that I would simplistically summarize as:

  • If a user doesn't specify a stream, they get the default stream and they will stay on a default stream during a major distro upgrade — potentially switching to a new default stream.
  • If a user does specify a stream, they get that stream and they will stay on the specific stream during a major distro upgrade.

In other words:

  • Asking for a default always results in a default. Streams can be potentially changed during an upgrade.
  • Asking for a specific stream always results in the specific stream. Defaults don't matter with this choice.

I don't agree with the following:

  • Module commands lock you on a stream
  • Package commands give you a default

... at least the way I understand it which is:

(I assume the default for nodejs is 8)

  • dnf module install nodejs:8 — installs packages from the 8 stream, locks you on the 8 stream
  • dnf module install nodejs — installs packages from the 8 stream, locks you on the 8 stream
  • dnf install nodejs — installs packages from the 8 stream, keeps you on a default which can potentially switch streams in the future

I see a big disadvantage of this approach — it takes the convenient module commands (including the use case-based installation — profiles) from users who want to consume the default streams. That's basically the well known scenario Fedora users experience from the traditional model — when there is only one version.

The change I propose is as follows:

  • Specifying a stream locks you on a stream
  • Not specifying a stream gives you a default

... which means:

(I assume the default for nodejs is 8)

  • dnf module install nodejs:8 — installs packages from the 8 stream, locks you on the 8 stream
  • dnf module install nodejs — installs packages from the 8 stream, keeps you on a default which can potentially switch streams in the future
  • dnf install nodejs — installs packages from the 8 stream, keeps you on a default which can potentially switch streams in the future

Clarification: the functionality of the module install command is the same as of the module enable command.

Since the beginning, we say "If you choose a stream, you'll stay on that stream, even when the distribution upgrades. An explicit action is required to change a stream. But if you don't want to care, we have defaults." — I believe my proposed change actually fits this statement completely.

I talked about the user experience so far, but let me explain how I imagine the actual implementation:

Modules can be installed / enabled with a stream specified in one of the following three ways:

  1. Specific stream — This already exists.
  2. The default stream — This is new (clarified below).
  3. One of multiple streams — This already exists because of stream expansion and dependent modules (clarified below).

Clarification of "The default stream": Instead of remembering that a module is installed in a specific stream, DNF would remember that a module is installed in "a default stream" — whatever that actually means. This could result in potentially changing streams with upgrades — which is exactly what we want for defaults.

Clarification of the "One of multiple streams" covering Stephen's complex example:

When a module has a modular dependency, it can either depend on one specific stream, or one of multiple streams because it has been built using a stream expansion. For example, an application might work with both language-runtime:1.0 and language-runtime:2.0.

I propose a simple algorithm to choose the right stream of a dependency (or another way to put it: the right context of the application that requires the right stream of a dependency):

  • Rule 1: Always prefer a default.
    • If there are multiple choices and one of them is a default, choose the default.
    • This also applies during a system upgrade:
      • If a default changes and is compatible, switch to the new default.
      • If a default changes but is not compatible, stay on the current stream.
      • When a previously non-compatible default becomes compatible, or a new default appears, switch to that default.
  • Rule 2: No default means "just pick one".
    • If there are multiple viable choices and none of them is a default, pick one. This behavior is already well known to DNF. For example:
      • If I run dnf install /usr/bin/fooand two packages provide that file, DNF picks one of them.

I believe that implementing my proposal requires minimum changes in DNF:

Change 1: Implement the "The default stream" way of remembering which stream is installed.

Change 2 — although most of this should already work because stream expansion is nothing new: Implement the "One of multiple streams" way of remembering which stream is installed including the algorithm to choose the right stream and to re-evaluate the choice on system upgrades.


Conclusion:

I believe I have covered every scenario that the proposal you have put together in the meeting covers.

The benefit of mine is that it doesn't take away the convenient module command (including profiles) from users who want to consume defaults.

It is very easily explained, maps exactly to what we are already saying to people: "Asking for a stream gives you the stream forever. Don't wanna choose? We have defaults. Asking for a default gives you the default forever."

@psabata @sgallagh @langdon @dmach I would appreciate your input.

@asamalik Good analysis, the difference in the previously proposed behavior between using dnf module ... vs. traditional DNF commands flew right past me.

I concur with your modified proposal because its outcomes seem to carry the least surprise for the user: If a user doesn't specify the stream they probably won't expect it to be locked just because they used module commands, having to specify it if you want it locked feels natural to me.

I've read all of this, I've watched the meeting recording (I couldn't attend the meeting) and here's my take:
I agree with the general concept that I would simplistically summarize as:

Responses inline:

If a user doesn't specify a stream, they get the default stream and they will stay on a default stream during a major distro upgrade — potentially switching to a new default stream.
If a user does specify a stream, they get that stream and they will stay on the specific stream during a major distro upgrade.

In other words:

Asking for a default always results in a default. Streams can be potentially changed during an upgrade.
Asking for a specific stream always results in the specific stream. Defaults don't matter with this choice.

I don't agree with the following:

Module commands lock you on a stream
Package commands give you a default

... at least the way I understand it which is:
(I assume the default for nodejs is 8)

dnf module install nodejs:8 — installs packages from the 8 stream, locks you on the 8 stream
dnf module install nodejs — installs packages from the 8 stream, locks you on the 8 stream
dnf install nodejs — installs packages from the 8 stream, keeps you on a default which can potentially switch streams in the future

This is not the way it was intended. Specifically dnf module install nodejs and dnf install nodejs should be equivalent. In other words, both treats this as "keep on a default which can potentially change". If I was unclear in my summary, that's my fault.

tl;dr: Your statement was what I had intended to say above, so I agree with you.

So I suppose this is probably fine, although I'm worried about the different behavior of the same command depending on whether you specify the stream or not, that feels rather inconsistent and confusing.

@psabata Do you mean the dnf module install command? I see it as an installation command that installs the exact thing you ask it to install: either a specific stream, or a default stream. And it remembers your choice during upgrade.

No, I mean the difference between

dnf module install nodejs (doesn't lock the stream)

and

dnf module install nodejs:8 (does lock the stream)

No, I mean the difference between
dnf module install nodejs (doesn't lock the stream)
and
dnf module install nodejs:8 (does lock the stream)

While it's not perfect, I think this is still the closest we can get to sticking with user expectation. The rest will have to be documented carefully.

Do we not have something like dnf module lock <foo> or dnf module install --lock <foo>?

I really don't like the "lock" terminology here. It's confusing to me.

In my proposal I specifically talk about having "a specific stream" installed or "the default stream" installed. Both choices reflected during the major distro upgrade.

How is it confusing? You are requesting to stay on that stream until it's literally no longer possible to do so.

Here are the decisions from the previous two WG meetings:

  • if users don't specify a stream, they will get the current default one which, on upgrade of the system, will track the respective new default (+4, 2, -0)
  • "dnf update" should inform users about streams being switched and perform distro-sync on the affected modular package set, taking hotfixes into consideration (+3, 0, 0)

Find information about why we consider the "lock" terminology confusing in the log of last week's meeting.

Metadata Update from @nphilipp:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

7 days ago

I can see that the ticket is closed already, still I would like to add some thoughts.
What scares me a bit is that general behaviour is driven by rawhide needs.
I believe the proposed behaviour would be confusing for the user. Also running distro sync is also not good idea unless you are able to limit it specifically to the modular content as it could have undesired effect on other installed content. Did you consider introducing the concept of module stream obsoletes? I can imagine that specific stream could state that it obsoletes a different stream (assuming that the package migration is functional and safe and tested ... or we don't care). For rawhide the stream upgrade could be ensured on modulemd file level (e.g. by supporting wildcard/* syntax in stream obsoletes).

Did you consider introducing the concept of module stream obsoletes?

Please, no.

Obsoletes would mean that you won't be able to go backwards.

Assuming that you understand "obsoletes" in a way it works today for RPMs. However it may not be such strict for modules. Moreover, you could go backwards by overriding it locally.

Login to comment on this ticket.

Metadata