#112 Discussion: Module lifecycles
Opened 3 months ago by asamalik. Modified 5 days ago

Discussion for the #3 Module life cycles epic.

This is very packager focused. I want to design the packager experience first before diving into implementation details.

Traditional branching and lifecycles:

There is a branch for every release. To ship a specific version of a package for a specific Fedora release, you simply push the right source into the right branch in dist-git.

Dist-git branches:

  • nodejs 8 source -> "f27" branch
  • nodejs 8 source -> "f28" branch
  • nodejs 10 source -> "f29" branch
  • nodejs 10 source -> "master" branch

What's available:

  • Fedora 27: nodejs 8
  • Fedora 28: nodejs 8
  • Fedora 29: nodejs 10
  • Fedora rawhide: nodejs 10

Lifecycle expectation: Package is being branched for new releases and built forever until retired in rawhide.

Very simple, yet limited: you can't have multiple versions in one release. And you might be pushing the same source into multiple branches.

Modular branching and lifecycles:

There are stream branches — structured in any way the packager desires — but typically based on major versions. You push each version only once to its respective branch. But what goes where?

Modules are a way of defining what goes where. One version can go to multiple Fedora releases, and one Fedora release can receive multiple versions. And defaults are a way of deciding which of the multiple version is available as a default — so traditional workflows keep functioning.

Dist-git branches:

  • nodejs source 8 -> "8" branch
  • nodejs source 10 -> "10" branch

NEW: Module definitions:

  • nodejs 8 -> Fedora 27, Fedora 28, Fedora 29, Fedora 30 which is rawhide now
  • nodejs 10 -> Fedora 28, Fedora 29, Fedora 30 which is rawhide now

NEW: Default definitions:

  • Fedora 27: nodejs 8
  • Fedora 28: nodejs 8
  • Fedora 29: nodejs 10
  • Fedora 30 (rawhide now): nodejs 10

What's available:

  • Fedora 27: nodejs 8 by default
  • Fedora 28: nodejs 8 by default, but also nodejs 10
  • Fedora 29: nodejs 10 by default, but also nodejs 8
  • Fedora rawhide: nodejs 10 by default, but also nodejs 8

However, we don't want you to manually list releases in your module definition and requiring you to update it every time there is a new release.

That's why you can declare:

  • nodejs 8 -> all Fedora releases
  • nodejs 10 -> all Fedora releases

But what to do when nodejs 8 reaches its EOL (end of life)? We still don't want to ask you to suddenly list "all Fedora releases up until now" in the module definition so it stops building for the next one. We need a way for you to define an EOL so the build system can figure this out automatically.

We also want to keep the Lifecycle expectation: Package is being built forever until retired.

Proposal

Edit 2019-02-07: I have removed the old proposal, and added a specific proposal for the format of the lifecycle definition to this comment: https://pagure.io/modularity/issue/112#comment-552614

I will follow up with additional proposals for the place to store the information, the mechanics of setting and updating that information, and policy around all of that.


Metadata Update from @asamalik:
- Issue tagged with: Meeting

3 months ago

Metadata Update from @asamalik:
- Issue assigned to asamalik

3 months ago

IMO EOL information can be where all SCM data is kept, in git.

I think that's generally reasonable, but:

  • We don't really have a place to store the EOL at the moment. It used to be PDC but...
  • Both GA and release EOL dates can slip

I think the easiest option would be just relying on the platform dependencies here but we'd like some input from other groups, such as FESCo and releng. I'll file a FESCo ticket for this.

we'll discuss this one with FESCo

Note we are building a PDC replacement called fpdc: https://fpdc.fedoraproject.org/ https://github.com/fedora-infra/fpdc It could well contain this info.

I believe that module EOL has to correspond to a particular Fedora release. We can't leave users hanging with no security updates should a vulnerability be newly discovered after module EOL.

There was some suggestion at todays FESCo meeting that people wanted to store EOL data and the like in dist-git directly.

What PDC/FPDC data do we currently use and would it make sense to store some or all in dist-git?

CC @cverna @abompard

There was some suggestion at todays FESCo meeting that people wanted to store EOL data and the like in dist-git directly.
What PDC/FPDC data do we currently use and would it make sense to store some or all in dist-git?
CC @cverna @abompard

So currently in FPDC we store the following information about releases see --> https://fpdc.stg.fedoraproject.org/api/v1/release

I did not read the full proposal so I am not sure if that would be useful or not :smile:

I believe that module EOL has to correspond to a particular Fedora release. We can't leave users hanging with no security updates should a vulnerability be newly discovered after module EOL.

I'm with @jforbes. More onto the same topic:

Let's say a EOL of module stream is scheduled to 2020-01-01. When this day comes, what actually happens? If we just stop building, users are left with unmaintained (possibly defective or dangerous) software. How do we inform them? Do we remove the software (that's wicked)? Do we leave them in false sense of security and stability (that's evil)? In theory, if we could display the information in a very easily discoverable way (aka the users get constant notifications about the EOL when it approaches), then maybe. But that's not the case.

Also, IMHO defining the EOL as a date but making it effective aligned with Fedora release EOL is confusing (both for maintainers and for "readers" of such information). Hence:

I propose there is a way to declare an EOL for module streams in only one possible format: A specific Fedora release (as in: the last Fedora release that will have this module).

Side note: What happens with actuall installations after the stream EOLs? Traditionally, when we retire packages, we either let them rot (I hate that) or we obsolete them from fedora-obsolete-packages (and remove them from users installations, other people hate that). IIRC the current guideline is: obsolete them if they can cause broken dependencies or similar problems.
Dependency problems can happen with modules as well: Suppose I have a nodejs-8 module-stream that builds against system version of libuv (artificial example). As long as nodejs-8 is supported (before EOL), we rebuild it for any Fedoa release. If libuv is bumped, we rebuild nodejs-8. However, once nodejs-8 goes EOL in Fedora N and we bump libuv in Fedora N+k, we no longer rebuild nodejs-8 and the users who have it installed/enabled will have trouble upgrading to Fedora N+k.

Packages that are not in a module are not being built, so they don't really need an EOL declaration, because they don't exist from the user's perspective.

This assumes that everything is in modules. That's not really the case yet.

I propose there is a way to declare an EOL for module streams in only one possible format: A specific Fedora release

Yeah, I think that's reasonable.

We could provide an alternative format where a date is specified, with the assumption that if the EOL of the Fedora release slips, the EOL date of the module slips automatically too. But that might be too much magic. So if people are strongly on the side of "a specific fedora release" I can agree with that.

Side note: What happens with actuall installations after the stream EOLs?

I think we should follow the same policy as with "ursine" packages, but with the preference to add Obsoletes for EOL streams more aggressively than for ursine packages. I.e. unless there's a good reason to allow people to keep the EOL stream, add the stream to fedora-obsolete-package.

Can nonmodular package even obsolete modular packages?

To move this forward, I’m updating the proposal with a specific format of the EOL definition. This is only about the format, I’ll follow up with where to store it and how to set / update the information, including a policy around all of that.

Proposal of the lifecycle format

I propose that a module lifeycle is defined as an EOL (end of life) information mapping to a specific Fedora release and/or RHEL minor release for EPEL.

A single module stream will have two lifecycle definitions associated with it, one for Fedora and one for EPEL. Separation of these two is necessary as their EOL are not aligned, and I want to prevent module streams reaching their EOL mid-release.

Besides a specific EOL, I’m also allowing two additional values: “build forever” and “don’t build”.

The format could look similar to the following pseudo-syntax:

EOL:
    - Fedora: <VAL>
    - EPEL: <VAL>

… where <VAL> is either True to indicate “build forever”, a specific release such as 35 or 8.5 to indicate an EOL, or False to indicate “don’t build”. Not setting an EOL would implicitly default to True for both Fedora and EPEL.

Because we might not want to store a boolean value and a string in a single field, the format could be extended to separate them. However, the exact syntax is not necessarily in scope here.

Scenarios

I include some example scenarios to demonstrate why I propose such format.

Scenario 1: Fedora forever

“I want to build this module stream only for Fedora (not EPEL) forever.”

EOL:
    - Fedora: true
    - EPEL: false

... or:

EOL:
    - EPEL: false

Scenario 2: Fedora until f35

“I want to build this module stream only for Fedora (not EPEL) but only until Fedora 35.”

EOL:
    - Fedora: “f35”
    - EPEL: false

Scenario 3: Only EPEL until 8.5

“I want to build this module stream only for EPEL (not Fedora).”

EOL:
    - Fedora: false
    - EPEL: “8.5”

Scenario 4: Fedora until f35, EPEL until RHEL 8.5

“I want to build this module until Fedora 35 and RHEL 8.5.”

EOL:
    - Fedora: “f35”
    - EPEL: “8.5”

Scenario 5: Everywhere forever

“I want to build this module stream forever in both Fedora and EPEL. I might set a specific EOL value later.”

EOL:
- Fedora: True
- EPEL: True

Providing no EOL information would have the same effect.

EOL: False

I read as: EOL did not happen yet.

EOL: True

I read as: already EOL

Yeah, the reverse meaning with EOL is too confusing. I tried to figure out a mechanism where different terms are used, keeping the "EOL" section, but this is hard. The problem is that "EOL" is a negative term, i.e. something like all those NoFeature= options. It's usually much better to switch the option name to positive, i.e. Feature=. In this particular case, the opposite of EOL is "Support" or "Build-until" or something like this. Your five scenarios then become:

# 1
Build-until:
     - Fedora: "always"
# 2
Build-until:
     - Fedora: "f35"
# 3 
Build-until:
    - EPEL: “8.5”
# 4
Build-until:
    - Fedora: “f35”
    - EPEL: “8.5”
# 5
Build-until:
    - Fedora: “always”
    - EPEL: “always”

What about not setting it would mean "build forever" and setting it to 0 would mean "don't build"?

# 1
EOL:
     - EPEL: "0"
# 2
EOL:
     - Fedora: "f35"
     - EPEL: "0"
# 3 
EOL:
    - Fedora: "0"
    - EPEL: “8.5”
# 4
EOL:
    - Fedora: “f35”
    - EPEL: “8.5”
# 5
(nothing) 

So, to comment on the intent (rather than the syntax), the proposal is essentially:
Allow maintainers to specify that a module stream will live until the EOL date of a particular Fedora release or EPEL minor release, with special cases for "just keep building until I say otherwise"?

If so, I think that makes good sense and I'm +1 on it. The exact format we store this data in can be worked out once we agree on that.

+1 to @sgallagh

What about not setting it would mean "build forever" and setting it to 0 would mean "don't build"?

This still has the problem that what is not shown is more important than what is. E.g. imagine that we want to add a new release output, e.g. EPEL-for-opensuse. Now all modules are implicitly declared to support it. This is bound to be confusing to newcomers and even to advanced packagers who do a quick glance.

Thanks for the feedback.

This still has the problem that what is not shown is more important than what is. E.g. imagine that we want to add a new release output, e.g. EPEL-for-opensuse. Now all modules are implicitly declared to support it. This is bound to be confusing to newcomers and even to advanced packagers who do a quick glance.

@zbyszek Good point, however, look at it from the other side. We're not very likely to introduce things like EPEL-for-opensuse and such that often, but we do expect people to create new modules or module streams. The way I propose to handle the EOL means that their new module just builds, without the extra step of marking it as "not EOL" that you propose. Not introducing new steps (or even removing or simplifying the existing) for newcomers is quite important to me.

Login to comment on this ticket.

Metadata