#61 proposal for a new modularity objective
Closed 2 years ago by bcotton. Opened 2 years ago by langdon.
Fedora-Council/ langdon/council-docs modularity-objective-2019  into  master

@@ -40,24 +40,24 @@ 

  

  '''

  

- === Fedora Modularization — The Release

+ === Fedora Modularization — Modularity Stabilization & Improvements

  

  image::modularity-badge.png[float="right"]

  

  Summary::

- Implement a new https://docs.pagure.org/modularity/[modular Fedora] design, building on lessons learned in prototype phase.

+ As https://docs.pagure.org/modularity/[Modularity] in Fedora comes in to widespread use, the Modularity Team would like to address issues discovered and improve the experience.

  

  Objective Lead::

  https://fedoraproject.org/wiki/User:Langdon[Langdon White]

  

  Timeframe::

- Release with F28, improvements in F29, with publicity to follow.

+ Ongoing changes and improvements F31 through F33.

  

  Latest Status Report::

  

  Objective Details::

- https://fedoraproject.org/wiki/Objectives/Fedora_Modularization_%E2%80%94_The_Release[Fedora Modularization — The Release]

+ xref:objectives/objective-modularity-f30-f33.adoc[Fedora Modularization -- Modularity & Stabilization Improvements]

  

  Documentation::

  https://docs.pagure.org/modularity/

@@ -0,0 +1,75 @@ 

+ = Objective: Modularity Stabilization & Improvements

+ 

+ == Goal

+ 

+ Modularity has landed in Fedora and we are starting to see gaps in the implementation and the packager experience.

+ As a result, we have identified a number of projects that will improve the usefulness of Modularity and the experience of creating modules for packagers.

+ 

+ == Detailed Goals

+ 

+ At present, the buildroot can not contain content provided by modules

+ 

+ As a result, the content that may be turned in to modules is limited.

+ One of the Objective efforts, nicknamed “Ursa Prime,” focuses on supporting modules in the buildroot.

This is technically possible for a long time. Mock can install and enable modules in buildroot. Therefore it is just a matter of having the right config in Koji. I would be more specific here. What you are trying to do, technically.

+ 

+ Many packagers find the effort to create and support modules to be too high.

+ Despite having a number of tools to support module creators and maintainers, they are not well integrated into the Fedora ecosystem.

There must be strong restrictions to create modules, because each module will increase demands for release engineering, infrastructure and testing.
At present there are multiple reports about broken dependencies between nonmodular content and enabled modules and their dependencies on systems that never used dnf module commands (consequence of enablement of defaults and their dependencies).
Wee need to focus on QE and testing all module streams combinations to ensure integrity of distribution (modular repoclosure).

+ There are also gaps in these tools for some use cases. We will focus efforts on improving integration & documentation and addressing gaps in the tooling.

+ 

+ The Modularity Team has also recieved a number of documentation requests.

+ Requests have included clarification architecture, modular approaches, and when and how to use tools.

+ During this objective effort, we would like to address these documentation gaps and solicit new ones as much as possible.

+ 

+ == Deliverables

+ 

+ Fedora 31 Cycle (Ending Oct '19):

+ 

+  - Simple local, offline module builds and documentation

+  - Hackfest at Flock

+ 

+ Fedora 32 Cycle (Ending May '20):

Any significant changes must be ready before 2020-02-25 - Beta freeze

+ 

+ - Module Tagging Service: auto tags builds in to multiple releases of Fedora

+ - Ursa Prime: Allow the default streams of modules to make their RPMs available in the buildroot

+ - Rolling Defaults: When a user makes "no choice," they get the default stream on upgrade.

We need to provide an alternative approach how to resolve system-upgrade in fedora. I do not agree to use "Rolling Defaults", because it is a fragile construct that makes modularity more complicate and not transparent.

+ When a user makes a choice, even matching the default stream, they will stay on that stream on upgrade.

+ - EPEL supports modules

+ - Hackfest (to be scheduled)

+ 

+ Fedora 33 Cycle (Ending Oct '20):

+ 

+ - Support for changing dependencies within a stream

+ - Support for RPM module header in dnf

+   - do the right thing on missing modulemd

+   - modulemd caching

Is this a fail-safe mechanism? The fail-safe mechanism is ready to deploy.

+ - Only active module streams considered for dependency solving

I do not understand that, sorry.

+ - Complex local, offline module builds (alternate archs, Fedora versions, etc)

+ - Module scratch builds

+ - Hackfest (to be scheduled)

+ 

+ == Modularity Hackfests

+ 

+ While somewhat open-ended, the intent of these get togethers is to have 1-2 days to collaborate, in person, with the team.

+ We also would like to use these events to explicitly elicit feedback from the community.

+ We plan to follow the model that has been successful in the past with one or two non-core team members joining for the entirety of the hackfest and/or a 2 hour session with a number of non-core team members participating.

+ 

+ == Modularity Team

+ 

+ This group was established as part of a prior phase of the Modularity Objective, and it will continue.

+ However, in light of the Council change to the “team” nomenclature, the group is now called the “Modularity Team.”

+ Nothing else about the group’s operations have changed.

+ 

+ == Objective Lead

+ 

+ https://fedoraproject.org/wiki/User:Langdon[Langdon White]

+  (langdon)

+ 

+ == Timeframe

+ We are outlining this Objective to cover the F31 through F33 development cycles as described in the Deliverables above.

+ 

+ == History

+ Follows from :

+ 

+ * https://fedoraproject.org/wiki/Objectives/Fedora_Modularization,_Prototype_Phase[Fedora Modularization, Prototype Phase]

+ * https://fedoraproject.org/wiki/Objectives/Fedora_Modularization,_Requirements_Phase[Fedora Modularization, Requirements Phase]

+ * https://fedoraproject.org/wiki/Objectives/Fedora_Modularization_%E2%80%94_The_Release[Fedora Modularization — The Release]

also makes a new objectives folder for these files

I'd like to see a vote on this delayed until after Flock and when the TBD sections are filled out based on the Flock hackfest.

rebased onto 210189a

2 years ago

1 new commit added

  • adds explanation for modularity hackfests
2 years ago

I like this proposal, however I would like to point to one thing.

In the goals section you are talking about improving packager experience and experience of creating modules for packagers. But I could not find anywhere in deliverables anything about "create tooling for automated creation of modules which is run as part of Fedora". For example:

  • I push to rust-regex rpm repo
  • Some tooling detects that it is used by ripgrep and zola modules
  • And regenerates modulemd (or delivers the change in some other way)
  • And builds new version of a module

Nowadays, it consumes quite a lot of time for me to maintain 30+ Rust modules.

Do you think it is feasible to add on the roadmap?

It seems like dnf will need a lot of changes. Is the dnf team on board? cc @jmracek

Before encouraging people to make modules, can you please resolve the upgrades first? https://pagure.io/modularity/issue/151
This has been the most common issue during upgrades from F30 to F31 and there is known workaround but not known general solution yet.

This is technically possible for a long time. Mock can install and enable modules in buildroot. Therefore it is just a matter of having the right config in Koji. I would be more specific here. What you are trying to do, technically.

Can we make this part of the objective? https://pagure.io/fesco/issue/2114

Should we stop modularizing libraries and focus on "leaf stacks" instead?

There must be strong restrictions to create modules, because each module will increase demands for release engineering, infrastructure and testing.
At present there are multiple reports about broken dependencies between nonmodular content and enabled modules and their dependencies on systems that never used dnf module commands (consequence of enablement of defaults and their dependencies).
Wee need to focus on QE and testing all module streams combinations to ensure integrity of distribution (modular repoclosure).

We need to provide an alternative approach how to resolve system-upgrade in fedora. I do not agree to use "Rolling Defaults", because it is a fragile construct that makes modularity more complicate and not transparent.

Is this a fail-safe mechanism? The fail-safe mechanism is ready to deploy.

Any significant changes must be ready before 2020-02-25 - Beta freeze

There must be strong restrictions to create modules, because each module will increase demands for release engineering, infrastructure and testing.
At present there are multiple reports about broken dependencies between nonmodular content and enabled modules and their dependencies on systems that never used dnf module commands (consequence of enablement of defaults and their dependencies).
Wee need to focus on QE and testing all module streams combinations to ensure integrity of distribution (modular repoclosure).

+1

Complaints about dependency issues with enabled modules, when the users did not know modularity existed and they never had used any modular command, are numerous. Maybe modules should only be used when explicitly required by the user? Such approach would prevent a lot of discussed problems per se.

We need to provide an alternative approach how to resolve system-upgrade in fedora. I do not agree to use "Rolling Defaults", because it is a fragile construct that makes modularity more complicate and not transparent.

If we did not allow modules to have a default stream, we could still maintain the ursine packages for the default version of the available software. In that case, people who wouldn't be interested in modules could remain on a non-modular content (they are trying to do it anyway), without any problems with upgrades. The others, who'd like to use modules, could do it and they would be expected to know how to solve the upgrades by themselves.

I feel, that there is a longing for modular only system, but I am not sure that we are able to test that different modular streams do not break mutual dependencies. With most of the software in modules, with possibly multiple streams, there will be bazillion of different combinations and I am afraid we do not have enough human and machine power to test everything thoroughly.

What we also need urgently are clear guidelines how to create modules, so that the streams and profiles are correctly set (or even named). These guidelines should also be enforced and modules that do not meet the criteria should not be let into the system ( => gating).

Maybe introducing an automated tool that would take care of it is a good idea, so I am in favor of +1 to @ignatenkobrain .

If we abandon the idea of default streams everything will indeed get much easier for the users. As a bonus, we don't have to do ursa prime.

That indeed has my full support. The default version should be nonmodular. Modules should provide alternate versions for people who want them.

That indeed has my full support. The default version should be nonmodular. Modules should provide alternate versions for people who want them.

The problem with this approach that this "alternative version" maintenance workflow would be different. We either need to have same workflow for standard and alternative versions or we should not have modularity. Trying to be somewhere in the middle is not going to end up well.

We can fix the users experience + regular packager experience first and worry about the modular contributor experience later? If the experience is "nah, this is too complicated", we would still have the default nonmodular version.

We can fix the users experience + regular packager experience first and worry about the modular contributor experience later? If the experience is "nah, this is too complicated", we would still have the default nonmodular version.

I believe that users experience should be our mantra, because we do not want to make users leaving for other distros. So, for every technology we want to adopt into Fedora, we must nourish it into perfectness or think about something else.

If staying in the middle only represents some extra work to package the modules, I think it is a reasonable cost. The non-modular workflow is known, experienced, tested and trusted, so we do not have to do much in this field and we can focus on making the modular workflow as pleasant as possible while still staying in the middle.

By the way, I like staying in the middle because it is far from every extreme.

The plan also forget to talk about:
1. Module or stream removal process from distribution (EOL)
2. Life-cycle of the stream
3. Modular upgrade path - missing documentation and tooling
4. What about module:streams obsoletes?

Pull-Request has been closed by bcotton

2 years ago

The ownership of Modularity development within Red Hat has changed. So we're retiring the current Modularity objective and work with the new team to develop a new one.