From 0d530703a08596f6019a83030327008d0424a7e5 Mon Sep 17 00:00:00 2001 From: Martin Curlej Date: Mar 08 2022 14:42:45 +0000 Subject: Documentation overhaul. Signed-off-by: Martin Curlej --- diff --git a/modules/ROOT/nav.adoc b/modules/ROOT/nav.adoc index 025e2cd..de69246 100644 --- a/modules/ROOT/nav.adoc +++ b/modules/ROOT/nav.adoc @@ -2,17 +2,11 @@ * xref:index.adoc[Introduction] * xref:community.adoc[Community] * xref:glossary.adoc[Glossary] -* xref:architecture.adoc[Core Concepts] -** xref:architecture/building.adoc[Building Software] -** xref:architecture/consuming.adoc[Consuming Software] -** xref:architecture/stream-expansion.adoc[Stream Expansion] -** xref:architecture/consuming/upgrade-paths.adoc[Upgrade Paths] -* **Basic usage** +include::{partialsdir}/core-concepts-links.adoc[tag=core-concepts-links] + +* **Consuming Modules** * xref:using-modules.adoc[Using Modules] -** xref:installing-modules.adoc[Installation and Discovery] -** xref:using-modules-switching-streams.adoc[Switching Streams] -** xref:removing-modules.adoc[Removing Modules] * xref:building-modules/index.adoc[Building Modules] ** xref:building-modules/fedora/index.adoc[Fedora] *** xref:building-modules/fedora/adding-new-modules.adoc[Adding New Modules] @@ -31,12 +25,9 @@ * **Advanced** * xref:hosting-modules.adoc[Creating Modular Repositories] -* xref:using-other-build-systems.adoc[Using Other Build Systems] -* xref:architecture/nsvca.adoc[NSVCA vs NEVRA] +* xref:core-concepts/nsvca.adoc[NSVCA vs NEVRA] * **Other** * xref:references.adoc[References] -* xref:archive.adoc[Archive] -** xref:architecture/consuming/dnf-behavior.adoc[DNF Behavior] * xref:history.adoc[History and Background] ** xref:requirements.adoc[Requirements and Use Cases] diff --git a/modules/ROOT/pages/_partials/core-concepts-links.adoc b/modules/ROOT/pages/_partials/core-concepts-links.adoc new file mode 100644 index 0000000..ccb6c38 --- /dev/null +++ b/modules/ROOT/pages/_partials/core-concepts-links.adoc @@ -0,0 +1,13 @@ +# tag::core-concepts-links[] +* xref:core-concepts/index.adoc[Core Concepts] +** xref:core-concepts/modular-repository.adoc[Modular Repository] +** xref:core-concepts/module-stream.adoc[Module Stream] +** xref:core-concepts/module-context.adoc[Module Context] +** xref:core-concepts/module-buildorder.adoc[Module Buildorder] +** xref:core-concepts/module-artifacts.adoc[Module Artifacts] +** xref:core-concepts/module-dependency-resolution.adoc[Module dependency resolution] + +** xref:core-concepts/building.adoc[Building Software] +** xref:core-concepts/consuming.adoc[Consuming Software] +** xref:core-concepts/upgrade-paths.adoc[Upgrade Paths] +# end::core-concepts-links[] diff --git a/modules/ROOT/pages/architecture.adoc b/modules/ROOT/pages/architecture.adoc deleted file mode 100644 index 7a495fe..0000000 --- a/modules/ROOT/pages/architecture.adoc +++ /dev/null @@ -1,73 +0,0 @@ -= Modularity Architecture - -NOTE: This page needs to be rewritten - -Modularity reuses some existing and creates some new concepts in the build and release process in Fedora. - -== Packages - -image::mod-doc-package.png[,80%,] - -We are reusing packages as they are. - -Packages are the smallest unit of software delivery. -Right now, Fedora uses RPM packages to build and ship all of its software. -There is a distinction between source RPM (SRPM) packages and binary RPM packages. - -As the name suggests, *source packages* contain the source code, but also the build process definition including a list of build dependencies. -Building a source package results in one or multiple binary packages. - -*Binary packages* are produced by building source packages in a build environment called a buildroot. -Different buildroots produce different binary packages. -This means that a source package without a context is not a complete definition of a resulting binary package. - -Every Fedora release has its own buildroot. -Even when multiple releases contain the same package using the same source, they are built multiple times, once in each buildroot, producing different (and often incompatible) binary packages. -Also, over time, as the distribution gets updated, so does its buildroot. -As a result, a package built today may not result in the same binary as a package built tomorrow. - -*Packages are either a part of a release, or a part of a module.* - -== Releases - -image::mod-doc-release.png[,80%,] - -Releases are the generations of Fedora e.g. “Fedora 28”, “Fedora 29”, or “EPEL 7”. - -Each release can only contain one major version of every package (by policy). -In general, Fedora tends to ship the latest stable versions of software available. -At the same time, packages need to maintain a certain API/ABI stability throughout the life cycle of the release they are part of. - -The problem with this approach is finding the right balance of being too fast vs. too slow. -For example, developers in general tend to prefer the newest versions of software, while server administrators want API/ABI stability for a longer period of time. -This problem is solved by Modules. - -== Modules and Streams - -image:mod-doc-module.png[,80%,] - -*Modules* are collections of packages representing a logical unit e.g. an application, a language stack, a set of tools. -These packages are built, tested, and released together. - -Each module defines its own life cycle which is closer to the upstream rather than the Fedora release. - -Modules are the mechanism of delivering multiple streams (versions) of software for the same release. -This also works the other way round, the same stream for multiple releases. - -image::mod-doc-stream.png[,80%,] - -A *stream* is a series of versions having a goal such as backwards API/ABI compatibility or delivering the latest stable version. -A “nodejs” interpreter can have streams such as https://src.fedoraproject.org/modules/nodejs/branch/6[6] or https://src.fedoraproject.org/modules/nodejs/branch/8[8] (two current versions of nodejs), but also “devel” or “latest”. -Some packages don’t have a major version. -An example could be the “calc” package that is being developed in two branches “stable” and “unstable”. - -Simply put, streams are the mechanism of delivering and updating software at different speeds, allowing developers to have bleeding-edge software, and server admins longer stability. - -== Artifacts - -image::mod-doc-repositories.png[,80%,] - -Artifacts are consumable outputs of the build and release processes e.g. an ISO image, an RPM repository, a container base image, or an OSTree tree. - -Modules will be delivered as an RPM repository that will work on top of existing releases. -This repository will also be available in the Fedora container base image, enabling users to build containers with multiple versions of software. diff --git a/modules/ROOT/pages/architecture/building.adoc b/modules/ROOT/pages/architecture/building.adoc deleted file mode 100644 index e48b133..0000000 --- a/modules/ROOT/pages/architecture/building.adoc +++ /dev/null @@ -1,55 +0,0 @@ -= Building Software with Modularity - -This is a high-level overview of how builds work with Fedora Modularity and how it compares with traditional builds. - -== Traditional package builds - -image::mod-doc-trad-builds.png[,80%,] - -In the existing Fedora world, packages are maintained in branches that map to specific Fedora releases. -Package repositories have branches e.g. “f27” for Fedora 27, “f28” for Fedora 28, etc. - -There are no changes to the existing packager workflow. -Maintaining release-specific branches and using the existing commands to build them will keep working. - -== Modular package builds - -image::mod-doc-stream-builds.png[,80%,] - -Modularity changes package branching to be version-focused. - -Branching packages according to their major version as opposed to the distribution release brings more clarity — because we can see what lives in a branch, and removes source duplication — because there is one branch per version. - -While this approach makes more sense from the software perspective, there is no longer a clear mapping of a package branch to a particular release. -Modules are a way of defining this. - -== Building one source for multiple releases - -image::mod-doc-build-more-releases.png[,40%,] - -Modules define what package branches are built for which Fedora releases. -One module can be built for multiple releases. - -The above example produces three different binaries, one for each release. In case there are multiple architectures, three binaries will be produced (one each). - -== Building multiple versions - -image::mod-doc-build-more-versions.png[,40%,] - -Multiple versions of packages can be built for one release by defining multiple modules. -Users are able to choose which version they want to use on their system. - -The above example produces two different binaries, each with a different Python version. -Again, in case there are multiple architectures, two binaries will be produced for each one. - -Modularity also leverages existing technologies such as containers to enable users to run multiple versions at the same time on one "machine." - -== Building against multiple versions - -image::mod-doc-build-matrix.png[,60%,] - -Modules can be also be built against other modules to achieve more complex results. -Client tooling always makes sure the right version is installed for a given context. - -The above example produces two different binaries, each bound to a different Python version. -Again, in case there are multiple architectures, two binaries will be produced for each one. diff --git a/modules/ROOT/pages/architecture/consuming.adoc b/modules/ROOT/pages/architecture/consuming.adoc deleted file mode 100644 index 9528a1a..0000000 --- a/modules/ROOT/pages/architecture/consuming.adoc +++ /dev/null @@ -1,61 +0,0 @@ -= Consuming Software with Modularity - -This is a high-level overview of how users can install software with Fedora Modularity and how is it different from the traditional workflow(s). - -== Understanding the Delivery Channels - -Modular Fedora will ship with two sets of repositories: - -. The *traditional base repository* representing the distribution as we know it today — there are no user-visible changes in this part. - -. A new *modular repository* (often referred to as the "Application Stream" or AppStream for short) including all the additional versions delivered as modules. - -The Modules repository will be optional for users. - -image::mod-doc-repositories.png[,80%,] - -== Consuming the traditional packages - -There are no changes to the traditional user experience. -Packages from the traditional repository will be installed and updated using the same methods as before. -Everything keeps working as it used to. - -== Consuming the modular packages - -If a user desires to use the optional Modular repositories in order to consume non-default versions of software, there will be some new concepts introduced in the client tooling to manage them. -We outline these below. - -=== Enabling a module - -Enabling a module makes its packages available on the system. -Packages delivered as part of a module always have a priority over the ones from the traditional base, regardless of their actual version. -Packages in modules are often replacements of the ones in the traditional base. - -Modularity brings parallel availability, not parallel installability. -Only one stream of a given module can be enabled on a system — so it is always clear which version gets installed. -Installing and running multiple versions of software can be achieved by using existing technologies, such as containers. - -=== Installing a module - -To make installation easy, some modules can be also installed as a unit, without the need of enabling them first and then installing individual packages. -Installing a module doesn’t necessarily mean installing all of its packages. -Modules can define something called an "installation profile" to help users with the installation. - -==== Installation profile - -Installation profiles are essentially lists of packages that help users with the module installation. To give a specific example, a database module could have two profiles: server and client. This helps the user to install what they need without the need of thinking about the package names. However, installation profiles are just an optional feature and users can still install packages directly. - -==== Updating the system - -Updating the system always respects user’s module choices (or lack of module) even when there are multiple (and possibly higher) versions available. - -If the user doesn’t enable any modules, all packages on their system get updated to the latest versions provided by the traditional base repository. -However, if the user enables a module stream, packages get updated to the newest version provided by the module. - -Thanks to this mechanism, the user has better control over the versions of packages on their system while receiving updates (such as security patches) for the whole system. - -==== Running multiple versions using containers - -Modularity brings parallel availability, not parallel installability. There are other technologies e.g. Linux containers or software collections that deal with this. - -All the steps described above can be used in a container the same way as on a traditional system. Producing up-to-date containers with multiple versions of software in an automated way is also one of the goals. diff --git a/modules/ROOT/pages/architecture/consuming/dnf-behavior.adoc b/modules/ROOT/pages/architecture/consuming/dnf-behavior.adoc deleted file mode 100644 index 0221220..0000000 --- a/modules/ROOT/pages/architecture/consuming/dnf-behavior.adoc +++ /dev/null @@ -1,145 +0,0 @@ -= Modular DNF Behavior - -WARNING: This page has been archived. It contains information that is either no longer true, relevant, or it should not be part of this documentation. It has been kept for historical reasons and can be removed at any time. - -This page captures the key behaviors of modular DNF. - -== Definitions - -* Ursine ("bare") packages are those that are not part of any module. -* A module's "context" is a unique hash value generated by the build - system to distinguish binaries built for different Fedora releases as - a result of stream expansion. - See <>. - -== General - -* Existing use of RPM-based operations must not change and must - continue to work just as they always have. -* DNF install/update commands can be invoked by using syntax of either - `dnf module $modspec ...` or `dnf @$modspec ...`. -* The "platform" pseudo-module cannot be explicitly enabled, installed, - deleted, or have its stream changed. -* A module is specified by its name, stream, version, context, and - architecture (e.g., "module:stream:version:context:arch"). -** If a module has a default stream defined on a system, "stream" may be - omitted. Otherwise, "stream" must be explicitly provided. -** For a given "module:stream", "version" can always be omitted. If - "version" is not explicitly provided, the latest "version" for the - given "module:stream" will be chosen. -** If the module "context" is not specified, DNF will use heuristics to - determine the appropriate "context". - See <>. -** If the module "arch" is not specified, the system's base architecture - will be used. -** Specifying a non-existent "module" name should fail. -** Specifying a non-existent "stream" (for a valid "module") should fail. -** Specifying specifying a non-existent "version" - (for a valid "module:stream") should fail. -** Specifying specifying a non-existent "context" - (for a valid "module:stream:version") should fail. -** Specifying a non-existent "arch" - (for a valid "module:stream:version:context") should fail. - -== System modular defaults - -* DNF must look for modular default stream and installation profile - data in the repodata of the package installation repos--which is - sourced from <> during the Fedora build - and release process. -* DNF must look for _override_ data for modular default stream and - installation profile on the local system in - `/etc/dnf/modules.defaults.d/`. - -== Module listing - -* DNF must be able to list all modular content (modules, streams, - enabled streams, default streams, profiles, module packages, - profile packages). - -== Module enablement - -* Enabling a module stream makes all of its packages available on the - system, but does not install them. -* At most one stream of a module may be enabled at a time on a system. - - -== Module Auto-enablement - -* If an uninstalled/unenabled module has a valid default stream - configured, DNF will consider the default stream of the module and - all of its packages available for resolving dependencies--and will - automatically enable the default stream of that module if any packages - from it are installed directly or indirectly (to resolve dependencies). -* If an uninstalled/unenabled module:stream is listed as a runtime - dependency of another module being installed or enabled, DNF will - automatically enable that dependency. -* Once a particular stream of a module has been enabled, DNF may not - automatically enable a different stream. - -== Module/package installation - -* Installing a module enables the specified stream (or default stream, - if not specified) and installs all of the packages associated with the - specified profile (or "default" profile, if not specified). -* If a module does not explicitly have a "default" profile specified it - its modulemd, DNF must pretend that it exists and is empty. - See <>. -* Each profile of a module stream must be able to be installed. - However, a module's profiles may conflict amongst themselves such - that they cannot all be installed at the same time. - (A hypothetical example would be a module that provides something - analogous to the coreutils and coreutils-single packages in different - profiles. The coreutils packages provide the same functionality, but - conflict.) - -== Switching module streams - -* This operation is not currently allowed. The currently enabled stream needs to be either reset or disabled before a different stream will be available for enablement or installation. - -== Module update - -* Updating a module will update all installed packages that belong to - the enabled module stream--regardless if the packages were installed - by an install profile or ad hoc. - -== Module disablement - -* Disabling a module does not remove any packages that belong to the - module. -* Disabling a module disables modules that depend on it--unless they - still have packages installed. - -== Module removal - -* Uninstalling a module removes all the packages that belong to the - module and disables it. -* Ad hoc removal of all packages from an installed module does not - disable the module. - -== Module locking - -* A locked module cannot have a different stream enabled or be - disabled--without first unlocking it. - -== Stream expansion and contexts - -* Packages from module contexts that do not match the current - transaction are disregarded. - See <>. - -== Other - -* Any package in an enabled module takes priority over any ursine RPM - with the same package name, regardless of which repositories those - modules and packages may be in. -* If there's an ursine or modular RPM with a higher NVR than the hotfix - one, the ursine or modular one wins. This is to ensure new proper - updates override the temporary hotfix RPMs. - -[bibliography] -== References - -* [[[mmd-spec-v2]]] https://github.com/fedora-modularity/libmodulemd/blob/master/spec.v2.yaml -* [[[fedora-module-defaults-repo]]] https://pagure.io/releng/fedora-module-defaults -* [[[module-stream-expansion]]] https://fedoraproject.org/wiki/Infrastructure/Factory2/Focus/StreamExpansion diff --git a/modules/ROOT/pages/architecture/consuming/upgrade-paths.adoc b/modules/ROOT/pages/architecture/consuming/upgrade-paths.adoc deleted file mode 100644 index 3a7230b..0000000 --- a/modules/ROOT/pages/architecture/consuming/upgrade-paths.adoc +++ /dev/null @@ -1,17 +0,0 @@ -= Upgrade Paths in Modularity - -On the package level, upgrades of a modular system work the same way as on a traditional system — using NEVRA comparison to determine which packages are the newest. There is, however, one additional step right before the NEVRA comparison that needs to happen on a modular system — limiting which packages are going to be part of the comparison based on what modules are enabled — and that is the key difference. - -There are up to three classes of RPM packages available to a modular system: - -1. **Standalone packages** (also refered-to as "bare RPMs" or "ursine RPMs") — packages not being part of a module. In Fedora, these are coming from the Everything repository. -2. **Modular packages** — packages being part of a module. In Fedora, these are coming from the Modular repository. -3. **Hotfixes** — standalone packages created on-demand by users or vendors meant to fix a critical issue before an official upgrade comes from the distribution. These need to be provided in a separate repository with a hotfix flag set. Fedora doesn't provide such packages. - -To determine the limited set of packages for the NEVRA comparison, the following algorithm is used: - -1. Take all standalone packages. -2. Add modular packages that are part of an enabled module, and potentially replace any standalone packages having a same name. To do this, look at the target modulemd rather than the current one. This step ensures that modular packages have always a higher priority than standalone packages. In other words, standalone packages can never upgrade modular packages. -3. Add all hotfix packages. These are just added, not replacing anything. That means hotfixes can potentially upgrade both traditional and modular packages. - -The next step is to take this set of packages, and run a NEVRA comparison to determine the highest version of each package. The highest versions are then installed as a part of the upgrade process. \ No newline at end of file diff --git a/modules/ROOT/pages/architecture/nsvca.adoc b/modules/ROOT/pages/architecture/nsvca.adoc deleted file mode 100644 index 2744c61..0000000 --- a/modules/ROOT/pages/architecture/nsvca.adoc +++ /dev/null @@ -1,166 +0,0 @@ - -= NSVCA vs NEVRA - -1. **<>** -2. **<>** -3. **<>** -4. **<>** - -== Naming and identifying modules in Fedora - -With the introduction of Modularity to the packager ecosystem it also introduced several challenges. First was that the NEVRA (name-epoch-version-release-architecture) package naming convention used in Fedora was insufficient. Modules are uniquely identified by the NSVCA naming convention, which stands for _name_, _stream_, _version_, _context_, and _architecture_. This page describes both the source-level ID (NSV) and the binary-level ID (NSVCA). - -== Source-level module ID (NSV) - -At the source level, modules are only identified by the first three: _name_, _stream_, and _version_. Name is defined as a name of the module’s repository in https://src.fedoraproject.org/[DistGit], stream is a name of the branch in https://src.fedoraproject.org/[DistGit], and version is the timestamp of a commit. - -=== Name - -Name of the module corresponds to the name of the application or the language stack it represents. -An example of a name could be postgresql for a PostgreSQL database module, or nodejs for a Node.js runtime. - -=== Stream - -Streams are variants of a module with a certain promise. - -In most cases, streams promise backwards compatibility with a major version of the application or the language stack they provide. For example, let’s say the Node.js runtime is supported in two major versions: 6 and 8. In this case, the module nodejs would have two streams: 6 and 8. - -However, streams can also promise different things such as stability. A good example of this is the calc package in Fedora which is maintained in two upstream branches: stable for the latest stable release and unstable for the latest development version. Using modularity, this package could be built as a calc module in two different streams: stable and unstable. - -In addition to the version promise, streams are also a way for packagers to communicate the level of maintenance. Does the maintainer plan to apply every minor patch? Will they apply security fixes quickly? Or is the module updated only twice a year? This can also be part of the promise. - -Another different example could be a stream that provides the software compiled using some experimental flags increasing the performance. - -Anyway, you get the idea. Streams are a very flexible and powerful tool. Use them wisely. - -=== Version - -Versions are just updates of a given stream. Technically, version is a number generated by the build system. Higher number always wins. This means that version does not identify a major/minor version of the software but the update/commit to a particular stream. - - -== Binary-level ID (NSVCA) - -Building a module from one source can result in multiple different binaries. Different binaries are typically produced for different architectures (i.e. x86_64, armv7hl, etc.) and different Fedora releases (i.e. Fedora 28, Fedora 29, EPEL 7, etc.). - -Module binaries use the full NSVCA — so in addition to the name, stream, and version fields described above, there are two more for binaries: architecture and context. - -=== Architecture - -Fedora is built for many different architectures. The architecture field simply distinguishes architecture-specific binaries from each other. The value is typically the same as with RPM packages, i.e. x86_64, armv7hl, etc. - -=== Context - -Context is used to distinguish binaries built for different Fedora releases. Thanks to stream expansion, modules can also be built against multiple streams of other modules, i.e. different versions of a language runtime etc. - -The value is generated by the build system and is usually hidden from the user as it doesn’t have any informational value by itself — it is a hash. However, the client tooling consuming this value can present it in a useful way. - -One way of representing the context could be listing the Fedora releases for which a certain module has been built. - -== NSVCA Definition - -This defines naming policy for modulemd metadata of final (built) modules. -This policy does **NOT** apply on sources such as modulemd yaml in dist-git. - -The goal is to provide unique identifiers for modules -that are both human readable and also suitable for machine processing. - -== Fields - -* **N** - Name -* **S** - Stream -* **V** - Version -* **C** - Context -* **A** - Arch -* **P** - Profile - - -== Separators - -Fields are separated with ':' (colon): N:S:V:C:A. - -If P is specified, it's separated from N:S:V:C:A with '/' (forward slash): N:S:V:C:A/P. - -=== Examples - ----- -# N:S:V:C:A -mariadb:3.6:1:0123abcd:x86_64 -# N:S:V:C:A/P -mariadb:3.6:1:0123abcd:x86_64/server ----- - -== Forms - -A form is a sequence of fields that fully or partially identifies a module. - -=== Full Forms - -N:S:V:C:A:: - Unique identifier of a module. -N:S:V:C:A/P:: - Unique identifier of a module profile. - - -=== Partial Forms - -Supported partial forms are: `N [ : S [ :V [ :C ] ] ] [ :A ] [ /P ]` - -Namely: - -* `N` -* `N::A` -* `N:S` -* `N:S::A` -* `N:S:V` -* `N:S:V::A` -* `N:S:V:C` -* `N:S:V:C:A` (identical to `N:S:V:C::A`) -* and all combinations with `/P` - -Missing fields **SHOULD** be populated with recommended defaults: - -Stream:: - defaults to the enabled or system default stream for the module in this particular order -Version:: - defaults to the latest available version in the module stream -Context:: - defaults to a value matching with already installed modules or modules involved in the transaction (not yet installed) -Arch:: - defaults to the system arch (e.g. DNF's $basearch) -Profile:: - defaults to the system default or 'default' profile - - -=== Allowed Characters - - -**N** - Name:: - a-z A-Z 0-9 . - _ + -**S** - Stream:: - a-z A-Z 0-9 . - _ + -**V** - Version:: - 0-9 -**C** - Context:: - 0-9 a-f -**A** - Arch:: - a-z A-Z 0-9 . - _ + -**P** - Profile:: - a-z A-Z 0-9 . - _ + - -All fields **MUST** start and end with an alphanumeric character: -a-z A-Z 0-9 - - -=== Forbidden Characters - -This paragraph serves as a design decision for future changes. - -Following characters **MUST NOT** be part of any field: - -* `:` (colon) - separator -* `/` (forward slash) - profile separator -* `\` (backslash) - comon control character -* `*` (asterisk) - common wildcard -* `?` (question mark) - common wildcard -* `@` (at) - grpspec in YUM and DNF -* ` ` (space) - common separator diff --git a/modules/ROOT/pages/architecture/stream-expansion.adoc b/modules/ROOT/pages/architecture/stream-expansion.adoc deleted file mode 100644 index 65e277e..0000000 --- a/modules/ROOT/pages/architecture/stream-expansion.adoc +++ /dev/null @@ -1,17 +0,0 @@ -= Module Stream Expansion - -Stream Expansion allows for automatic submission of multiple builds of the same component based on multiple versions of its build dependencies. - -== Build - -image::stream-expansion-build.png[,100%,] - -One module can be built against multipe Fedora releases, or against multiple streams of other modules such as language runtimes, producing a matrix of binaries for all defined combinations. - -These are the same modules of the same stream, but of a different context. - -== User view - -image::stream-expansion-user-view.png[,100%,] - -User only sees the modules and their streams, but the context is hidden. The client tooling chooses the right streams of dependencies based on the version of system the user is running, what dependencies have been already installed or enabled by the user, or based on defaults. \ No newline at end of file diff --git a/modules/ROOT/pages/building-modules/fedora/defining-modules.adoc b/modules/ROOT/pages/building-modules/fedora/defining-modules.adoc index 96a138d..1b36682 100644 --- a/modules/ROOT/pages/building-modules/fedora/defining-modules.adoc +++ b/modules/ROOT/pages/building-modules/fedora/defining-modules.adoc @@ -10,8 +10,8 @@ Feel free to copy/paste this example when creating your new module. [source,yaml] ---- -document: modulemd -version: 2 +document: modulemd-packager +version: 3 data: # === Information about this module ================================== # (Can be copied from the main RPM package, but doesn't need to be) @@ -21,19 +21,94 @@ data: the obligatory lorem ipsum dolor sit amet goes right here. # === License of this modulemd file ================================== - # (Package licenses will be added automatically by the build system) license: - module: - - MIT - - # === Modular dependencies =========================================== + - MIT + # === Module context configurations =========================================== # (For which Fedora releases to build?) - dependencies: - - buildrequires: - platform: [] # <- Build for all Fedora releases + configurations: + # context: + # A string of up to ten [a-zA-Z0-9] characters representing a + # a build and runtime configuration for this stream. This string is + # arbitrary but must be unique in this module stream. + # Type: MANDATORY + - context: CTX1 + # platform: + # Defines the distribution and release to build on and run against. + # Type: MANDATORY + platform: f32 + # buildrequires: + # A dictionary of the build-time dependencies on other module streams. + # Each configuration may depend on a single stream of a dependency. + # The dictionary key is the name of the module and the dictionary value + # is a single-element list containing the name of the stream. + # + # Type: Optional + buildrequires: + appframework: [v1] + # requires: + # A dictionary of the run-time dependencies on other module streams. + # Each configuration may depend on a single stream of a dependency. + # The dictionary key is the name of the module and the dictionary value + # is a single-element list containing the name of the stream. + # + # Type: Optional requires: - platform: [] # <- Run on all Fedora releases - + appframework: [v1] + # buildopts: + # Component build options + # Additional per component type module-wide build options. + # + # IMPORTANT: Due to limitations in the modulemd-stream v2 format, the + # buildopts from the first configuration in the list will apply to + # ALL configurations when building for modulemd-stream v2. They will + # apply separately when building for module-stream v3. + # + # Type: OPTIONAL + buildopts: + # rpms: + # RPM-specific build options + # + # Type: OPTIONAL + rpms: + # macros: + # Additional macros that should be defined in the + # RPM buildroot, appended to the default set. Care should be + # taken so that the newlines are preserved. Literal style + # block is recommended, with or without the trailing newline. + # + # Type: OPTIONAL + macros: | + %demomacro 1 + %demomacro2 %{demomacro}23 + + # whitelist: + # Explicit list of package build names this module will produce. + # By default the build system only allows components listed under + # data.components.rpms to be built as part of this module. + # In case the expected RPM build names do not match the component + # names, the list can be defined here. + # This list overrides rather then just extends the default. + # List of package build names without versions. + # + # Type: OPTIONAL + whitelist: + - fooscl-1-bar + - fooscl-1-baz + - xxx + - xyz + # arches: + # Instructs the build system to only build the + # module on this specific set of architectures. + # Includes specific hardware architectures, not families. + # See the data.arch field in the modulemd-stream spec for details. + # Defaults to all available arches. + # + # Type: OPTIONAL + arches: [i686, x86_64] + + # Alternate example with no dependencies + - context: CTX2 + platform: f33 # === Module API (optional, but encouraged) ========================== # (Which packages are API-stable?) api: @@ -86,8 +161,8 @@ Every modulemd starts with these three lines: [source,yaml] ---- -document: modulemd -version: 2 +document: modulemd-packager +version: 3 data: ... <1> ---- @@ -117,26 +192,14 @@ The build system adds licenses of all packages to this list automatically. [source,yaml] ---- license: - module: - - MIT <1> + - MIT <1> ---- <1> A license for this modulemd file. Fedora content, such as SPEC files or patches not included upstream, uses the MIT license by default, unless the component packager declares otherwise. -=== Modular dependencies - -Simply put: For which Fedora releases to build? - -To build your module for all Fedora releases that are actively maintained, use the following definition. For anything more than this, such as building against other modules or requiring other modules during run time, please see the Advanced section below. +=== Module context configurations -[source,yaml] ----- - dependencies: - - buildrequires: - platform: [] - requires: - platform: [] ----- +Each context configuration describes how a module stream and it components should be build and run. === Installation profiles (optional, but encouraged) @@ -157,13 +220,9 @@ To help users install your module, define installation profiles. These profiles - package-one-cli ... <4> ---- - <1> Name of the profile. - <2> A quick summary of the profile. - <3> Binary packages to be installed with this profile. - <4> List as many profiles as you need. === Module API (optional, but encouraged) @@ -197,13 +256,9 @@ List all source SRPM packages this module should include, referenced them by the ref: latest ... <4> ---- - <1> Name of the package — maps to a DistGit repository name. - <2> The reason why is this package here. Mostly for humans. - <3> DistGit branch, tag, or a commit — so the right version of the package gets included. - <4> List as many packages as you need. == Advanced definitions @@ -219,11 +274,8 @@ You can also provide references to the upstream community, documentation, or to documentation: http://www.example.com/ <2> tracker: http://www.example.com/ <3> ---- - <1> Upstream community website, if it exists. - <2> Upstream documentation, if it exists. - <3> Upstream bug tracker, if it exists. === Building in a specific order (optional) @@ -247,105 +299,9 @@ In this specific example, `first-package` gets built first, and `second-package` ref: latest buildorder: 10 <1> ---- - <1> A number of the build group. -=== Build macros (optional) - -Use this if you need to build your packages with a specific RPM macro. Applies to all packages in the module. - -[source,yaml] ----- - buildopts: - rpms: - macros: | - %demomacro 1 - %demomacro2 %{demomacro}23 ----- - -=== Advanced dependencies (optional) - -modules can be - -* built against other modules -* require other modules during run time -* built against one or more streams of the same module -* work with one or more streams of another module - -**Building only for Fedora 28**: - -[source,yaml] ----- - dependencies: - - buildrequires: - platform: [f28] - requires: - platform: [f28] ----- - -**Building for everything else than Fedora 28**: - -[source,yaml] ----- - dependencies: - - buildrequires: - platform: [-f28] - requires: - platform: [-f28] ----- - -**Building only for Fedora 28 and Fedora 29**: - -[source,yaml] ----- - dependencies: - - buildrequires: - platform: [f28, f29] - requires: - platform: [f28, f29] ----- - -**Building against other modules:**: - -Your module can also depend on another modules. Specific streams can be referenced the same way as above in "Building for a specific Fedora release(s) only". - -[source,yaml] ----- - dependencies: - - buildrequires: - platform: [] - nodejs: [] - requires: - platform: [] - nodejs: [] ----- - -**A complex example:** - -Simple things should simple, complex things should be possible. Let's say my module requires `nodejs` during build and run time. It also requires `build-tools` only during build. To make it even more complex, it also requires a specific stream of a `pizza-module` during build and run time, but only on Fedora 27. - -[source,yaml] ----- - dependencies: - - buildrequires: - platform: [-f27] - buildtools: [v1, v2] - nodejs: [] - requires: - platform: [-f27] - nodejs: [] - - buildrequires: - platform: [f27] - buildtools: [v1, v2] - nodejs: [] - pizza-module: [3.6] - requires: - platform: [f27] - nodejs: [] - pizza-module: [3.6] ----- - -For even more complex scenarios, please study the https://github.com/fedora-modularity/libmodulemd/blob/master/spec.v2.yaml[modulemd specification]. +For even more complex scenarios, please study the https://github.com/fedora-modularity/libmodulemd/blob/main/yaml_specs/modulemd_packager_v3.yaml[modulemd-packager specification]. === Filtered Packages (optional, defaults to no filters) @@ -417,25 +373,22 @@ All of the `customdocgen` artifacts will be automatically added to the `data.fil === An absolute minimum -This module includes two source RPM packages built for all Fedora releases. +This module includes two source RPM packages built for the Fedora 35 releases. [source,yaml] ---- -document: modulemd -version: 2 +document: modulemd-packager +version: 3 data: summary: An example module description: >- A module for the demonstration of the metadata format. Also, the obligatory lorem ipsum dolor sit amet goes right here. license: - module: - - MIT - dependencies: - - buildrequires: - platform: [] - requires: - platform: [] + - MIT + configurations: + - context: CTX1 + platform: f35 components: rpms: first-package: @@ -448,25 +401,22 @@ data: === Including profiles and API (recommended) -This module includes two source RPM packages built for all Fedora releases. It makes clear which packages are considered the API, and helps users with installation thanks to the profiles. +This module includes two source RPM packages built for the Fedora 35 releas. It makes clear which packages are considered the API, and helps users with installation thanks to the profiles. [source,yaml] ---- -document: modulemd -version: 2 +document: modulemd-packager +version: 3 data: summary: An example module description: >- A module for the demonstration of the metadata format. Also, the obligatory lorem ipsum dolor sit amet goes right here. license: - module: - - MIT - dependencies: - - buildrequires: - platform: [] - requires: - platform: [] + - MIT + configurations: + - context: CTX1 + platform: f35 api: rpms: - package-one @@ -494,15 +444,3 @@ data: rationale: Web UI for the first-package. ref: latest ---- - -== Using fedmod - -To get started quickly, you can use the `fedmod` tool (from the package of the same name) to create a skeletal modulemd file for your module based on one or more source packages, e.g. - -[source,shell] ----- -$ fedmod fetch-metadata -$ fedmod rpm2module dwm > dwm.yaml ----- - -If you specify more than one package it will fill in the summary and description fields from their counterparts in the first specified package. diff --git a/modules/ROOT/pages/building-modules/fedora/managing-defaults.adoc b/modules/ROOT/pages/building-modules/fedora/managing-defaults.adoc index 25caa9d..93ee226 100644 --- a/modules/ROOT/pages/building-modules/fedora/managing-defaults.adoc +++ b/modules/ROOT/pages/building-modules/fedora/managing-defaults.adoc @@ -1,5 +1,7 @@ = Managing module defaults in Fedora +NOTE: Default streams are *NOT* allowed in Fedora right now according to policy. Please check the xref:policies/index.adoc[Policy section] for more information. + Setting or changing a default stream or a default installation profile of a module constitutes a major behavior change as defined in the Fedora Updates Policy. The following rules apply: 1. Module stream defaults MUST be only changed in an upcoming Fedora release diff --git a/modules/ROOT/pages/building-modules/fedora/module-obsoletes.adoc b/modules/ROOT/pages/building-modules/fedora/module-obsoletes.adoc index 2879513..5335826 100644 --- a/modules/ROOT/pages/building-modules/fedora/module-obsoletes.adoc +++ b/modules/ROOT/pages/building-modules/fedora/module-obsoletes.adoc @@ -1,10 +1,14 @@ +:toc: macro = Module Obsoletes -:toc: [.lead] Module obsoletes allow for stream obsoleting another module or a stream being EOLed. The feature has been introduced in https://github.com/fedora-modularity/libmodulemd/releases/tag/2.10.0[libmodulemd-2.10.0]. + + + + + +toc::[] == Module Obsoletes in the Pipeline (Quick Start) diff --git a/modules/ROOT/pages/community.adoc b/modules/ROOT/pages/community.adoc index 26c957b..b2f0cdb 100644 --- a/modules/ROOT/pages/community.adoc +++ b/modules/ROOT/pages/community.adoc @@ -1,9 +1,11 @@ +:toc: macro = Community -This page lists the communication channels of members of the Modularity Working Group (Modularity WG), related projects and also how you can contribute to the project. +This page lists the communication channels of members of the Modularity Working Group (Modularity WG), related projects and also how you can contribute to the project. + + + + -1. **<>** -2. **<>** +toc::[] == Channels diff --git a/modules/ROOT/pages/core-concepts/building.adoc b/modules/ROOT/pages/core-concepts/building.adoc new file mode 100644 index 0000000..e48b133 --- /dev/null +++ b/modules/ROOT/pages/core-concepts/building.adoc @@ -0,0 +1,55 @@ += Building Software with Modularity + +This is a high-level overview of how builds work with Fedora Modularity and how it compares with traditional builds. + +== Traditional package builds + +image::mod-doc-trad-builds.png[,80%,] + +In the existing Fedora world, packages are maintained in branches that map to specific Fedora releases. +Package repositories have branches e.g. “f27” for Fedora 27, “f28” for Fedora 28, etc. + +There are no changes to the existing packager workflow. +Maintaining release-specific branches and using the existing commands to build them will keep working. + +== Modular package builds + +image::mod-doc-stream-builds.png[,80%,] + +Modularity changes package branching to be version-focused. + +Branching packages according to their major version as opposed to the distribution release brings more clarity — because we can see what lives in a branch, and removes source duplication — because there is one branch per version. + +While this approach makes more sense from the software perspective, there is no longer a clear mapping of a package branch to a particular release. +Modules are a way of defining this. + +== Building one source for multiple releases + +image::mod-doc-build-more-releases.png[,40%,] + +Modules define what package branches are built for which Fedora releases. +One module can be built for multiple releases. + +The above example produces three different binaries, one for each release. In case there are multiple architectures, three binaries will be produced (one each). + +== Building multiple versions + +image::mod-doc-build-more-versions.png[,40%,] + +Multiple versions of packages can be built for one release by defining multiple modules. +Users are able to choose which version they want to use on their system. + +The above example produces two different binaries, each with a different Python version. +Again, in case there are multiple architectures, two binaries will be produced for each one. + +Modularity also leverages existing technologies such as containers to enable users to run multiple versions at the same time on one "machine." + +== Building against multiple versions + +image::mod-doc-build-matrix.png[,60%,] + +Modules can be also be built against other modules to achieve more complex results. +Client tooling always makes sure the right version is installed for a given context. + +The above example produces two different binaries, each bound to a different Python version. +Again, in case there are multiple architectures, two binaries will be produced for each one. diff --git a/modules/ROOT/pages/core-concepts/consuming.adoc b/modules/ROOT/pages/core-concepts/consuming.adoc new file mode 100644 index 0000000..9528a1a --- /dev/null +++ b/modules/ROOT/pages/core-concepts/consuming.adoc @@ -0,0 +1,61 @@ += Consuming Software with Modularity + +This is a high-level overview of how users can install software with Fedora Modularity and how is it different from the traditional workflow(s). + +== Understanding the Delivery Channels + +Modular Fedora will ship with two sets of repositories: + +. The *traditional base repository* representing the distribution as we know it today — there are no user-visible changes in this part. + +. A new *modular repository* (often referred to as the "Application Stream" or AppStream for short) including all the additional versions delivered as modules. + +The Modules repository will be optional for users. + +image::mod-doc-repositories.png[,80%,] + +== Consuming the traditional packages + +There are no changes to the traditional user experience. +Packages from the traditional repository will be installed and updated using the same methods as before. +Everything keeps working as it used to. + +== Consuming the modular packages + +If a user desires to use the optional Modular repositories in order to consume non-default versions of software, there will be some new concepts introduced in the client tooling to manage them. +We outline these below. + +=== Enabling a module + +Enabling a module makes its packages available on the system. +Packages delivered as part of a module always have a priority over the ones from the traditional base, regardless of their actual version. +Packages in modules are often replacements of the ones in the traditional base. + +Modularity brings parallel availability, not parallel installability. +Only one stream of a given module can be enabled on a system — so it is always clear which version gets installed. +Installing and running multiple versions of software can be achieved by using existing technologies, such as containers. + +=== Installing a module + +To make installation easy, some modules can be also installed as a unit, without the need of enabling them first and then installing individual packages. +Installing a module doesn’t necessarily mean installing all of its packages. +Modules can define something called an "installation profile" to help users with the installation. + +==== Installation profile + +Installation profiles are essentially lists of packages that help users with the module installation. To give a specific example, a database module could have two profiles: server and client. This helps the user to install what they need without the need of thinking about the package names. However, installation profiles are just an optional feature and users can still install packages directly. + +==== Updating the system + +Updating the system always respects user’s module choices (or lack of module) even when there are multiple (and possibly higher) versions available. + +If the user doesn’t enable any modules, all packages on their system get updated to the latest versions provided by the traditional base repository. +However, if the user enables a module stream, packages get updated to the newest version provided by the module. + +Thanks to this mechanism, the user has better control over the versions of packages on their system while receiving updates (such as security patches) for the whole system. + +==== Running multiple versions using containers + +Modularity brings parallel availability, not parallel installability. There are other technologies e.g. Linux containers or software collections that deal with this. + +All the steps described above can be used in a container the same way as on a traditional system. Producing up-to-date containers with multiple versions of software in an automated way is also one of the goals. diff --git a/modules/ROOT/pages/core-concepts/dnf-behavior.adoc b/modules/ROOT/pages/core-concepts/dnf-behavior.adoc new file mode 100644 index 0000000..0221220 --- /dev/null +++ b/modules/ROOT/pages/core-concepts/dnf-behavior.adoc @@ -0,0 +1,145 @@ += Modular DNF Behavior + +WARNING: This page has been archived. It contains information that is either no longer true, relevant, or it should not be part of this documentation. It has been kept for historical reasons and can be removed at any time. + +This page captures the key behaviors of modular DNF. + +== Definitions + +* Ursine ("bare") packages are those that are not part of any module. +* A module's "context" is a unique hash value generated by the build + system to distinguish binaries built for different Fedora releases as + a result of stream expansion. + See <>. + +== General + +* Existing use of RPM-based operations must not change and must + continue to work just as they always have. +* DNF install/update commands can be invoked by using syntax of either + `dnf module $modspec ...` or `dnf @$modspec ...`. +* The "platform" pseudo-module cannot be explicitly enabled, installed, + deleted, or have its stream changed. +* A module is specified by its name, stream, version, context, and + architecture (e.g., "module:stream:version:context:arch"). +** If a module has a default stream defined on a system, "stream" may be + omitted. Otherwise, "stream" must be explicitly provided. +** For a given "module:stream", "version" can always be omitted. If + "version" is not explicitly provided, the latest "version" for the + given "module:stream" will be chosen. +** If the module "context" is not specified, DNF will use heuristics to + determine the appropriate "context". + See <>. +** If the module "arch" is not specified, the system's base architecture + will be used. +** Specifying a non-existent "module" name should fail. +** Specifying a non-existent "stream" (for a valid "module") should fail. +** Specifying specifying a non-existent "version" + (for a valid "module:stream") should fail. +** Specifying specifying a non-existent "context" + (for a valid "module:stream:version") should fail. +** Specifying a non-existent "arch" + (for a valid "module:stream:version:context") should fail. + +== System modular defaults + +* DNF must look for modular default stream and installation profile + data in the repodata of the package installation repos--which is + sourced from <> during the Fedora build + and release process. +* DNF must look for _override_ data for modular default stream and + installation profile on the local system in + `/etc/dnf/modules.defaults.d/`. + +== Module listing + +* DNF must be able to list all modular content (modules, streams, + enabled streams, default streams, profiles, module packages, + profile packages). + +== Module enablement + +* Enabling a module stream makes all of its packages available on the + system, but does not install them. +* At most one stream of a module may be enabled at a time on a system. + + +== Module Auto-enablement + +* If an uninstalled/unenabled module has a valid default stream + configured, DNF will consider the default stream of the module and + all of its packages available for resolving dependencies--and will + automatically enable the default stream of that module if any packages + from it are installed directly or indirectly (to resolve dependencies). +* If an uninstalled/unenabled module:stream is listed as a runtime + dependency of another module being installed or enabled, DNF will + automatically enable that dependency. +* Once a particular stream of a module has been enabled, DNF may not + automatically enable a different stream. + +== Module/package installation + +* Installing a module enables the specified stream (or default stream, + if not specified) and installs all of the packages associated with the + specified profile (or "default" profile, if not specified). +* If a module does not explicitly have a "default" profile specified it + its modulemd, DNF must pretend that it exists and is empty. + See <>. +* Each profile of a module stream must be able to be installed. + However, a module's profiles may conflict amongst themselves such + that they cannot all be installed at the same time. + (A hypothetical example would be a module that provides something + analogous to the coreutils and coreutils-single packages in different + profiles. The coreutils packages provide the same functionality, but + conflict.) + +== Switching module streams + +* This operation is not currently allowed. The currently enabled stream needs to be either reset or disabled before a different stream will be available for enablement or installation. + +== Module update + +* Updating a module will update all installed packages that belong to + the enabled module stream--regardless if the packages were installed + by an install profile or ad hoc. + +== Module disablement + +* Disabling a module does not remove any packages that belong to the + module. +* Disabling a module disables modules that depend on it--unless they + still have packages installed. + +== Module removal + +* Uninstalling a module removes all the packages that belong to the + module and disables it. +* Ad hoc removal of all packages from an installed module does not + disable the module. + +== Module locking + +* A locked module cannot have a different stream enabled or be + disabled--without first unlocking it. + +== Stream expansion and contexts + +* Packages from module contexts that do not match the current + transaction are disregarded. + See <>. + +== Other + +* Any package in an enabled module takes priority over any ursine RPM + with the same package name, regardless of which repositories those + modules and packages may be in. +* If there's an ursine or modular RPM with a higher NVR than the hotfix + one, the ursine or modular one wins. This is to ensure new proper + updates override the temporary hotfix RPMs. + +[bibliography] +== References + +* [[[mmd-spec-v2]]] https://github.com/fedora-modularity/libmodulemd/blob/master/spec.v2.yaml +* [[[fedora-module-defaults-repo]]] https://pagure.io/releng/fedora-module-defaults +* [[[module-stream-expansion]]] https://fedoraproject.org/wiki/Infrastructure/Factory2/Focus/StreamExpansion diff --git a/modules/ROOT/pages/core-concepts/index.adoc b/modules/ROOT/pages/core-concepts/index.adoc new file mode 100644 index 0000000..1525e81 --- /dev/null +++ b/modules/ROOT/pages/core-concepts/index.adoc @@ -0,0 +1,7 @@ += Core Concepts of Modularity + +The Modularity project added a lot of new concepts to an already established packaging ecosystem. A lot of those concepts are based on already existing non-modular ones. On the following pages we will try to describe and outline what are the cornerstones of the Modularity project. + +== Table of contents + +include::{partialsdir}/core-concepts-links.adoc[tag=core-concepts-links] diff --git a/modules/ROOT/pages/core-concepts/modular-repository.adoc b/modules/ROOT/pages/core-concepts/modular-repository.adoc new file mode 100644 index 0000000..13a8b30 --- /dev/null +++ b/modules/ROOT/pages/core-concepts/modular-repository.adoc @@ -0,0 +1,82 @@ +:toc: macro += Modular repository + +A standard non-modular RPM repository contains RPM binary files and the repodata directory which is generated by the `createrepo_c` tool. In a non-modular repository the metadata information is stored inside the RPM binary files themselves. So when running `createrepo_c` command on a directory with RPM files present, the metadata is extracted from each of the headers of the RPM files and bundled together in the repodata directory. + +Modular repository is a standard RPM repository which is extended with modular metadata and modular binary RPM files. A modular repository can consist of non-modular binary RPMs, modular binary RPMs and their corresponding modular metadata. Modular metadata defines a new entity inside a modular repository which is called a module stream. The simplest explanation of a module stream is that a module stream is a group of binary RPM artifacts bound together with extra modular metadata inside a YAML file. + + + + + +toc::[] + +:listing-caption: Example + +== Basic modular repository + +.of a Modular repository with one module stream. +[source,console,subs="verbatim,quotes"] +---- +[mcurlej@localhost final_repo]$ tree +. +├── flatpak-rpm-macros-35-4.module_fc35+devel+1234.src.rpm +├── flatpak-rpm-macros-35-4.module_fc35+devel+1234.x86_64.rpm +├── flatpak-runtime-config-35-1.module_fc35+devel+1234.src.rpm +├── flatpak-runtime-config-35-1.module_fc35+devel+1234.x86_64.rpm +├── *flatpak-runtime:devel:20220211094818:1234:x86_64.modulemd.yaml* <.> +└── repodata + ├── 3f869b4cf0d94c0d1b246d05b2c538259430bc03df65277a6b94b96cd0f5aa2d-other.xml.gz + ├── 58a1183d84dc7e319fb365b45c661c6653a530eddf0022a42c5caaf591adb2b8-filelists.sqlite.bz2 + ├── 686ca60dd902af9d4c49968833b3561fe35fd1b8f8f6f722f1a463dfdf559bdc-filelists.xml.gz + ├── afe6147edfe3577fc76ca94d86960e3e008f3edfa7ac5dca497f56a3ac41b22b-primary.sqlite.bz2 + ├── *b68df50bf8d984cdb05fc08d119e72280f7c998486a1ed74c20e632589c00d25-modules.yaml.gz* <.> + ├── d9b1a93ec2353fe6c87795dc59cc7142cf54b89cf762e60bf7f74f7266db75f1-primary.xml.gz + ├── ef096d0ee7ba59ef7c73c13e38a18c3ead7fef5f21abd115b367e2d9d3c1c39b-other.sqlite.bz2 + └── repomd.xml + +1 directory, 13 files +[mcurlej@localhost final_repo]$ +---- +<.> modulemd YAML file which holds information about one module stream +<.> modules.yaml.gz file which holds information about all the module streams in the repository + +In the first example we have a simple tree view of a modular repository. The repository contains the RPM packages `flatpak-rpm-macros` and `flatpak-runtime-config` both of those are the components of the module stream which is defined in the `flatpak-runtime:devel:20220211094818:1234:x86_64.modulemd` YAML file. The binary RPM files are also defined in the YAML metadata as the artifacts of the module stream. + +NOTE: A shorthand expression for modular metadata is *modulemd* + +Running `createrepo_c` on such a directory will generate a repodata directory but with one extra file. The file is `\*-modules.yaml.gz` and holds the YAML modulemd from all the present module stream YAML modulemd files in the directory. The `*-modules.yaml.gz` describes to DNF how many module streams are present in the modular repository, how they can be installed and which module streams cover which RPM packages. More on module streams can be found xref:core-concepts/module-stream.adoc[here]. + +WARNING: Modular binary RPM files can *NOT* be installed without the corresponding metadata. A module stream inside a repository *HAS* to always consist of a modulemd YAML file and the modular binary RPM files. + +== Mixed modular repository + +.of a modular repository with multipe module streams and non-modular packages +[source,console,subs="verbatim,quotes"] +---- +[mcurlej@localhost final_repo]$ tree +. +├── flatpak-rpm-macros-35-4.module_fc35+common+1234.src.rpm +├── flatpak-rpm-macros-35-4.module_fc35+common+1234.x86_64.rpm +├── flatpak-rpm-macros-35-4.module_fc35+devel+1234.src.rpm +├── flatpak-rpm-macros-35-4.module_fc35+devel+1234.x86_64.rpm +├── *flatpak-runtime:common:20220211094818:1234:x86_64.modulemd.yaml* <.> +├── flatpak-runtime-config-35-1.module_fc35+common+1234.src.rpm +├── flatpak-runtime-config-35-1.module_fc35+common+1234.x86_64.rpm +├── flatpak-runtime-config-35-1.module_fc35+devel+1234.src.rpm +├── flatpak-runtime-config-35-1.module_fc35+devel+1234.x86_64.rpm +├── *flatpak-runtime:devel:20220211094818:1234:x86_64.modulemd.yaml* <.> +├── *module-build-0.1.0-1.fc35.noarch.rpm* <.> +├── module-build-0.1.0-1.fc35.src.rpm +└── repodata +⋮ +1 directory, 20 files +[mcurlej@localhost final_repo]$ +---- +<.> modulemd YAML file of module stream `flatpak-runtime:common` +<.> modulemd YAML file of module stream `flatpak-runtime:devel` +<.> non-modular package `module-build` + +In the second example we have a modular repository with multiple module streams and a non-modular package. The module streams are always present in the repository but are not xref:core-concepts/module-dependency-resolution.adoc#_enabling_a_module_stream[enabled] by default. Module streams need to be enabled by the user before installation. If a module is not enabled by the user the binary RPM artifacts are not included in the dependency resolution and content set creation transactions which are done by DNF. In our example our modular repository contains the `flatpak-runtime` module which has the module streams `flatpak-runtime:common` and `flatpak-runtime:devel`. Each module stream consists of its modulemd YAML file and corresponding binary RPM files whose filenames are listed in the `artifacts` section of the modulemd YAML file. + +The non-modular package `module-build` in our modular repository behaves like it would in a non-modular repository. More about how you can enable and install module streams can be found in the section xref:using-modules.adoc[Using modules]. + +NOTE: RPMs inside a modular repository are still standard source and binary RPM files. They are still using the NEVRA (name-epoch-version-release-architecture) naming convention. As you can see in our example the RPM files have the same name and version. The unique identifier here is the release (or distag) of the filename. The modular RPM files in the example were built by the tool link:https://github.com/mcurlej/module-build[module-build]. The release part is defined by the tool or distribution pipeline you are using to build the RPM files. diff --git a/modules/ROOT/pages/core-concepts/module-artifacts.adoc b/modules/ROOT/pages/core-concepts/module-artifacts.adoc new file mode 100644 index 0000000..54ba543 --- /dev/null +++ b/modules/ROOT/pages/core-concepts/module-artifacts.adoc @@ -0,0 +1,25 @@ += Module Artifacts + +Artifacts are consumable outputs of the build and release processes e.g. an ISO image, an RPM repository, a container base image, or an OSTree tree. + +The artifacts of a module stream are a modulemd YAML file and the source and binary RPM files. Each component of a module stream represents a RPM package. All the built source and binary RPM files are recorded in the `artifacts` section of a xref:core-concepts/module-stream.adoc#_output_modulemd[output modulemd YAML file]. We are reusing packages as they are. Packages are the smallest unit of software delivery. Fedora uses RPM packages to build and ship all of its software. There is a distinction between source RPM (SRPM) packages and binary RPM packages. + +As the name suggests, *source packages* contain the source code, but also the build process definition including a list of build dependencies. Building a source package results in one or multiple binary packages. + +*Binary packages* are produced by building source packages in a build environment called a buildroot. Different buildroots produce different binary packages. This means that a source package without a context is not a complete definition of a resulting binary package. + +Module streams are delivered as an RPM repository that will work on top of existing releases. The modular repository will also be available in the Fedora container base image, enabling users to build containers with multiple versions of software. + +== Failsafe + +Artifacts which are the result of a module stream build are ordinary source and binary RPM files. There is one difference. Each binary RPM package built within a module stream contains a `modularitylabel` flag inside its file header. This flag is set during the build process of a RPM package. We need this to be able to distinguish between the binary RPM files built within a module stream and RPM files built the non-modular way. + +NOTE: We also refer to RPM artifacts built within a modules stream as module (or modular) RPMs. + +The `modularitylabel` flag is an important feature of modularity. Module RPM artifacts can not be installed by themselves. If a `modularitylabel` flag is detected in the header of a RPM file the installation will be rejected. This is also called a `failsafe`. Module streams need to be enabled first by DNF and then the RPM packages within a module stream will be available for installation. The `failsafe` guarantees that the correct RPM packages are visible and available for installation and all necessary modular dependencies are satisfied before the DNF transaction can proceed. All this information is stored in the modulemd YAML file. Without this file DNF does not know which RPM package should be installed and what is their relation with the rest of the RPM repository. + +== Compatibility + +As we already established earlier, module streams are only bundles of RPM packages which are installed together. This means that existing non-modular RPM packages can be bundled into a module stream just with adding the correct modulemd YAML file to the repodata. + +WARNING: We strongly discourage you to create module streams out of non-modular binary RPM as this can cause wrong dependency resolution of your installation transaction and break your system. Alway build your module streams through a distribution pipeline or local cli tools. diff --git a/modules/ROOT/pages/core-concepts/module-buildorder.adoc b/modules/ROOT/pages/core-concepts/module-buildorder.adoc new file mode 100644 index 0000000..7ec0bb7 --- /dev/null +++ b/modules/ROOT/pages/core-concepts/module-buildorder.adoc @@ -0,0 +1,64 @@ += Module build order + +Packages which are built within a module stream can have dependency relationships between them. This means that packages within a module can use other packages from the same module as build or runtime dependencies. For this to be possible we are using the `buildorder` property which can be added to each of the components defined in a module stream. + +By default, if no `buildorder` property is specified the build order of the packages is equal to 0 for the purposes of the module stream build. This means that all the packages can be built at the same time. + +If the `buildorder` property is specified for each component of the module stream, the build system needs to respect this ordering. This means that each `buildorder` number represents a group or a batch of packages that need to be built together. Each higher “build batch” needs the RPM artifacts from the previous build batches to be present and enabled in the current one. So they can serve as dependencies for the current batch. This continues until the last batch is built. The last batch will contain all the other artifacts from the previous batches. + + + + + +.build order of a `perl-bootstrap:5.32` module stream +[source,yaml,subs="verbatim,quotes"] +---- +⋮ +components: + rpms: + perl: + rationale: The Perl interpreter. + ref: f33 + buildorder: 0 + perl-Fedora-VSP: + rationale: A build dependency. + ref: f33 + buildorder: 1 + perl-generators: + rationale: A build dependency. + ref: f33 + buildorder: 2 + perl-Algorithm-Diff: + rationale: A build dependency. + ref: f33 + buildorder: 3 + ⋮ + perl-Config-AutoConf: + rationale: A build dependency. + ref: f33 + buildorder: 4 + ⋮ + perl-IO-Compress-Lzma: + rationale: A build dependency. + ref: f33 + buildorder: 5 + ⋮ + perl-Data-OptList: + rationale: A build dependency. + ref: f33 + buildorder: 6 + ⋮ + perl-Sub-Exporter: + rationale: A build dependency. + ref: f33 + buildorder: 7 + perl-Data-Section: + rationale: A build dependency. + ref: f33 + buildorder: 8 + perl-Software-License: + rationale: A build dependency. + ref: f33 + buildorder: 9 +⋮ +---- + +In this example we have the build order of the `perl-bootstrap:5.32` module. The component with the lowest `buildorder` number will start building first. In our case this is the package `perl` with the `buildorder` number 0. All the other component (package) builds need to wait until the first build batch is built. A build batch consists of packages which share the same `buildorder` number. After the `perl` package is built then the build batch has finished building all its components and can move to a higher order of the build order. The build of package `perl` needs to be added and enabled to the buildroot of the build of packages of the next `buildorder` number. In our case the next build batch is the build batch with the `buildorder` number 2. A build batch can have one or more packages. This means that the packages can be built simultaneously. Each build batch uses all the previous build batches as build dependencies which are available and enabled in its buildroots. The only build batch which does not have any previous build batch dependencies is the build batch with `buildorder` number 0. The only build batch which is not used as build batch dependencies is the build batch with the highest `buildorder` number. In our example the build batch with `buildorder` number 5 consists of 9 packages and number 4 consists of over 100 packages. All the packages built in build batches of `buildorder` numbers 0-4 will be used as build time dependencies for the build batch with the `buildorder` number 5. diff --git a/modules/ROOT/pages/core-concepts/module-context.adoc b/modules/ROOT/pages/core-concepts/module-context.adoc new file mode 100644 index 0000000..4792e1f --- /dev/null +++ b/modules/ROOT/pages/core-concepts/module-context.adoc @@ -0,0 +1,132 @@ +:toc: macro += Module Context + +A module stream can be built in/for several different environments. Each of those environments has its own configuration. We call this configuration a context configuration or a context for short. + + + + + +toc::[] + +:listing-caption: Example + +.input modulemd file context configuration of a `perl:5.32` module stream +[source,yaml,subs="verbatim,quotes"] +---- + . + . + . + configurations: + - context: '866eb18a' <.> + platform: f34 <.> + buildrequires: <.> + perl-bootstrap: ['5.32'] + buildopts: <.> + rpms: + macros: | + %_with_perl_enables_groff 1 + %_without_perl_enables_syslog_test 1 + %_with_perl_enables_systemtap 1 + ⋮ + - context: 'ab7e7bfe' + platform: f35 + buildrequires: + perl-bootstrap: ['5.32'] + buildopts: + rpms: + macros: | + %_with_perl_enables_groff 1 + %_without_perl_enables_syslog_test 1 + %_with_perl_enables_systemtap 1 + ⋮ + - context: '553f8e7a' + platform: f36 + buildrequires: + perl-bootstrap: ['5.32'] + buildopts: + rpms: + macros: | + %_with_perl_enables_groff 1 + %_without_perl_enables_syslog_test 1 + %_with_perl_enables_systemtap 1 + ⋮ + - context: 'd5eb079e' + platform: eln + buildrequires: + perl-bootstrap: ['5.32'] + buildopts: + rpms: + macros: | + %_with_perl_enables_groff 1 + %_without_perl_enables_syslog_test 1 + %_with_perl_enables_systemtap 1 + ⋮ + ⋮ +---- +<.> name of the context +<.> platform describes for which release you are building the context +<.> modular dependencies of a context +<.> build options which you want to enable in the buildroot of your context + +== Context name + +The name of the context specifically identifies a module stream. The name of the context is specified by the packager of the module stream. The name of the context should be the same during the whole lifetime of a module stream. If you change the name of an existing context it will automatically be considered as a new context within a module stream and there will be no connection between the new and the old context. This means that the upgrade path will end with the old context. The new context will be its own entity independent of the old context. + +=== Static vs Dynamic context + +When modularity was released the context configuration was generated automatically by the build system. In the `modulemd-packager` version 2 the name of a context was not present in the input modulemd YAML file. The context name was a hash which was generated by the build system from the modular `buildrequires` and `requires` of a module stream. The context name, modular dependencies and the number of resulting context combinations was known and added to the output modulemd YAML file post build. This had the benefit that the YAML file was concise and short. Also you did not need to know which platforms are available in the build pipeline or the specific names of module streams. The build system did that for you. + +But there were several issues with the dynamic context. First the modulemd YAML file did not reflect how a module stream should be built as the context configuration will be known after the build has taken place. Second issue was that you were totally reliant on a build pipeline of a distribution. The whole implementation was tightly coupled with the pipeline which led to an implicit behavior of the technology (i. e. users did not know what was going on if they did not possess intricate knowledge of modularity.) Third issue with dynamic context was that every time you changed your modular `requires` or `buildrequires` the name of the context was also changed. This led to a broken upgrade path between the old and new dynamic context names. DNF then could not identify precisely which module stream’s context name is the continuation of the old context. + +To fix this issue we introduced the idea of a static context configuration with the `modulemd-packager` version 3. The static context configuration needs to be precisely specified by the packager for every context. No automation is done by the pipeline. The context name is now an arbitrary string. For more information on the changes please check link:https://github.com/fedora-modularity/libmodulemd/tree/main/yaml_specs[the modulemd spec files]. + +== Platform + +The `platform` property of the context configuration represents a specific configuration for the Module Build System (MBS). The `platform` is also known as a `virtual module stream`. This defines for which release or specific target a module should be built. After the context configurations of a module stream are processed by the build system, the `platform` module stream can then be found in the dependencies section of the `modulemd` output file. + +.output modulemd of the `perl:5.32:3520210922083143:ab7e7bfe` module stream with platform virtual module stream as dependency +[source,yaml,subs="verbatim,quotes"] +---- +--- +document: modulemd +version: 2 +data: + name: perl + stream: "5.32" + version: 3520210922083143 + context: ab7e7bfe + static_context: true + arch: x86_64 + summary: Practical Extraction and Report Language + description: > + Perl is a high-level programming language with roots in C, sed, awk and shell + scripting. Perl is good at handling processes and files, and is especially good + at handling text. Perl's hallmarks are practicality and efficiency. While it is + used to do a lot of different things, Perl's most common applications are system + administration utilities and web programming. + license: + module: + - MIT + content: + - (Copyright only) and (Artistic or GPL+) + - (GPL+ or Artistic) and (GPLv2+ or Artistic) and MIT + - (GPL+ or Artistic) and Artistic 2.0 and UCD + ⋮ + xmd: {} + dependencies: + - buildrequires: + perl-bootstrap: [5.32] + platform: [f35] + requires: + platform: [f35] +⋮ +---- + +== Modular dependencies + +Each context configuration can have its own modular dependencies. This means that a module stream can depend on other module streams during build-time or run-time. The modular dependencies are defined by the packager in the `modulemd-packager` version 3. + +NOTE: Modular dependencies are another reason why module binary RPMs files can not be installed without modular metadata and why the `modularitylabel` flag exists. The information about modular dependencies is not included in the headers of RPM files. + +The `buildrequires` section of a context configuration can specify module streams which need to be enabled in a buildroot to be available for all of the components of a module stream. This can replace non-modular packages with their modular counterparts or introduce new packages which are not provided by the distribution itself. + +The same goes for the `requires` section of the context configuration. But those module streams need to be enabled during runtime so they can satisfy runtime dependencies of your module stream during installation. diff --git a/modules/ROOT/pages/core-concepts/module-dependency-resolution.adoc b/modules/ROOT/pages/core-concepts/module-dependency-resolution.adoc new file mode 100644 index 0000000..e6b45f6 --- /dev/null +++ b/modules/ROOT/pages/core-concepts/module-dependency-resolution.adoc @@ -0,0 +1,298 @@ +:toc: macro += Module dependency resolution + +Module streams are groups of RPM packages which provide alternatives to the default content which is present in a RPM repository. Module streams contain RPM packages which can have the same name as non-module RPM packages of the default content of a RPM repository. To be able to deliver the correct content to your system a few changes need to be introduced to the dependency resolution algorithm inside the DNF libraries and tools. + + + + + +toc::[] + + +In the next couple of paragraphs we will try to explain how DNF is processing requests for enabling, installing and switching module streams. The first example below represents the state of a RPM repository on a fresh system. Nothing is installed from the RPM repository. The RPM repository is a modular RPM repository as it contains two module streams `perl:5.24` and `per:5.26`. It also contains some non-modular packages perl, perl-VSP, foo and bar. + +NOTE: The following examples are more a higher level overview how modularity works under the hood. For specific instruction how to use module streams with DNF check the section xref:using-modules.adoc[Using modules]. + + ++++ + ++++ + +[[example-1]] +_Example 1. basic state of a modular repository_ + ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + +
non-modularperl:5.24perl:5.32
perl-1-f36.rpmperl-2-module_524.rpmperl-3-module_532.rpm
perl-Fedora-VSP-1-f36.rpmperl-Fedora-VSP-2-module_524.rpmperl-Fedora-VSP-1-module_532.rpm
foo-1-f36.rpmfoo-1-module_524.rpmbar-2-module_532.rpm
bar-1-f36.rpm
++++ + ++++ + + + + + + + + + + + + + + + + + +
un-available, can't be installed
available, not enabled
available, can be installed
installed
++++ + +The color coding illustrates the state of the software content and how DNF sees it. All the non-modular content is available for installation right from the get go as usual. All the content which is available for installation is color coded green. Also all the green content is included in a DNF dependency resolution transaction and content set creation when installing. The module streams are not available for installation as module streams need to be enabled first. Content which is available but not enabled is color coded yellow. Yellow content is not considered for DNF dependency resolution transactions and content set creation. If the content is in a state where it can not be installed it is color coded red. Red content is also not considered for DNF dependency resolution transactions and content set creation. Any installed content is color coded blue. + +== Enabling a module stream + +[[example-2]] +_Example 2. state of the RPM repository when the 5.24 stream is enabled_ + ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + +
non-modularperl:5.24perl:5.32
perl-1-f36.rpmperl-2-module_524.rpmperl-3-module_532.rpm
perl-Fedora-VSP-1-f36.rpmperl-Fedora-VSP-2-module_524.rpmperl-Fedora-VSP-1-module_532.rpm
foo-1-f36.rpmfoo-1-module_524.rpmbar-2-module_532.rpm
bar-1-f36.rpm
++++ + ++++ + + + + + + + + + + + + + + + + + +
un-available, can't be installed
available, not enabled
available, can be installed
installed
++++ + +A module stream needs to be enabled before its content can be installed. In example two we enabled the `perl:5.24` module stream. All the packages provided by the `5.24` stream are now available for installation. + +If we try to install the `perl` package the DNF transaction will include the package from the `5.24` stream. The non-modular packages will be filtered out by `modular filtering` during content set creation. The non-modular packages and the RPM file NEVRAs are not considered for installation. + +If there is a non-modular package with the same name as a package from an enabled module stream, this non-modular package is filtered out for the purpose of installation by `modular filtering`. Modular filtering guarantees that the correct packages are chosen for the DNF content set creation and for the following installation to your system. + +The only package which is not included in the `5.24` stream, but is also a non-modular package in the repository, is the `bar` package. The `bar` package can be still installed as normal as the enablement of `5.24` stream does not affect it as it is not a part of the `5.24` stream. + +NOTE: Everytime you choose to add a module stream to your RPM repository it can be used to replace already existing non-modular packages when enabled and installed. Please always check if your module stream will not break your dependencies of your other installed software, after enablement and installation of your module streams. + +The `5.26` stream is still available for enabling, but only one stream of the module name `perl` can be enabled at a time on a system. To be able to enable stream `5.26` we first need to reset and remove the installation of the `5.24` stream and all its packages. + +== Installing a module stream + +[[example-3]] +_Example 3. state of the RPM repository when all the available packages are installed_ + ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + +
non-modularperl:5.24perl:5.32
perl-1-f36.rpmperl-2-module_524.rpmperl-3-module_532.rpm
perl-Fedora-VSP-1-f36.rpmperl-Fedora-VSP-2-module_524.rpmperl-Fedora-VSP-1-module_532.rpm
foo-1-f36.rpmfoo-1-module_524.rpmbar-2-module_532.rpm
bar-1-f36.rpm
++++ + ++++ + + + + + + + + + + + + + + + + + +
un-available, can't be installed
available, not enabled
available, can be installed
installed
++++ + +When a module stream is enabled you can install all the packages which are included in the enabled module streams. Also you can install all non-modular packages which have a different package name as packages include in the enabled module stream. All other non-modular packages are filtered out by modular filtering. Module stream RPM files can have different NEVRAS (newer or older) than the non-modular RPM files. That is why we need modular filtering to ensure that only the correct RPM file NEVRAs are used for defining what is the latest content for the purpose of updating the installed software. + +== Switching a module stream + +[[example-4]] +_Example 4. state of the RPM repository when switching to 5.26 stream_ + ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + +
non-modularperl:5.24perl:5.32
perl-1-f36.rpmperl-2-module_524.rpmperl-3-module_532.rpm
perl-Fedora-VSP-1-f36.rpmperl-Fedora-VSP-2-module_524.rpmperl-Fedora-VSP-1-module_532.rpm
foo-1-f36.rpmfoo-1-module_524.rpmbar-2-module_532.rpm
bar-1-f36.rpm
++++ + ++++ + + + + + + + + + + + + + + + + + +
un-available, can't be installed
available, not enabled
available, can be installed
installed
++++ + +Before we can enable another module stream from the same module we have to first reset the already enabled module stream and remove it. In our situation in <> we have enabled the `5.24` stream of the `perl` module and installed it. To enable the `5.26` stream and install it we first need to uninstall all the packages from the `5.24` stream and reset it. + +The reset of the stream ensures that the state of the module stream in the DNF database is updated. Only after resetting a module stream, a new stream can be enabled and installed. + +WARNING: Always remove all installed packages from a module stream before resetting. It is possible to reset a module stream without removing the old RPM packages. The problem is that when enabling a new module stream and installing it, the new module stream will try to upgrade/downgrade the existing installed RPM packages. This is not desired and can cause broken dependency issues. More about this in the section xref:using-modules.adoc[Using Modules]. diff --git a/modules/ROOT/pages/core-concepts/module-lifecycles.adoc b/modules/ROOT/pages/core-concepts/module-lifecycles.adoc new file mode 100644 index 0000000..1b14e59 --- /dev/null +++ b/modules/ROOT/pages/core-concepts/module-lifecycles.adoc @@ -0,0 +1,9 @@ += Module stream lifecycles + +One of the major problems which modularity should solve is to enable different life cycles of software content. For example the life cycle of packaged software in the Fedora repository is for the lifetime of the release. There was no easy way to include software which would have life cycles closer to the upstream release cycle of a software. + +== EOL and obsolete metadata + +With EOL and obsolete metadata we can configure any module stream to be obsoleted by another one or set it to EOL. This metadata is interpreted by DNF as any other modular metadata and can be included into a RPM repository with the `createrepo_c` command. + +More about how to use modular EOL and obsolete metadata please refer to the xref:building-modules/fedora/module-obsoletes.adoc[Managing obsoletes] chapter and the link:https://github.com/fedora-modularity/libmodulemd/tree/main/yaml_specs[yaml specification file]. diff --git a/modules/ROOT/pages/core-concepts/module-stream.adoc b/modules/ROOT/pages/core-concepts/module-stream.adoc new file mode 100644 index 0000000..e2c8230 --- /dev/null +++ b/modules/ROOT/pages/core-concepts/module-stream.adoc @@ -0,0 +1,285 @@ +:toc: macro += Module Stream + +A module stream is the cornerstone of the Modularity project. A module stream represents a new entity which can be a part of a RPM repository. With the inclusion of module streams in a RPM repository the repository will be extended to a modular RPM repository. + +A module stream is a group of RPM packages which are built, installed and shipped together. The RPM packages are bound together by modular metadata. The artifacts of a module stream are binary and source RPM files and a module metadata (modulemd) yaml file. + + + + + +toc::[] + +:listing-caption: Example + +== Modular Metadata + +A standard RPM package consists of binary and source RPM files which are carrying all the necessary metadata inside the RPM file header. With the Modularity project we introduced a lot of additional metadata which describes how a module stream should be built, how it should be installed, but also how it fits and interacts within a modular repository. + +There are two basic types of modular metadata. The modulemd yaml file which we are using as an input for building a module stream and a modulemd yaml file which is an artifact of the module stream build process. Of course we have more types of modulemd but for now we will only focus on the input `modulemd-packager` and the output `modulemd` files. + +NOTE: For the sake of space, both examples of module streams were shortened. For example a Perl module stream consists of 170-180 components. If you want to check out the un-shortened version please check the link:https://src.fedoraproject.org/modules/perl[Perl module dist-git repository]. + +The specification of all the metadata formats used by Modularity can be found in link:https://github.com/fedora-modularity/libmodulemd/tree/main/yaml_specs[the libmodulemd git repository]. `libmodulemd` is a C library which provides an API (with python byndings) which enables us to work with module metadata. + +=== Input modulemd-packager + +.modulemd-packager metadata of a Perl module stream +[source,yaml,subs="verbatim,quotes"] +---- +--- +document: modulemd-packager <.> +version: 3 +data: + stream: '5.32' <.> + summary: Practical Extraction and Report Language + description: > + Perl is a high-level programming language with roots in C, sed, awk + and shell scripting. Perl is good at handling processes and files, and + is especially good at handling text. Perl's hallmarks are practicality + and efficiency. While it is used to do a lot of different things, + Perl's most common applications are system administration utilities + and web programming. + license: [MIT] + configurations: <.> + - context: '866eb18a' <.> + platform: f34 + buildrequires: <.> + perl-bootstrap: ['5.32'] + buildopts: <.> + rpms: + macros: | + %_with_perl_enables_groff 1 + %_without_perl_enables_syslog_test 1 + %_with_perl_enables_systemtap 1 + %_without_perl_enables_tcsh 1 + %_without_perl_Archive_Tar_enables_optional_test 1 + %_without_perl_autodie_enables_optional_test 1 + %_without_perl_Compress_Bzip2_enables_optional_test 1 + ⋮ + - context: 'ab7e7bfe' + ⋮ + - context: '553f8e7a' + ⋮ + - context: 'd5eb079e' + ⋮ + references: + community: https://www.perl.org/ + documentation: https://www.perl.org/docs.html + tracker: https://bugzilla.redhat.com/buglist.cgi?classification=Fedora&product=Fedora%20Modules&component=perl + profiles: <.> + common: + description: Interpreter and all Perl modules bundled within upstream Perl. + rpms: + - perl + minimal: + description: Only the interpreter as a standalone executable. + rpms: + - perl-interpreter + api: <.> + rpms: + - perl + - perl-Archive-Tar + - perl-Attribute-Handlers + - perl-autodie + - perl-AutoLoader + - perl-AutoSplit + - perl-autouse + - perl-B + - perl-base + - perl-Benchmark + ⋮ + filter: <.> + rpms: + - perl-tests + - perl-Compress-Raw-Bzip2-tests + - perl-Compress-Raw-Zlib-tests + - perl-Digest-tests + - perl-ExtUtils-MakeMaker-tests + - perl-IO-Compress-tests + - perl-Module-CoreList-tests + - perl-Storable-tests + - perl-URI-tests + components: <.> + rpms: + # We can rebuild in a random order because we have all binary + # packages available from perl-bootstrap. + perl: + rationale: The Perl interpreter. + ref: f33 + perl-Algorithm-Diff: + rationale: A run-time dependency. + ref: f33 + perl-Archive-Tar: + rationale: Core Perl API. + ref: f33 + perl-Archive-Zip: + rationale: A run-time dependency. + ref: f33 + perl-autodie: + rationale: Core Perl API. + ref: f33 + perl-bignum: + rationale: Core Perl API. + ref: f33 + ⋮ +... +---- +<.> the definition of the type of modulemd yaml document +<.> the name of the stream of a module stream +<.> definition of context configurations for different releases or build options +<.> each name of a context is defined by the user and uniquely identifies the builtime and runtime configuration combination of a module stream +<.> with modular dependencies a module stream can depend on other modules in buildtime or runtime +<.> with a context configuration we can set options which we want to be passed done to the build process for all the components of module stream +<.> profiles provide us with a feature to be able to install different combinations of components at runtime +<.> a list of binary RPM names which are provided by the module stream +<.> a list of binary RPM names which should be filtered out after building from the module stream +<.> a list of RPM packages which should be build, distributed, enabled and installed by the module stream + + +The `modulemd-packager` document is used for defining the whole of a module stream. The document is used as an input file to the build system you are using to build modules in. In our example we have a Perl module stream with the name `perl:5.32`. The name of a module stream consists of a combination of a module's name and the name of its stream. As you can see in the example, only the stream name (5.32) is defined. This is because, if you are building your module streams in Fedora you don't need to provide the stream name or the module name. Both are taken at build time from the name of the dist-git (module name) repository and its branch (stream name). For more information on the definition of `modulemd-packager` please refer to the link:https://github.com/fedora-modularity/libmodulemd/tree/main/yaml_specs[modulemd yaml file specifications]. + +The example has four context configurations. This means that this definition file will produce four different module stream builds for each context. Each context defines how, where and for what purpose a module stream is built. You can find more information on contexts in the xref:core-concepts/module-context.adoc[Module context section]. + +At the time of writing a standard Perl module stream needs to build around 180 RPM packages. Those RPM packages build around 400 source and binary RPM files. So when building four context configurations you get to around 1600 source and binary RPM files. + +When the input `modulemd-packager` file is processed by the build system it is split into output `modulemd` files which are copies of the `modulemd-packager`, but are updated with the context configuration of each context and the build process of the build system. In our example this would equal to four output `modulemd` yaml files. + +=== Output modulemd + +.output modulemd metadata of a Perl module stream created by the build system +[source,yaml,subs="verbatim,quotes"] +---- +--- +document: modulemd +version: 2 +data: + name: perl <.> + stream: "5.32" + version: 3520210922083143 <.> + context: ab7e7bfe <.> + static_context: true <.> + arch: x86_64 <.> + summary: Practical Extraction and Report Language + description: > + Perl is a high-level programming language with roots in C, sed, awk and shell + scripting. Perl is good at handling processes and files, and is especially good + at handling text. Perl's hallmarks are practicality and efficiency. While it is + used to do a lot of different things, Perl's most common applications are system + administration utilities and web programming. + license: + module: + - MIT + content: + - (Copyright only) and (Artistic or GPL+) + - (GPL+ or Artistic) and (GPLv2+ or Artistic) and MIT + - (GPL+ or Artistic) and Artistic 2.0 and UCD + - (GPL+ or Artistic) and BSD + ⋮ + xmd: {} <.> + dependencies: <.> + - buildrequires: + perl-bootstrap: [5.32] + platform: [f35] + requires: + platform: [f35] + references: + community: https://www.perl.org/ + documentation: https://www.perl.org/docs.html + tracker: https://bugzilla.redhat.com/buglist.cgi?classification=Fedora&product=Fedora%20Modules&component=perl + profiles: + common: + description: Interpreter and all Perl modules bundled within upstream Perl. + rpms: + - perl + minimal: + description: Only the interpreter as a standalone executable. + rpms: + - perl-interpreter + api: + rpms: + - perl + - perl-Archive-Tar + - perl-Attribute-Handlers + - perl-AutoLoader + - perl-AutoSplit + - perl-B + - perl-Benchmark + - perl-CPAN + ⋮ + filter: <.> + rpms: + - perl-Compress-Raw-Bzip2-tests + - perl-Compress-Raw-Zlib-tests + - perl-Digest-tests + - perl-ExtUtils-MakeMaker-tests + - perl-IO-Compress-tests + - perl-Module-CoreList-tests + - perl-Storable-tests + - perl-URI-tests + - perl-tests + buildopts: + rpms: + macros: > + %_with_perl_enables_groff 1 + %_without_perl_enables_syslog_test 1 + %_with_perl_enables_systemtap 1 + %_without_perl_enables_tcsh 1 + ⋮ + components: + rpms: + perl: + rationale: The Perl interpreter. + ref: f33 + arches: [aarch64, armv7hl, i686, ppc64le, s390x, x86_64] + perl-Algorithm-Diff: + rationale: A run-time dependency. + ref: f33 + arches: [aarch64, armv7hl, i686, ppc64le, s390x, x86_64] + ⋮ + artifacts: <.> + rpms: + - perl-4:5.32.1-471.module_f35+12589+8a7d3254.src + - perl-4:5.32.1-471.module_f35+12589+8a7d3254.x86_64 + - perl-Algorithm-Diff-0:1.1903-17.module_f35+11299+2c136bae.noarch + - perl-Algorithm-Diff-0:1.1903-17.module_f35+11299+2c136bae.src + - perl-Archive-Tar-0:2.38-3.module_f35+11299+2c136bae.noarch + - perl-Archive-Tar-0:2.38-3.module_f35+11299+2c136bae.src + - perl-Archive-Zip-0:1.68-3.module_f35+11299+2c136bae.noarch + - perl-Archive-Zip-0:1.68-3.module_f35+11299+2c136bae.src + - perl-Attribute-Handlers-0:1.01-471.module_f35+12589+8a7d3254.noarch + ⋮ +... +---- +<.> the module name added by the build system +<.> the module version added by the build system +<.> the module context name added from the context configuration of the input `modulemd-packager` +<.> marks this contexts as a static contexts. More on this in the xref:core-concepts/module-context.adoc[Module Context section]. +<.> the system architecture for which this modules stream was build +<.> `xmd` or extra metadata which can be added by the build system +<.> the modular runtime and build time dependencies of a module stream +<.> filter property filters out binary RPMs which should not be provided in the final module stream +<.> the list of source and binary RPM artifact filenames which are associated with this module + +Every `modulemd-packager` yaml file is split into `modulemd` files at build time. For each context defined we create its own `modulemd` file. The `modulemd` files describe the final form of a module stream build. The output `modulemd` file is the file to be included in the RPM repository together with the source and binary RPM files associated with the metadata. + +The `modulemd` file does not contain the context configuration section as the file is the result artifact of build of a module stream context combination. Each `modulemd` yaml file is processed and configured to its respective context configuration from its input `modulemd-packager` yaml file. + +The `artifacts` section is a list of source and binary RPM filenames which where built by the module stream build process. This section specifically describes which RPM files should be included into the DNF transaction when xref:using-modules.adoc[enabling and installing modules]. + +== Module profiles + +.the profiles property of `postgres:10` module stream +[source,yaml,subs="verbatim,quotes"] +---- +⋮ + profiles: + server: + rpms: + - postgres-server + client: + rpms: + - postgres +⋮ +---- + +The `profile` of a module stream describes the combination of RPM packages that can be installed together. For example the `postgres:10` module stream provides at the time of writing, two installation profiles `client` and `server`. diff --git a/modules/ROOT/pages/core-concepts/nsvca.adoc b/modules/ROOT/pages/core-concepts/nsvca.adoc new file mode 100644 index 0000000..15c578c --- /dev/null +++ b/modules/ROOT/pages/core-concepts/nsvca.adoc @@ -0,0 +1,165 @@ +:toc: macro += NSVCA vs NEVRA + + + + + +toc::[] + +== Naming and identifying modules in Fedora + +With the introduction of Modularity to the packager ecosystem it also introduced several challenges. First was that the NEVRA (name-epoch-version-release-architecture) package naming convention used in Fedora was insufficient. Modules are uniquely identified by the NSVCA naming convention, which stands for _name_, _stream_, _version_, _context_, and _architecture_. This page describes both the source-level ID (NSV) and the binary-level ID (NSVCA). + +== Source-level module ID (NSV) + +At the source level, modules are only identified by the first three: _name_, _stream_, and _version_. Name is defined as a name of the module’s repository in https://src.fedoraproject.org/[DistGit], stream is a name of the branch in https://src.fedoraproject.org/[DistGit], and version is the timestamp of a commit. + +=== Name + +Name of the module corresponds to the name of the application or the language stack it represents. +An example of a name could be postgresql for a PostgreSQL database module, or nodejs for a Node.js runtime. + +=== Stream + +Streams are variants of a module with a certain promise. + +In most cases, streams promise backwards compatibility with a major version of the application or the language stack they provide. For example, let’s say the Node.js runtime is supported in two major versions: 6 and 8. In this case, the module nodejs would have two streams: 6 and 8. + +However, streams can also promise different things such as stability. A good example of this is the calc package in Fedora which is maintained in two upstream branches: stable for the latest stable release and unstable for the latest development version. Using modularity, this package could be built as a calc module in two different streams: stable and unstable. + +In addition to the version promise, streams are also a way for packagers to communicate the level of maintenance. Does the maintainer plan to apply every minor patch? Will they apply security fixes quickly? Or is the module updated only twice a year? This can also be part of the promise. + +Another different example could be a stream that provides the software compiled using some experimental flags increasing the performance. + +Anyway, you get the idea. Streams are a very flexible and powerful tool. Use them wisely. + +=== Version + +Versions are just updates of a given stream. Technically, version is a number generated by the build system. Higher number always wins. This means that version does not identify a major/minor version of the software but the update/commit to a particular stream. + + +== Binary-level ID (NSVCA) + +Building a module from one source can result in multiple different binaries. Different binaries are typically produced for different architectures (i.e. x86_64, armv7hl, etc.) and different Fedora releases (i.e. Fedora 28, Fedora 29, EPEL 7, etc.). + +Module binaries use the full NSVCA — so in addition to the name, stream, and version fields described above, there are two more for binaries: architecture and context. + +=== Architecture + +Fedora is built for many different architectures. The architecture field simply distinguishes architecture-specific binaries from each other. The value is typically the same as with RPM packages, i.e. x86_64, armv7hl, etc. + +=== Context + +Context is used to distinguish binaries built for different Fedora releases. Thanks to stream expansion, modules can also be built against multiple streams of other modules, i.e. different versions of a language runtime etc. + +The value is generated by the build system and is usually hidden from the user as it doesn’t have any informational value by itself — it is a hash. However, the client tooling consuming this value can present it in a useful way. + +One way of representing the context could be listing the Fedora releases for which a certain module has been built. + +== NSVCA Definition + +This defines naming policy for modulemd metadata of final (built) modules. +This policy does **NOT** apply on sources such as modulemd yaml in dist-git. + +The goal is to provide unique identifiers for modules +that are both human readable and also suitable for machine processing. + +== Fields + +* **N** - Name +* **S** - Stream +* **V** - Version +* **C** - Context +* **A** - Arch +* **P** - Profile + + +== Separators + +Fields are separated with ':' (colon): N:S:V:C:A. + +If P is specified, it's separated from N:S:V:C:A with '/' (forward slash): N:S:V:C:A/P. + +=== Examples + +---- +# N:S:V:C:A +mariadb:3.6:1:0123abcd:x86_64 +# N:S:V:C:A/P +mariadb:3.6:1:0123abcd:x86_64/server +---- + +== Forms + +A form is a sequence of fields that fully or partially identifies a module. + +=== Full Forms + +N:S:V:C:A:: + Unique identifier of a module. +N:S:V:C:A/P:: + Unique identifier of a module profile. + + +=== Partial Forms + +Supported partial forms are: `N [ : S [ :V [ :C ] ] ] [ :A ] [ /P ]` + +Namely: + +* `N` +* `N::A` +* `N:S` +* `N:S::A` +* `N:S:V` +* `N:S:V::A` +* `N:S:V:C` +* `N:S:V:C:A` (identical to `N:S:V:C::A`) +* and all combinations with `/P` + +Missing fields **SHOULD** be populated with recommended defaults: + +Stream:: + defaults to the enabled or system default stream for the module in this particular order +Version:: + defaults to the latest available version in the module stream +Context:: + defaults to a value matching with already installed modules or modules involved in the transaction (not yet installed) +Arch:: + defaults to the system arch (e.g. DNF's $basearch) +Profile:: + defaults to the system default or 'default' profile + + +=== Allowed Characters + + +**N** - Name:: + a-z A-Z 0-9 . - _ + +**S** - Stream:: + a-z A-Z 0-9 . - _ + +**V** - Version:: + 0-9 +**C** - Context:: + 0-9 a-f +**A** - Arch:: + a-z A-Z 0-9 . - _ + +**P** - Profile:: + a-z A-Z 0-9 . - _ + + +All fields **MUST** start and end with an alphanumeric character: +a-z A-Z 0-9 + + +=== Forbidden Characters + +This paragraph serves as a design decision for future changes. + +Following characters **MUST NOT** be part of any field: + +* `:` (colon) - separator +* `/` (forward slash) - profile separator +* `\` (backslash) - comon control character +* `*` (asterisk) - common wildcard +* `?` (question mark) - common wildcard +* `@` (at) - grpspec in YUM and DNF +* ` ` (space) - common separator diff --git a/modules/ROOT/pages/core-concepts/upgrade-paths.adoc b/modules/ROOT/pages/core-concepts/upgrade-paths.adoc new file mode 100644 index 0000000..3a7230b --- /dev/null +++ b/modules/ROOT/pages/core-concepts/upgrade-paths.adoc @@ -0,0 +1,17 @@ += Upgrade Paths in Modularity + +On the package level, upgrades of a modular system work the same way as on a traditional system — using NEVRA comparison to determine which packages are the newest. There is, however, one additional step right before the NEVRA comparison that needs to happen on a modular system — limiting which packages are going to be part of the comparison based on what modules are enabled — and that is the key difference. + +There are up to three classes of RPM packages available to a modular system: + +1. **Standalone packages** (also refered-to as "bare RPMs" or "ursine RPMs") — packages not being part of a module. In Fedora, these are coming from the Everything repository. +2. **Modular packages** — packages being part of a module. In Fedora, these are coming from the Modular repository. +3. **Hotfixes** — standalone packages created on-demand by users or vendors meant to fix a critical issue before an official upgrade comes from the distribution. These need to be provided in a separate repository with a hotfix flag set. Fedora doesn't provide such packages. + +To determine the limited set of packages for the NEVRA comparison, the following algorithm is used: + +1. Take all standalone packages. +2. Add modular packages that are part of an enabled module, and potentially replace any standalone packages having a same name. To do this, look at the target modulemd rather than the current one. This step ensures that modular packages have always a higher priority than standalone packages. In other words, standalone packages can never upgrade modular packages. +3. Add all hotfix packages. These are just added, not replacing anything. That means hotfixes can potentially upgrade both traditional and modular packages. + +The next step is to take this set of packages, and run a NEVRA comparison to determine the highest version of each package. The highest versions are then installed as a part of the upgrade process. \ No newline at end of file diff --git a/modules/ROOT/pages/faq.adoc b/modules/ROOT/pages/faq.adoc index 3a709f6..9fd1347 100644 --- a/modules/ROOT/pages/faq.adoc +++ b/modules/ROOT/pages/faq.adoc @@ -20,14 +20,14 @@ How does this differ from Software Collections (SCLs)?:: Couldn’t we solve this with having compat packages, or putting versions into package names for different version streams?:: That could work for some scenarios, however, it would introduce an unnecessary complexity to the whole system by adding even more packages to choose from. You would also need to encode information such as the major version (stream) in the package name, in a similar way how SCLs do that. How would you distinguish packages built against different versions of runtimes or other dependencies? + + - Modularity groups packages into modules — a representation of an application, a language runtime, or any logical group. That makes the content more granular and easier to navigate. Modules can be available in multiple streams — usually representing a major version of the software they ship. And technologies such as xref::architecture/stream-expansion.adoc[Stream Expansion] allow packagers to automatically build their module against multiple streams of runtimes, or entire Fedora releases — significantly lowering the manual effort. And for users, the client tooling is smart enough to choose the right combinations without exposing the underlying complexity by default — making it easier to use. + Modularity groups packages into modules — a representation of an application, a language runtime, or any logical group. That makes the content more granular and easier to navigate. Modules can be available in multiple streams — usually representing a major version of the software they ship. Can I install more versions at once?:: As modularity uses standard RPM packaging in order to install software in the usual places, the answer is no. However, there are other existing technologies — such as containers — allowing exactly that and work well with Modularity. How is this different from RPM? Why not just different repos?:: Modularity uses RPM as the core building block. From the user perspective, replacing modules with repositories could be technically possible, however: + A) Modules as a core concept in the client tooling significantly simplifies usage. Users can easily query for the modules available and install them with a simple command, without the need of manually enabling repositories. + B) From our perspective, a repository is a source of software provided by a vendor, not necessarily representing a single application. We want to keep this principal and allow multiple applications to be present in a repository. + - C) Modularity also helps packagers to automatically submit multiple module builds at once based on their build dependencies, producing multiple combinations of binaries. And the client tooling then chooses the right binary when installing. See xref::architecture/stream-expansion.adoc[Stream Expansion] for more info. + C) Modularity also helps packagers to automatically submit multiple module builds at once based on their context configurations. Are you going to produce all versions? What life cycles and versions are envisioned?:: We are developing the technology such as the build pipeline and client tooling to enable contributors to build multiple versions. We envision longer life cycles of stacks mainly for server, and devel/rolling releases of some stacks for developers. diff --git a/modules/ROOT/pages/glossary.adoc b/modules/ROOT/pages/glossary.adoc index 3e98a84..851d892 100644 --- a/modules/ROOT/pages/glossary.adoc +++ b/modules/ROOT/pages/glossary.adoc @@ -1,12 +1,17 @@ +:toc: macro = Glossary Modularity introduces a lot of new keywords to the established packaging ecosystem. This is a short glossary to get you started on the basic terms. + + + + -== Modularity -* Modularity is an extension to the RPM ecosystem that allows to distribute and consume <> with alternative content. +toc::[] -== Module Repository -* Module repository is an RPM repository extended with <>. It allows users to query and fetch information about the available Module Streams as well as RPM packages from them. +== Modularity Project +* Modularity project is an extension to the RPM ecosystem that allows to distribute and consume <> with alternative content. + +== Modular Repository +* Modular repository is an RPM repository extended with <>. It allows users to query and fetch information about the available Module Streams as well as RPM packages from them. == Module Stream * Module Stream is a collection of RPM packages as defined by the <>. @@ -35,9 +40,4 @@ Modularity introduces a lot of new keywords to the established packaging ecosyst == NSVCA * This abbreviation describes the new naming conventions for <> i. e. Name:Stream:Version:Context:Architecture. -* For more information see xref:architecture/nsvca.adoc[this section]. - -== (Module) Stream Expansion -* Stream expansion is a feature of Modularity. It enables you to build your module streams for several OS releases. The <> specifies how and for what major releases of an operating system, your package will be built. -* For example: you can specify that mariadb:10 can be built for f29, f30, f31 and not built for f28 with specific options for each buildroot. -* For more information see xref:architecture/stream-expansion.adoc[this section]. \ No newline at end of file +* For more information see xref:core-concepts/nsvca.adoc[this section]. diff --git a/modules/ROOT/pages/index.adoc b/modules/ROOT/pages/index.adoc index 7e09b06..650e3a0 100644 --- a/modules/ROOT/pages/index.adoc +++ b/modules/ROOT/pages/index.adoc @@ -1,29 +1,48 @@ +:toc: macro = Introduction Welcome to the Modularity project documentation. -**Modularity** can be described as a set of tools that enables you to build your packages in an alternate way to the standard rpm packages. The major feature of Modularity is that it enables you to build multiple major versions of your software for multiple versions of Fedora releases. Modularity is a part of the “packaging ecosystem” that already exists and is used for building rpm packages in the Fedora. + +The **Modularity** project has been created for the purpose of enabling additional versions/flavours of RPM packages in one RPM repository. A standard RPM repository contains RPM packages which are uniquely identified by their name. Of course the package name is not the only package identifier, we have more identifiers (NEVRA) for more granular identification of packages. Basically it was not possible to have multiple RPM packages of the same name in one RPM repository. When we say multiple RPM packages with the same name, we mean RPM packages which are not updates, but RPM packages which are built with different options and features or are built from different source code. +In general, Fedora tends to ship the latest stable versions of software available. +At the same time, packages need to maintain a certain API/ABI stability throughout the life cycle of the release they are part of. +The problem with this approach is finding the right balance of being too fast vs. too slow. +For example, developers in general tend to prefer the newest versions of software, while server administrators want API/ABI stability for a longer period of time. +Modularity removes this limitation of a RPM repository with the addition of Module streams. + + + + + +toc::[] + +== Modular Repository + +In the picture below you can see a classic example of official RPM repositories in the Fedora linux distribution. Fedora Linux is providing one major version of RPM packages for each Fedora release. This is simple view of a “classic” or a non-modular repository. image::fedora-without-modularity.png[,100%,] -image::fedora-with-modularity.png[,100%,] +In the following picture we describe a xref:glossary.adoc#_modular_repository[modular repository]. A modular repository provides multiple module streams which contain additional RPM packages which can have the same name. A xref:glossary.adoc#_modular_repository[modular repository] can still provide RPM packages which are not part of any module but are part of the RPM repository. -== Multiple versions of software +image::fedora-with-modularity.png[,100%,] -Fedora provides only one supported major version of software per release. This means if you need to use an unsupported version you are on your own, you have to build it yourself or rely on some unofficial repository. Examples below can better illustrate the situation: +So in short, Modularity can expand a RPM repository with following features: -=== Examples +* Module streams add additional packages with the same name, but built from different sources and with different build configurations. +* Module streams can have different life cycles. For example a package in a distribution can have a different life cycle than the life cycle of the distribution. +* Module streams can have different installation profiles of its components i. e. you can define one or several module stream profiles which will contain groups of RPMs which should be installed together. -**Scenario 1**: Some users install packages coming from a different Fedora release in order to consume a specific version of a database that is compatible with their application. But thanks to Modularity they might not need to do that anymore, because multiple versions of the database can be available in each Fedora release. All they need to do is to consume the specific stream of that database right from the Fedora repositories for their system. +== When should you use Modularity? -**Scenario 2:** There were cases when users couldn't upgrade their system to a new Fedora release because their application wouldn't function with the new version of a language runtime coming with the upgrade. Modularity can fix this problem by providing the same language versions in both Fedora releases. With that, the user can consume a specific stream of the language and keep it even when they upgrade their system. And when the application is ready for the new language version, it can be upgraded later, independently from the OS, by switching to a different stream. +Any RPM repository can be converted into a modular RPM repository with the inclusion of modular metadata and corresponding binary RPMs. The following points will help you to decide if Modularity can be useful to you. -== Module -Module is a modular package, but instead of being only one rpm file as a non-modular package would be, it represents a collection of one or multiple rpm packages and metadata files which are the artifact of the module build process. The metadata of a module can define multiple rpm files and modules, which are bundled and installed on the host system. +* You need/want to provide additional major versions of your existing RPM packages i. e. postgresql 10, postgresql 11 within one Fedora release or a custom RPM repository. +* You need/want to provide additional custom flavours of your existing RPM packages i. e. custom build RPM packages with a variety of features. +* You need/want to have different life cycles of your packages in a RPM repository. +* You are building Flatpaks. RPM package dependencies installed into Flatpak sandoxes are module streams. +* In general Modularity is a good fit for providing alternative RPM packages for sandbox environments which use RPM based distributions. (Flatpak, linux containers etc.) -== Stream -A “stream”, in the context of the Modularity project, usually represents one major version of a software, but is not exclusive to. With a stream you can then distinguish between multiple versions of software for a Fedora release. Modules can also have a default stream which is chosen if no specific stream is defined by the user, when installing the software. +=== Examples -== Compatibility +**Scenario 1**: Some users install packages coming from a different Fedora release in order to consume a specific version of a database that is compatible with their application. But thanks to Modularity they might not need to do that anymore, because multiple versions of the database can be available in each Fedora release. All they need to do is to consume the specific stream of that database right from the Fedora repositories for their system. -As we already established earlier, Modules are only bundles of rpm packages which are installed together. This means that existing rpm packages can be bundled into a module. The workflow of installing modules is mostly the same as it is with non-modular packages. This is described in detail in the section xref:using-modules.adoc[Using modules]. +**Scenario 2:** There were cases when users couldn't upgrade their system to a new Fedora release because their application wouldn't function with the new version of a language runtime coming with the upgrade. Modularity can fix this problem by providing the same language versions in both Fedora releases. With that, the user can consume a specific stream of the language and keep it even when they upgrade their system. And when the application is ready for the new language version, it can be upgraded later, independently from the OS, by switching to a different stream. diff --git a/modules/ROOT/pages/installing-modules.adoc b/modules/ROOT/pages/installing-modules.adoc deleted file mode 100644 index 54cead2..0000000 --- a/modules/ROOT/pages/installing-modules.adoc +++ /dev/null @@ -1,65 +0,0 @@ -= Module Installation and Discovery - -== Discovering content - -=== Listing packages - -Packages available to the system can be discovered by the usual commands such as `dnf search NAME`, `dnf list NAME`, or by using the `dnf repoquery QUERY` command for more complex queries. However, it is important to note that those commands will, apart from traditional packages, only list modular packages coming from a _default_ or an _enabled_ module stream. - -=== Listing modules - -To list modules available to your system, and to see what streams are _default_ or have been _enabled_, use the following command: - - $ dnf module list - -== Consuming content - -=== Installing packages - -Packages can be installed the usual way by running the `dnf install NAME` command. Any traditional package, or a modular package coming from a _default_ or an _enabled_ module can be installed this way. - -Packages from other module streams can be consumed by either _enabling a module_ stream and then installing individual packages, or by _installing a module_ directly. - -=== Enabling modules - -To **enable a module stream** and make its packages available for installation, run the following command: - - $ dnf module enable NAME:STREAM - -For example, to make Node.js 8 packages available for installation, run: - - $ dnf module enable nodejs:8 - -Packages from enabled module streams can be then installed by the `dnf install NAME` command. - -=== Installing modules - -To install a module, use one of the following commands. Not specifying a _stream_ or a _profile_ causes DNF to choose the _default_. However, not every module has a _default stream_ or _default profile_. - - $ dnf module install NAME - $ dnf module install NAME:STREAM - $ dnf module install NAME/PROFILE - $ dnf module install NAME:STREAM/PROFILE - -For example, to install the Node.js 8 runtime and the client tooling of the default stream of MongoDB, run: - - $ dnf module install nodejs:8 - $ dnf module install mongodb/client - -=== Switching module streams - -NOTE: Switching streams is a risky operation that might not be always supported in packages, especially downgrades. - -Switching to a different stream than the one that is installed on a system is a two-step process. First, the current stream needs to be reset causing it not to be enabled anymore — this will however keep its packages installed. Second, a new stream needs to be installed. - - $ dnf module reset NAME - $ dnf module install NAME:STREAM - -For example, to switch from Node.js 8 to Node.js 10, run: - - $ dnf module reset nodejs - $ dnf module install nodejs:10 - -== Updating the system - -Updating a system by running the `dnf update` command causes all packages to be upgraded to their latest version provided by their module stream. \ No newline at end of file diff --git a/modules/ROOT/pages/removing-modules.adoc b/modules/ROOT/pages/removing-modules.adoc deleted file mode 100644 index f33762d..0000000 --- a/modules/ROOT/pages/removing-modules.adoc +++ /dev/null @@ -1,27 +0,0 @@ -= Removing modules - -In general, to remove a module installed on your system, use the following command: - - $ sudo dnf module remove MODULE:STREAM/PROFILE - -== Advanced - -There is a situation when a specific package has been installed first, and then a module has been installed after that. Example: - - $ sudo dnf install ruby - $ sudo dnf module install ruby:2.6/default - -In this case, running the `dnf module remove` command would not remove the `ruby` package, as DNF remembers that package has been explicitly installed. - -To make the `ruby` package uninstalled with the `dnf module remove` command, run the following: - - $ sudo dnf mark group ruby - $ sudo dnf module remove ruby:2.6/default - -That is because DNF remembers a reason why a package has been installed. There are three, sorted from the strongest: - -* __user__ -* __group__ -* __dependency__ - -Because modules use the __group__ reason, which is weker than __user__ used by the `dnf install` command, the package stays on the system after running the `dnf module remove` command. "Downgrading" it to __group__, however, makes the `dnf module remove` remove it as well. diff --git a/modules/ROOT/pages/using-modules-switching-streams.adoc b/modules/ROOT/pages/using-modules-switching-streams.adoc deleted file mode 100644 index 1c45990..0000000 --- a/modules/ROOT/pages/using-modules-switching-streams.adoc +++ /dev/null @@ -1,33 +0,0 @@ -= Switching module streams - -NOTE: This page needs to be extended. - -NOTE: Switching streams is a risky operation that might not be always supported in packages, especially downgrades. - -Switching to a different stream than the one that is installed on a system is a two-step process. First, the current stream needs to be reset causing it not to be enabled anymore — this will however keep its packages installed. Second, a new stream needs to be installed. - - $ sudo dnf module reset NAME - $ sudo dnf module install NAME:STREAM - -For example, to switch from Node.js 8 to Node.js 10, run: - - $ sudo dnf module reset nodejs - $ sudo dnf module install nodejs:10 - -When switching RPMs that are not in a profile of stream, that is a three step process. First, the current stream needs to be reset. Second, a new stream needs to be enabled. Three, the specified RPMs are synchronized on the new stream. - - $ sudo dnf module reset NAME - $ sudo dnf module enable NAME:STREAM - $ sudo dnf --allowerasing distro-sync [RPM]... - -For example, to switch installed RPMs from Ruby 2.5 to Ruby 2.6, run: - - $ sudo dnf module reset ruby - $ sudo dnf module enable ruby:2.6 - $ sudo dnf --allowerasing distro-sync - -To switch specified RPMs rubygem-mysql2 and rubygem-pg from Ruby 2.5 to Ruby 2.6, run: - - $ sudo dnf --allowerasing distro-sync rubygem-mysql2 rubygem-pg - -HINT: `dnf module info NAME:STREAM` Artifacts is helpful to check RPMs in a module. \ No newline at end of file diff --git a/modules/ROOT/pages/using-modules.adoc b/modules/ROOT/pages/using-modules.adoc index 14545f9..dd064b6 100644 --- a/modules/ROOT/pages/using-modules.adoc +++ b/modules/ROOT/pages/using-modules.adoc @@ -1,14 +1,301 @@ -= Using Modules in Fedora - -Modularity is a mechanism of making multiple versions of software available to your system. +:toc: macro += Using modules in Fedora **Modules** are special package groups usually representing an application, a language runtime, or a set of tools. They are available in one or **multiple streams** which usually represent a major version of a piece of software, giving you an option to choose what versions of packages you want to consume. To simplify installation, modules usually define one or more **installation profiles** that represent a specific use case. For example a `server` or a `client` profile in a database module. -And because having too many choices might be overwhelming, Fedora ships with a set of module **defaults** — so you only need to make choices when desired. +This is a quick overview how to use modules and module streams with the package manager DNF. + + + + + +toc::[] + +:listing-caption: Example + +== Module stream installation and discovery + +=== Listing packages + +Packages available to the system can be discovered by the usual commands such as `dnf search NAME`, `dnf list NAME`, or by using the `dnf repoquery QUERY` command for more complex queries. However, it is important to note that those commands will, apart from traditional packages, only list modular packages coming from already _enabled_ module stream. + +=== Listing modules + +To list modules available to your system, and to see what streams are _default_ or have been _enabled_, use the following command: + +[source,console,subs="verbatim,quotes"] +---- + $ dnf module list +---- + +=== Installing packages + +Packages can be installed the usual way by running the `dnf install NAME` command. Any traditional package, or a modular package coming from an _enabled_ module can be installed this way. + +Packages from other module streams can be consumed by either _enabling a module_ stream and then installing individual packages, or by _installing a module_ directly. + +=== Enabling modules + +To **enable a module stream** and make its packages available for installation, run the following command: + +[source,console,subs="verbatim,quotes"] +---- + $ dnf module enable NAME:STREAM +---- + +For example, to make Node.js 8 packages available for installation, run: + +[source,console,subs="verbatim,quotes"] +---- + $ dnf module enable nodejs:8 +---- + +Packages from enabled module streams can be then installed by the `dnf install NAME` command. + +=== Installing modules + +To install a module, use one of the following commands. Not specifying a _stream_ or a _profile_ causes DNF to choose the _default_. However, not every module has a _default stream_ or _default profile_. + +[source,console,subs="verbatim,quotes"] +---- + $ dnf module install NAME + $ dnf module install NAME:STREAM + $ dnf module install NAME/PROFILE + $ dnf module install NAME:STREAM/PROFILE +---- + +For example, to install the Node.js 8 runtime and the client tooling of the default stream of MongoDB, run: + +[source,console,subs="verbatim,quotes"] +---- + $ dnf module install nodejs:8 + $ dnf module install mongodb/client +---- + +=== Switching module streams + +NOTE: Switching streams is a risky operation that might not be always supported in packages, especially downgrades. + +Switching to a different stream than the one that is installed on a system done with the `switch-to` argument. + +[source,console,subs="verbatim,quotes"] +---- + $ dnf module switch-to NAME:STREAM +---- + +An example, to switch from Node.js 8 to Node.js 10, run: + +[source,console,subs="verbatim,quotes"] +---- + $ dnf module switch-to nodejs:10 +---- + +An example, to switch from Node.js 10 to Node.js 8, run: + +[source,console,subs="verbatim,quotes"] +---- + $ dnf module switch-to nodejs:8 +---- + +NOTE: The `switch-to` argument is the recommended and prefered way to switch streams. + +The `switch-to` argument includes multiple actions which need to be executed to safetly switch to another module stream. + +In this example we show a manual switch from Node.js 8 to Node.js 10 without the `switch-to` argument run: + +[source,console,subs="verbatim,quotes"] +---- + $ dnf remove nodejs + $ dnf module reset nodejs:8 + $ dnf module enable nodejs:10 + $ dnf install nodejs +---- + +NOTE: The `dnf module info NAME:STREAM` command is helpful to check RPMs in a module stream. + +== Updating the system + +Updating a system by running the `dnf update` command causes all packages to be upgraded to their latest version provided by their module stream. + +== Removing modules + +In general, to remove a module installed on your system, use the following command: + +[source,console,subs="verbatim,quotes"] +---- + $ dnf module remove MODULE:STREAM/PROFILE +---- + +== Advanced + +There is a situation when a specific package has been installed first, and then a module has been installed after that. Example: + +[source,console,subs="verbatim,quotes"] +---- + $ dnf install ruby + $ dnf module install ruby:2.6/default +---- + +In this case, running the `dnf module remove` command would not remove the `ruby` package, as DNF remembers that package has been explicitly installed. + +To make the `ruby` package uninstalled with the `dnf module remove` command, run the following: + +[source,console,subs="verbatim,quotes"] +---- + $ dnf mark group ruby + $ dnf module remove ruby:2.6/default +---- + +That is because DNF remembers a reason why a package has been installed. There are three, sorted from the strongest: + +* __user__ +* __group__ +* __dependency__ + +Because modules use the __group__ reason, which is weaker than __user__ used by the `dnf install` command, the package stays on the system after running the `dnf module remove` command. "Downgrading" it to __group__, however, makes the `dnf module remove` remove it as well. + +== Modular filtering and conflicts + +When consuming module streams a lot of operations are being executed. A module stream contains alternative modular RPM packages which can have the same name as already existing non-modular RPM packages, a lot of conflicts can arise. In the next few examples we will show you some of the conflicts. + +=== Switching a module stream without removing the RPM packages from the old module stream + +.broken dependencies when switching module streams without removing the installed content +[source,console,subs="verbatim,quotes"] +---- +[root@5e7d134a8883 /]# dnf module install nodejs:14/common +Last metadata expiration check: 0:02:38 ago on Thu Mar 3 08:57:00 2022. +Dependencies resolved. + + Problem 1: cannot install the best candidate for the job + - nothing provides /usr/bin/pwsh needed by nodejs-1:14.19.0-2.module_f35+13766+ad18d3e5.x86_64 + Problem 2: package npm-1:6.14.16-1.14.19.0.2.module_f35+13766+ad18d3e5.x86_64 requires nodejs = 1:14.19.0-2.module_f35+13766+ad18d3e5, but none of the providers can be installed + - cannot install the best candidate for the job + - nothing provides /usr/bin/pwsh needed by nodejs-1:14.19.0-2.module_f35+13766+ad18d3e5.x86_64 +=========================================================================================================================================================================== + Package Architecture Version Repository Size +=========================================================================================================================================================================== +Upgrading: + nodejs x86_64 1:14.17.2-2.module_f35+12348+fe4be0bd fedora-modular 93 k + nodejs-docs noarch 1:14.17.2-2.module_f35+12348+fe4be0bd fedora-modular 6.0 M + nodejs-full-i18n x86_64 1:14.17.2-2.module_f35+12348+fe4be0bd fedora-modular 7.8 M + nodejs-libs x86_64 1:14.17.2-2.module_f35+12348+fe4be0bd fedora-modular 13 M +Downgrading: + npm x86_64 1:6.14.13-1.14.17.2.2.module_f35+12348+fe4be0bd fedora-modular 3.3 M +Installing module profiles: + nodejs/common +Skipping packages with broken dependencies: + nodejs x86_64 1:14.19.0-2.module_f35+13766+ad18d3e5 updates-modular 199 k + npm x86_64 1:6.14.16-1.14.19.0.2.module_f35+13766+ad18d3e5 updates-modular 3.3 M + +Transaction Summary +=========================================================================================================================================================================== +Upgrade 4 Packages +Downgrade 1 Package +Skip 2 Packages + +Total download size: 30 M +Is this ok [y/N]: +---- + +In this situation we enabled and installed the `nodejs:12` module stream on Fedora 35. Then we reset the `12` stream and enabled the `14` stream. After that we wanted to install the `nodejs:14` module stream. + +First DNF is trying to upgrade or downgrade the existing installed software. It will not remove the existing software during the execution of the `reset` argument. The `reset` only disables the enabled stream of a module. The NEVRAs of the RPM files in the module stream can have different versions from the installed software (newer or older) and also from other streams from the same module. The runtime dependencies can be different between RPM packages from different module streams. + +This conflict is correct and is not a bug. As you are trying to update installed RPM packages from the `12` stream with RPM packages from the `14` stream. Module streams from the same module are mutually exclusive and only one stream at a time should be enabled and installed. + +If you are switching streams and not sure about the manual process, please use the `switch-to` argument as this is the recommended way. + + +=== Installing a RPM package from a wrong module stream. + +.of a RPM package requested to be installed from the wrong module stream +[source,console,subs="verbatim,quotes"] +---- +[root@bdaeaab947e6 /]# dnf module enable perl:5.30 +Fedora 35 - x86_64 6.8 MB/s | 79 MB 00:11 +Fedora 35 openh264 (From Cisco) - x86_64 3.1 kB/s | 2.5 kB 00:00 +Fedora Modular 35 - x86_64 2.8 MB/s | 3.3 MB 00:01 +Fedora 35 - x86_64 - Updates 7.3 MB/s | 27 MB 00:03 +Fedora Modular 35 - x86_64 - Updates 2.3 MB/s | 2.8 MB 00:01 +Dependencies resolved. +=========================================================================================================================================================================== + Package Architecture Version Repository Size +=========================================================================================================================================================================== +Enabling module streams: + perl 5.30 + +Transaction Summary +=========================================================================================================================================================================== + +Is this ok [y/N]: y +Complete! +[root@bdaeaab947e6 /]# dnf install perl-Tie-RefHash +Last metadata expiration check: 0:00:22 ago on Thu Mar 3 09:41:23 2022. +Error: + Problem: package perl-Tie-RefHash-1.40-478.fc35.noarch requires perl(:MODULE_COMPAT_5.34.0), but none of the providers can be installed + - conflicting requests + - package perl-libs-4:5.34.0-481.fc35.i686 is filtered out by modular filtering + - package perl-libs-4:5.34.0-481.fc35.x86_64 is filtered out by modular filtering + - package perl-libs-4:5.34.0-482.fc35.i686 is filtered out by modular filtering + - package perl-libs-4:5.34.0-482.fc35.x86_64 is filtered out by modular filtering +(try to add '--skip-broken' to skip uninstallable packages) +[root@bdaeaab947e6 /]# +---- + +The example describes a situation when you try to install a RPM package which is not provided from the enabled `perl:5.30` module stream. We first enable the `perl:5.30` module stream. After that we are trying to install the package `perl-Tie-RefHash`. The DNF package manager is trying to tell you that the `perl-Tie-RefHash` cannot be installed because it is not provided by any of the enabled module streams. Modular filtering will automatically filter out all the RPM packages which are not provided by the actual enabled module streams. + +To fix this conflict you have to enable the correct module stream. In our case it is the `perl:5.32` module stream. + + +=== Installing a RPM package without enabling the module stream which provides it. + + +.of a installation conflict when installing a package without enabling the correct module stream +[source,console,subs="verbatim,quotes"] +---- +[root@bdaeaab947e6 /]# dnf install perl-DBI +Last metadata expiration check: 3:13:30 ago on Thu Mar 3 09:41:23 2022. +Error: + Problem: package perl-DBI-1.643-10.fc35.x86_64 requires libperl.so.5.34()(64bit), but none of the providers can be installed + - conflicting requests + - package perl-libs-4:5.34.0-481.fc35.x86_64 is filtered out by modular filtering + - package perl-libs-4:5.34.0-482.fc35.x86_64 is filtered out by modular filtering +(try to add '--skip-broken' to skip uninstallable packages) +[root@bdaeaab947e6 /]# dnf module enable perl-DBI +Last metadata expiration check: 3:14:38 ago on Thu Mar 3 09:41:23 2022. +Dependencies resolved. +=========================================================================================================================================================================== + Package Architecture Version Repository Size +=========================================================================================================================================================================== +Enabling module streams: + perl-DBI 1.643 + +Transaction Summary +=========================================================================================================================================================================== + +Is this ok [y/N]: y +Complete! +[root@bdaeaab947e6 /]# dnf install perl-DBI +Last metadata expiration check: 3:14:46 ago on Thu Mar 3 09:41:23 2022. +Dependencies resolved. +=========================================================================================================================================================================== + Package Architecture Version Repository Size +=========================================================================================================================================================================== +Installing: + perl-DBI x86_64 1.643-7.module_f35+12493+425c54a8 fedora-modular 700 k + +Transaction Summary +=========================================================================================================================================================================== +Install 1 Package -Finally, because big changes are not always welcome, Modularity has been built in a way it can be basically invisible to the user. The usual installation commands continue to work — so packages with a default stream can be installed the same way as before regardles of them being modularized or not. +Total download size: 700 k +Installed size: 1.9 M +Is this ok [y/N]: +---- -Continue to the xref:installing-modules.adoc[Installation and Discovery] page to learn about the actual commands. +In this example we are trying to install the `perl-DBI` package. On our system we have previously enabled and installed the `perl:5.32` module stream. The conflict tells us that we are trying to install non-modular `perl-DBI` which depends on the non-modular `perl` but the non-modular `perl` package is not available due to modular filtering. The only RPM packages which are available for dependency resolution are the RPM packages from the `5.32` stream. + +WARNING: Non-modular RPM packages can not depend on modular content. If your non-modular package needs a modular dependency please modularize your content. +To fix this we need to enable the `perl-DBI:1.643` module stream. When enabling the `perl-DBI:1.643` we are satisfying the modular dependency for the `perl` module. The non-modular `perl-DBI` RPM package is now filtered by modular filtering and it is not considered in dependency resolution and content set creation. diff --git a/modules/ROOT/pages/using-other-build-systems.adoc b/modules/ROOT/pages/using-other-build-systems.adoc deleted file mode 100644 index 442810b..0000000 --- a/modules/ROOT/pages/using-other-build-systems.adoc +++ /dev/null @@ -1,3 +0,0 @@ -= Using Other Build System with Modularity - -NOTE: This page needs to be written.