#2 lots of content changes
Merged 6 years ago by asamalik. Opened 6 years ago by langdon.
fedora-docs/ langdon/modularity master  into  master

file modified
+3 -3
@@ -29,8 +29,6 @@ 

  #    Alias: en-US/brian

    - Name: What is Modularity

      File: index

-   - Name: FAQ

-     File: FAQ

    - Name: Architecture

      Dir: architecture

      Topics:
@@ -56,4 +54,6 @@ 

        - Name: Managing Streams using DNF

          File: test

    - Name: References

-     File: References 

\ No newline at end of file

+     File: References

+   - Name: FAQ

+     File: FAQ

file modified
+18 -10
@@ -2,27 +2,35 @@ 

  

  [qanda]

  Exactly what problem are you trying to solve?::

-     The “too fast vs. too slow” problem, outdated containers, and complex packaging workflow. See The Problem to be Solved page for more detailed information.

+     The “too fast vs. too slow” problem, outdated containers, and complex packaging workflow. 

+     See The Problem to be Solved page for more detailed information.

  How is this different from containers?::

      This is a build step that happens before containers.

      Modularity provides parallel availability of multiple versions that can be consumed by containers to enable parallel installation.

- How does this differ from SCLs?::

-     Modularity is about building multiple versions of packages, and making them available to users.

+ How does this differ from Software Collections (SCLs)?::

+     Modularity is about building multiple versions of packages, and making them all available to users.

      SCLs are about installing software in parallel using RPM.

-     These two don’t overlap, they complement each other. You can build an SCL using Modularity.

+     These two don’t overlap, they complement each other. You can build a Modular SCL.

  Couldn’t we solve this with having compat- packages, or putting versions into package names for different version streams?::

-     That works technically, but the packaging workload doesn’t scale, and dependency interactions become complex. This will let packagers make more content available to users without an explosion of effort.

+     That works technically, but the packaging workload doesn’t scale, and dependency interactions become complex. This will let packagers make more content available to users without an explosion of effort. 

+     Modules also help to solve the discoverability problem. 

+     For example, is httpd v2.4 in a package called httpd-2-4 or http-2?

  Can I install more versions at once?::

-     Modularity does not invent a new technology for parallel installation. Existing solution such as containers can be used to achieve this.

+     Modularity does not invent a new technology for parallel installation. 

+     Existing solution such as containers or Software Collections can be used to achieve this.

  How is this different from RPM? Why not just different repos?::

      Technical: DNF performance with many repositories. We envision having tens or even hundreds of modules available to a single system. +

-     User experience: A repository is a source of software provided by a vendor, not necessarily representing a single application, language stack, or any other logical unit of software. A repository typically contains multiple applications, stacks, etc. We do not want to change this concept.

+     User experience: A repository is a source of software provided by a vendor, not necessarily representing a single application, language stack, or any other logical unit of software. 

+     A repository typically contains multiple applications, stacks, etc. 

+     We do not want to change this concept.

  Are you going to produce all versions? What lifecycles and versions are envisioned?::

      We are developing the technology such as the build pipeline and client tooling to enable contributors to build multiple versions.

      We envision longer lifecycles of stacks mainly for server, and devel/rolling releases of some stacks for developers.

- I can just ‘dnf downgrade’ to get an older version.. why do we need modularity?::

-     Using an older version that is no longer maintain can be dangerous, mostly because of the lack of security updates.

+     However, contributors to the Fedora project will be the deciders and maintainers of any versions that are available.

+ I can just ‘dnf downgrade’ to get an older version.. Why do we need modularity?::

+     Using an older version that is no longer maintained can be dangerous, mostly because of the lack of security updates.

      Modules, on the other hand, represent a major version that is still being actively maintained by the upstream and therefore should receive updates.

  Is this different from single app VMs with custom versions?::

-     This helps with a single app VM with custom versions. Same as with containers, it helps users to get the right version for their system.

+     This helps with a single app VM with custom versions. 

+     Same as with containers, it helps users to get the right version for their system.

  

@@ -6,9 +6,11 @@ 

  

  image::mod-doc-trad-builds.png[,80%,]

  

- In the existing Fedora world, packages are maintained in branches that map to specific Fedora releases. Package repositories have branches e.g. “f27” for Fedora 27, “f28” for Fedora 28, etc.

+ In the existing Fedora world, packages are maintained in branches that map to specific Fedora releases. 

+ Package repositories have branches e.g. “f27” for Fedora 27, “f28” for Fedora 28, etc.

  

- There are no changes to the existing packager workflow. Maintaining release-specific branches and using the existing commands to build them keep working.

+ There are no changes to the existing packager workflow. 

+ Maintaining release-specific branches and using the existing commands to build them keep working.

  

  == Modular package builds

  
@@ -18,30 +20,36 @@ 

  

  Branching packages according to their major version as opposed to the distribution release brings more clarity — because we can see what lives in a branch, and removes source duplication — because there is one branch per version.

  

- While this approach makes more sense from the software perspective, there is no longer a clear mapping of a package branch to a particular release. Modules are a way of defining this.

+ While this approach makes more sense from the software perspective, there is no longer a clear mapping of a package branch to a particular release. 

+ Modules are a way of defining this.

  

  == Building one source for multiple releases

  

  image::mod-doc-build-more-releases.png[,40%,]

  

- Modules define what package branches are built for which Fedora releases. One module can be built against multiple releases.

+ Modules define what package branches are built for which Fedora releases. 

+ One module can be built for multiple releases.

  

- The above example produces three different binaries, one for each release. In case there are multiple architectures, three binaries will be produced for each one.

+ The above example produces three different binaries, one for each release. In case there are multiple architectures, three binaries will be produced (one each).

  

  == Building multiple versions

  

  image::mod-doc-build-more-versions.png[,40%,]

  

- Multiple versions of packages can be built for one release by defining multiple modules. Users are able to choose which version they want to use on their system.

+ Multiple versions of packages can be built for one release by defining multiple modules. 

+ Users are able to choose which version they want to use on their system.

  

- The above example produces two different binaries, each with a different Python version. Again, in case there are multiple architectures, two binaries will be produced for each one.

+ The above example produces two different binaries, each with a different Python version. 

+ Again, in case there are multiple architectures, two binaries will be produced for each one.

  

- Modularity also leverages existing technologies such as containers to enable users to run multiple versions at the same time.

+ Modularity also leverages existing technologies such as containers to enable users to run multiple versions at the same time on one "machine."

  

  == Building against multiple versions

  

  image::mod-doc-build-matrix.png[,60%,]

  

- Modules can be also built against other modules to achieve more complex results. Client tooling always makes sure the right version is installed for a given context.

+ Modules can be also be built against other modules to achieve more complex results. 

+ Client tooling always makes sure the right version is installed for a given context.

  

- The above example produces two different binaries, each bind to a different Python version. Again, in case there are multiple architectures, two binaries will be produced for each one.

+ The above example produces two different binaries, each bound to a different Python version. 

+ Again, in case there are multiple architectures, two binaries will be produced for each one.

@@ -1,6 +1,6 @@ 

  = Consuming Software with Modularity

  

- This is a high-level overview of how users can install software with Fedora Modularity and how is it different from the traditional workflows.

+ This is a high-level overview of how users can install software with Fedora Modularity and how is it different from the traditional workflow(s).

  

  == Understanding the Delivery Channels

  
@@ -8,7 +8,7 @@ 

  

  . The *traditional base repository* representing the distribution as we know it today — there are no user-visible changes in this part.

  

- . A new *AppStream repository* including all the additional versions delivered as modules.

+ . A new *modular repository*  (often referred to as the "Application Stream" or AppStream for short) including all the additional versions delivered as modules.

  

  The Modules repository will be optional for users.

  
@@ -16,32 +16,45 @@ 

  

  == Consuming the traditional packages

  

- There are no changes to the traditional user experience. Packages from the traditional repository will be installed and updated using the same ways as before. Everything keeps working as it used to.

+ There are no changes to the traditional user experience. 

+ Packages from the traditional repository will be installed and updated using the same methods as before. 

+ Everything keeps working as it used to.

  

  == Consuming the modular packages

  

- If a user desires to use the optioanl Modular repositories in order to consume non-default versions of software, there will be some new concepts introduced in the client tooling to manage them.

+ If a user desires to use the optioanl Modular repositories in order to consume non-default versions of software, there will be some new concepts introduced in the client tooling to manage them. 

+ We outline these below.

  

  === Enabling a module

  

- Enabling a module makes its packages available on the system. Packages delivered as part of modules modules have always a priority over the ones from the traditional base, regardless of their actual version. Packages in modules are replacements of the ones in the traditional base.

+ Enabling a module makes its packages available on the system. 

+ Packages delivered as part of a module always have a priority over the ones from the traditional base, regardless of their actual version. 

+ Packages in modules are often replacements of the ones in the traditional base.

  

- Modularity brings parallel availability, not parallel installability. Only one stream of a given module can be enabled on a system — so it is always clear which version gets installed. Installing and running multiple versions of software can be achieved by using existing technologies, such as containers.

+ Modularity brings parallel availability, not parallel installability. 

+ Only one stream of a given module can be enabled on a system — so it is always clear which version gets installed. 

+ Installing and running multiple versions of software can be achieved by using existing technologies, such as containers.

  

  === Installing a module

  

- To make installation easy, some modules can be also installed as a unit, without the need of enabling them first and then installing individual packages. Installing a module doesn’t necessarily mean installing all of its packages. Modules can define something called installation profiles to help users with the installation.

- Installation profile¶

+ To make installation easy, some modules can be also installed as a unit, without the need of enabling them first and then installing individual packages. 

+ Installing a module doesn’t necessarily mean installing all of its packages. 

+ Modules can define something called an "installation profile" to help users with the installation.

+ 

+ ==== Installation profile

  

  Installation profiles are essentially lists of packages that help users with the module installation. To give a specific example, a database module could have two profiles: server and client. This helps the user to install what they need without the need of thinking about the package names. However, installation profiles are just an optional feature and users can still install packages directly.

- Updating the system¶

  

- Updating the system always respects user’s choice even when there are multiple (and possibly higher) versions available.

+ ==== Updating the system

+ 

+ Updating the system always respects user’s module choices (or lack of module) even when there are multiple (and possibly higher) versions available.

  

- If the user doesn’t enable any modules, all packages on their system get updated to the latest versions provided by the traditional base repository. However, if the user enables a module stream, packages get updated to the newest version provided by the module.

+ If the user doesn’t enable any modules, all packages on their system get updated to the latest versions provided by the traditional base repository. 

+ However, if the user enables a module stream, packages get updated to the newest version provided by the module.

  

  Thanks to this mechanism, the user has better control over the versions of packages on their system while receiving updates (such as security patches) for the whole system.

- Running multiple versions using containers¶

+ 

+ ==== Running multiple versions using containers

  

  Modularity brings parallel availability, not parallel installability. There are other technologies i.e. linux containers or software collections that deal with this.

  

@@ -1,6 +1,6 @@ 

  = Main Concepts of Modularity

  

- Modularity reuses some existing and creates some new concepts of the build and release process in Fedora.

+ Modularity reuses some existing and creates some new concepts in the build and release process in Fedora.

  

  == Packages

  
@@ -8,13 +8,21 @@ 

  

  We are reusing packages as they are.

  

- Packages are the smallest unit of software delivery. Right now, Fedora uses RPM packages to build and ship all of its software. There is a distinction between source RPM (SRPM) packages and binary RPM packages.

+ Packages are the smallest unit of software delivery. 

+ Right now, Fedora uses RPM packages to build and ship all of its software. 

+ There is a distinction between source RPM (SRPM) packages and binary RPM packages.

  

- As the name suggests, *source packages* contain the source code, but also the build process definition including a list of build dependencies. Building a source package results in one or multiple binary packages.

+ As the name suggests, *source packages* contain the source code, but also the build process definition including a list of build dependencies. 

+ Building a source package results in one or multiple binary packages.

  

- *Binary packages* are produced by building source packages in a build environment called buildroot. Different buildroots produce different binary packages. This means that a source package without a context is not a complete definition of a resulting binary package.

+ *Binary packages* are produced by building source packages in a build environment called a buildroot. 

+ Different buildroots produce different binary packages. 

+ This means that a source package without a context is not a complete definition of a resulting binary package.

  

- Every Fedora release has its own buildroot. Even when multiple releases contain the same package using the same source, they are built multiple times, once in each buildroot, producing different (and often incompatible) binary packages.

+ Every Fedora release has its own buildroot. 

+ Even when multiple releases contain the same package using the same source, they are built multiple times, once in each buildroot, producing different (and often incompatible) binary packages.

+ Also, over time, as the distribution gets updated, so does its buildroot. 

+ As a result, a package built today may not result in the same binary as a package built tomorrow.

  

  *Packages are either a part of a release, or a part a module.*

  
@@ -22,27 +30,36 @@ 

  

  image::mod-doc-release.png[,80%,]

  

- Releases are the OS generations of Fedora e.g. “Fedora 28”, “Fedora 29”, or “epel 7”.

+ Releases are the generations of Fedora e.g. “Fedora 28”, “Fedora 29”, or “epel 7”.

  

- Each release can only contain one major version of every package. In general, Fedora tends to ship the latest stable versions of software available. At the same time, packages need to maintain a certain API/ABI stability throughout the lifecycle of the release they are part of.

+ Each release can only contain one major version of every package (by policy). 

+ In general, Fedora tends to ship the latest stable versions of software available. 

+ At the same time, packages need to maintain a certain API/ABI stability throughout the lifecycle of the release they are part of.

  

- The problem with this approach is finding the right balance of being too fast vs. too slow. For example, developers in general tend to prefer the newest versions of software, while server administrators want API/ABI stability for a longer period of time. This problem is solved by Modules.

+ The problem with this approach is finding the right balance of being too fast vs. too slow. 

+ For example, developers in general tend to prefer the newest versions of software, while server administrators want API/ABI stability for a longer period of time. 

+ This problem is solved by Modules.

  

  == Modules and Streams

  

  image:mod-doc-module.png[,80%,]

  

- *Modules* are collections of packages representing a logical unit e.g. an application, a language stack, a set of tools. These packages are built, tested, and released together.

+ *Modules* are collections of packages representing a logical unit e.g. an application, a language stack, a set of tools. 

+ These packages are built, tested, and released together.

  

  Each module defines its own lifecycle which is closer to the upstream rather than the Fedora release.

  

- Modules are the mechanism of delivering multiple streams (versions) of software for the same release. This also works the other way round, the same stream for multiple releases.

+ Modules are the mechanism of delivering multiple streams (versions) of software for the same release. 

+ This also works the other way round, the same stream for multiple releases.

  

  image::mod-doc-stream.png[,80%,]

  

- A *stream* is a series of versions having a goal such as backwards API/ABI compatibility or delivering the latest stable version. A “nodejs” interpreter can have streams such as “6” or “8”, but also “devel” or “latest”. Some packages don’t have a major version. An example could be the “calc” package that is being developed in two branches “stable” and “unstable”.

+ A *stream* is a series of versions having a goal such as backwards API/ABI compatibility or delivering the latest stable version. 

+ A “nodejs” interpreter can have streams such as “https://src.fedoraproject.org/modules/nodejs/branch/6[6]” or “https://src.fedoraproject.org/modules/nodejs/branch/8[8]” (two current versions of nodejs), but also “devel” or “latest”. 

+ Some packages don’t have a major version. 

+ An example could be the “calc” package that is being developed in two branches “stable” and “unstable”.

  

- Simply put, streams are the mechanism of delivering and updating software at different speeds, allowing developers to have a bleeding-edge software, and server admins longer stability.

+ Simply put, streams are the mechanism of delivering and updating software at different speeds, allowing developers to have bleeding-edge software, and server admins longer stability.

  

  == Artifacts

  
@@ -50,4 +67,5 @@ 

  

  Artifacts are consumable outputs of the build and release processes e.g. an iso image, an RPM repository, a container base image, or an ostree tree.

  

- Modules will be delivered as an RPM repository that will work on top of existing releases. This repository will also be available in the Fedora container base image, enabling users to build containers with multiple versions of software.

+ Modules will be delivered as an RPM repository that will work on top of existing releases. 

+ This repository will also be available in the Fedora container base image, enabling users to build containers with multiple versions of software.

file modified
+30 -10
@@ -1,6 +1,7 @@ 

  = What is Modularity

  

- Modularity introduces a new optional repository to Fedora called AppStream that ships additional versions of software on independent lifecycles. This way users can keep their operating system up-to-date while having the right version of an application for their use case, even when the default version in the distribution changes.

+ Modularity introduces a new optional repository to Fedora called Modular (often referred to as the "Application Stream" or AppStream for short) that ships additional versions of software on independent lifecycles. 

+ This way users can keep their operating system up-to-date while having the right version of an application for their use case, even when the default version in the distribution changes.

  

  image::modularity-appstream-overview.png[,100%,]

  
@@ -9,25 +10,44 @@ 

  

  === Too fast vs. too slow

  

- Different users have different needs. Developers want the latest versions possible, system administrators want stability for longer period of time. There are many Linux distributions out there, each targetting a different audience. A good example is Fedora and CentOS.

+ Different users have different needs. 

+ Developers want the latest versions possible, system administrators want stability for longer period of time. 

+ There are many Linux distributions out there, each targetting a different audience, for example Fedora vs CentOS. 

  

- Fedora generally ships the latest stable versions of its component packages when it is released twice per year. That is convenient for desktop users and developers. Even though many people use Fedora on a server, it is sometimes necessary to have a stable version of certain packages for a longer time, mostly because of third-party applications.

+ Fedora generally ships the latest stable versions of its component packages when it is released twice per year.

+ That is convenient for desktop users and developers. 

+ Even though many people use Fedora on a server, it is sometimes necessary to have a stable version of certain packages for a longer time, mostly because of third-party applications.

  

- At the same time, some people consider Fedora to be moving too slow for them and want even newer runtimes on their system. Some upstreams release their software faster than twice a year.

+ At the same time, some people consider Fedora to be moving too slowly for them and want even newer runtimes on their system. 

+ Some upstreams release their software faster than twice a year.

  

- On the other hand, CentOS targets long-term stability and releases a new version once every few years. This is convenient for server admins as there are fewer changes over longer periods of time. But some of the software gets old for modern applications and newer versions of runtimes might be needed.

+ On the other hand, CentOS targets long-term stability and releases a new version once every few years. 

+ This is convenient for server admins as there are fewer changes over longer periods of time. 

+ But some of the software gets old for modern applications and newer versions of runtimes might be needed.

  

- In other words, it would be convenient to be able to choose some parts of the system to be moving slow, and other parts to be moving fast. Could we do that?

+ In other words, it would be convenient to be able to choose some parts of the system to be moving slowly, and other parts to be moving faster. 

+ Can we do that?

+ 

+ https://example.com[YES!]

  

  === Outdated containers

  

- There are many containers out there. Majority of them are built manually, not actively maintained, not patched with security fixes, and still used by many people. This is especially true for the ones having software in a different version than the distribution provides.

+ There are many containers out there. 

+ The majority of them are built manually, not actively maintained, not patched with security fixes, and still used by many people. 

+ This is especially true for the ones using software of a different version than the distribution their are based on provides.

+ 

+ If Fedora had multiple versions of software that is actively maintained and built, could we use it to produce containers? 

+ Could these containers get automatically rebuilt every time the packages get updated?

  

- If Fedora had multiple versions of software that is actively maintained and built, could we use it to produce containers? Could these containers get automatically rebuilt every time the packages get updated?

+ https://example.com[YES!]

  

  === Complex packager workflows

  

- Fedora contributors maintain their packages in multiple branches — one for each release. Even when the packages are the same. And there is a series of manual steps associated with the build process.

+ Fedora contributors maintain their packages in multiple branches — one for each release. 

+ Even when the packages are the same. 

+ Leading to a series of manual steps associated with the build process.

  

- Could we enable packagers to maintain packages in branches that would map the package version instead of an arbitrary distribution release? Having a single branch that builds across multiple releases would save some work for packagers.

+ Could we enable packagers to maintain packages in branches that would follow the package version instead of an arbitrary distribution release version? 

+ Having a single branch that builds across multiple releases would save some work for packagers.

  

+ https://example.com[YES!]