From 530136a5222d2c4844398d033d88bdb98c91ed7d Mon Sep 17 00:00:00 2001 From: mprahl Date: Aug 02 2019 19:49:47 +0000 Subject: Add documentation about dependency resolution This was written with the help of Jan Kaluža. Co-authored-by: Jan Kaluža --- diff --git a/docs/DEPENDENCY_RESOLUTION.rst b/docs/DEPENDENCY_RESOLUTION.rst index 8dac714..1704ecc 100644 --- a/docs/DEPENDENCY_RESOLUTION.rst +++ b/docs/DEPENDENCY_RESOLUTION.rst @@ -1,3 +1,93 @@ +How Dependency Resolution Works in MBS +====================================== + +#. Evaluate any buildrequire or require overrides submitted manually by the user. + + #. If there is none, the MBS configuration of ``BR_STREAM_OVERRIDE_MODULE`` and + ``BR_STREAM_OVERRIDE_REGEXES`` are used to parse the branch name, to see if there are any + buildrequires that should be overridden. + +#. Perform module stream expansion. + + #. If a buildrequire is an empty list, then MBS gets all the latest versions of every available + stream in all contexts. + + #. If a buildrequire contains only streams with ``-`` prefix, the list of streams are treated as + a blacklist. + + #. For example, if the buildrequire is ``platform: ["-f29"]``, then the module is built for + all the streams of ``platform`` except the ``f29`` one. + +#. Get all the compatible latest buildrequires and recursively the requires of the buildrequires. + + #. Compatibility is determined by finding the compatible ``platforms`` that the module is + buildrequiring. For example, if you buildrequire ``platform: el8.1.0``, and you buildrequire + module ``foo`` that was built against ``platform: el8.0.0``, then that module ``foo`` is + compatible as a buildrequire. The reason is that all modules that have been built with a lower + ``platform`` of the same major version, are considered compatible. These ``platform`` modules, + must also all provide the same virtual stream. In the case of RHEL, that is a virtual stream + of ``el8``. + +#. If there are multiple sets of dependencies in the ``dependencies`` list, then these are treated + like all the possible valid combinations, and MBS will build the input module for each + combination if the dependency resolution succeeds for that combination. + +#. Resolve all the possible combinations of buildrequires. + + #. For each resolved buildrequire combination, a modulemd file is generated based on the original + modulemd, but with the buildrequires/requires changed based on the resolved combinations. + + #. If the buildrequire and require streams are the same on the original modulemd for a + particular module, then set the required stream on that module to the same as the + buildrequired stream. For examples, see the table below. + + #. In the event that you buildrequire two streams of two different modules, four module builds + are generated. + + #. MBS records the resulting buildrequires (and the recursive requires of the buildrequires) in + the ``xmd`` section of the modulemd. + + #. MBS records the context in the modulemd file since the buildrequires have been determined. + +#. Each generated modulemd file is submitted as a module build. + + +Examples of buildrequire and require changes from resolved combinations +----------------------------------------------------------------------- + ++--------------------------+-------------------------+----------------------------+-----------------------+ +| Buildrequires | Requires | Resulting Build #1 | Resulting Build #2 | ++==========================+=========================+============================+=======================+ +| .. code:: yaml | .. code:: yaml | .. code:: yaml | .. code:: yaml | +| | | | | +| platform: [f29, f30] | platform: [f29, f30] | buildrequires: | buildrequires: | +| | | - platform: [f29] | - platform: [f30] | +| | | requires: | requires: | +| | | - platform: [f29] | - platform: [f30] | ++--------------------------+-------------------------+----------------------------+-----------------------+ +| .. code:: yaml | .. code:: yaml | .. code:: yaml | .. code:: yaml | +| | | | | +| platform: [f29, f30] | platform: [f30] | buildrequires: | buildrequires: | +| | | - platform: [f29] | - platform: [f30] | +| | | requires: | requires: | +| | | - platform: [f30] | - platform: [f30] | ++--------------------------+-------------------------+----------------------------+-----------------------+ +| .. code:: yaml | .. code:: yaml | .. code:: yaml | | +| | | | | +| platform: [f30] | platform: [f29, f30] | buildrequires: | | +| | | - platform: [f30] | | +| | | requires: | | +| | | - platform: [f29, f30] | | ++--------------------------+-------------------------+----------------------------+-----------------------+ +| .. code:: yaml | .. code:: yaml | .. code:: yaml | | +| | | | | +| platform: [f29] | platform: [f29] | buildrequires: | | +| | | - platform: [f29] | | +| | | requires: | | +| | | - platform: [f29] | | ++--------------------------+-------------------------+----------------------------+-----------------------+ + + How libsolv Works in MBS ======================== @@ -38,3 +128,89 @@ Libsolv Terms solvables in the pool. - **Transaction** - this describes the solution from the solver execution. In MBS, this is always about installing the solvable that represents the module being built. + + +How It's Used +------------- + +There are two repos initialized in the constructor the ``MMDResolver`` class: ``build`` and +``available``. The ``build`` repo contains solvable objects that are created to represent the input +module to resolve. The ``available`` repo contains the solvable objects of the possible build +dependencies of the input module. + +There are two main methods: ``add_modules`` and ``solve``. + +add_modules +~~~~~~~~~~~ + +#. Gets the NSVC of the input module. + +#. If the context is set, then it’s treated as a dependency. + + #. A solvable object is created in the ``available`` repo with the name, version, and + architecture (hard-coded to "x86_64" since libsolv requires an architecture, but MBS is + architecture agnostic for dependency resolution). + + #. Fill in the ``Provides`` for the module by creating a ``Dep`` object. + + #. If it’s not a base module, it provides: + + #. ``module(foo)`` + #. ``module(foo:stream) = 2019`` + + #. The version is just used to find the latest version by libsolv. + + #. If it's a base module, it provides: + + #. ``module(platform)`` + + #. ``module(platform:el8.0.0) = 3`` + + #. This shouldn't be defined if a stream version is set (see #1334). + + #. ``module(platform:el8.0.0) = 80000`` + + #. ``80000`` is the "stream version" and not the version of the module. + + #. For each virtual stream if there is a "stream version": + + #. ``module(platform:virtual_stream) = stream_version`` + + #. Fills in the ``Requires``. + + #. ``_deps2reqs`` is called, which translates the elements of the dependencies array in the + modulemd to a libsolv ``Dep`` object. So for a simple example, with the input + ``deps = [{'gtk': ['1'], 'foo': ['1']}]``, the resulting ``solv.Dep`` expression will be + ``((module(gtk) with module(gtk:1)) and (module(foo) with module(foo:1)))``. + + #. Fills in the ``Conflicts``. + + #. This is so that modules of the same stream cannot both be used. For example, + ``module(bar:1)`` will conflict with ``module(bar)``, meaning any other module that also + provides ``module(bar)``. + +#. If the context is not set, it’s treated as the input module. + + #. For each buildrequires/requires pair, a solvable object is created in the ``build`` repo with + the name, version, and architecture (always ``src``). + + #. The context of the solvable is the index of the buildrequires/requires pair. This is later + used by the MSE code to distinguish the buildrequires/requires pair, and then which to keep in + the final modulemd and which to remove. + + #. The requires are filled in by ``_deps2reqs``, as described above. + + +solve method +~~~~~~~~~~~~ + +#. The input modulemd is of the input module. + +#. For each solvable in the ``build`` repo: + + #. Create a libsolv job to "install" the module. + + #. Iterate over all possible combinations of streams without trying to parallel install any + module. As part of this iteration, this is done by telling libsolv to favor that combination. + If what libsolv resolves is the same combination that was favored, we know it’s a valid + combination.