#12 Enable build dependency resolution
Closed Opened by ncoghlan.

Build dependency resolution is currently disabled because it fails for grep, as the solver currently expects to be able to find all of the platform module's build dependencies, not just grep's build dependencies.

This fails, since the platform module deliberately isn't self-hosting - you need a whole pile of additional packages to actually build it, and those are deliberately excluded from the modular release.

To fix this, the definition of "build dependencies" needs to be limited to just the direct build dependencies of the given packages (along with the runtime dependencies of those build dependencies), without transitively including the build dependencies of the build dependencies.

Checking for expected output with sudo dnf builddep grep gives the list:

gettext
autoconf
automake
pcre-devel
texinfo
perl-Class-Inspector
perl-Unicode-EastAsianWidth
perl-libintl-perl
perl-File-ShareDir (weak dependency, so probably OK to skip)

@ignatenkobrain Am I right in thinking that https://github.com/fedora-modularity/dependency-report-scripts currently deals with this by including the full Fedora package set in its candidates for build dependency resolution, rather than just the components available in the base Fedora Modular Server compose?

@ncoghlan dependency-report-scripts do not use neither full Fedora package set nor Modular Server compose. It uses repository provided by baseruntime-package-lists which is some mixture of f27, f28 which later becomes Modular Server.

@ignatenkobrain Watching the list of packages being downloaded by "make repo/devel" in https://github.com/fedora-modularity/baseruntime-package-lists/, I think my theory above is a pretty reasonable one - those package sets include a lot of stuff, so I expect they'll pass repoclosure for basically any package in Fedora.

At the same time, I'm also realising that may actually be what we need as an input to fedmod rpm2module as well, since we want to able to build the repo closure for arbitrary packages, carve out anything that's already available from an existing module, and then include the rest in the components list for the new modulemd. Right now it can't do that, since most packages aren't included in the modular server repo metadata.

Independently of all that though, the current translation of the builddeps flag into the selfhost flag inside the transplanted _depchase still isn't right, since it's running before we cull the package list of all the components that we're going to be getting from other modules, and that's why the solver is dragging in all those unrelated dependencies.

Instead, what fedmod needs to be doing is:

  1. Get the runtime repoclosure of the requested modules
  2. Generate module level runtime deps for everything that can be installed from an existing module
  3. Determine the extra deps needed to make whatever package set is still left self-hosting
  4. Generate module level build deps for any new requirements that can be installed from an existing module
  5. Generating suitable SRPM component entries for everything that couldn't be retrieved from another module

Implementation notes, since I'm about to wrap up for the day, and will need to pick up where I left off tomorrow:

  • fedmod._depchase.solve() currently combines getting the runtime deps with making the whole package set self-hosting
  • to switch to the above approach, the three steps (get runtime deps, get build deps for a given package set, get runtime deps for those build deps) are going to have to be broken out as distinct operations in order to interleave them properly with the module dependency generation steps

@ncoghlan that's why I said in the beginning that just using depchase will not satisfy your needs, you need way deeper integration with libsolv ;)

@ignatenkobrain Aye, but I don't think there's any way I could have learned the required parts of the libsolv API without the depchase code as an existing example to work from.

As far as I can see, the documentation is currently just a list of APIs and flag names, without much in the way of high level guidance on how to tie them together to solve dependency resolution problems.

@ignatenkobrain Checking I've understood the way Requires and BuildRequires are modelled at the libsolv level of things:

  • build requirements are stored as dependencies of the source RPMs
  • runtime requirements are stored as dependencies of the binary RPMs

So to ensure something can be installed, you do dependency resolution directly on the binary RPMs, while to ensure it can be built, you first get the corresponding list of source RPMs and then resolve the dependencies for those

libsolv level of things

This is how it is done in RPM actually

So to ensure something can be installed, you do dependency resolution directly on the binary RPMs, while to ensure it can be built, you first get the corresponding list of source RPMs and then resolve the dependencies for those

Not exactly, you just tell to solver to install SRPM and it will do the right thing.

@ignatenkobrain You understood what I meant, even if I didn't quite express it clearly :)

Closing based on https://pagure.io/modularity/fedmod/pull-request/14, although proper testing is going to have to wait until https://pagure.io/modularity/fedmod/issue/13 is resolved.

Metadata Update from @ncoghlan:
- Issue status updated to: Closed (was: Open)

Metadata