#35 Support the dependency-report use case
Opened 2 years ago by ncoghlan. Modified 2 years ago

https://github.com/fedora-modularity/dependency-report is a project to get an overview of the dependency trees and content duplication in the entire Fedora module ecosystem.

Currently, the related scripts in https://github.com/fedora-modularity/dependency-report-scripts rely on the depchase utility in https://github.com/fedora-modularity/depchase to determine the dependencies, which in turn relies on the manually maintained metadata in https://github.com/modularity-modules/ to track what the current dependencies are. (The dependency resolution code in fedmod was originally born in depchase, hence the _fedmod._depchase internal module name)

The main functional difference between fedmod and the original depchase is that fedmod obtains its module metadata from the actual module definitions in the F27 modular server release, rather than relying on the modularity-modules repositories.

Now that the Fedora module build pipeline in dist-git is working reliably, we can expect the manually maintained metadata in modularity-modules to become increasingly out of date, so it will make sense to switch dependency-reports-scripts over to using fedmod as well.

The key call that gets made to depchase is here: https://github.com/fedora-modularity/dependency-report-scripts/blob/master/resolve.sh#L56

For fedmod, I'd suggest structuring this a bit differently, and offering a single dependencies subcommand that emits a JSON structure on standard out, defining:

  • modular runtime dependencies (and which RPMs are used from each module)
  • included runtime SRPMs (and which RPMs are used from each SRPM)
  • modular build dependencies (and which RPMs are used from each module)
  • included build-only SRPMs (and which RPMs are used from each SRPM)

To fully replace depchase, fedmod will also need to be made architecture-aware (currently it assumes x86_64 only).

Note: some of the technical details in the above aren't right (e.g. I think depchase works purely with RPMs, and it's dependency-report-scripts itself that handles the mapping from RPM and SRPM names back to modules), but the key point is that we need to remove the current dependency on the modularity-modules metadata, and migrating to fedmod will let us do that.

Pinging @asamalik @langdon @nphilipp @ignatenkobrain for your thoughts on this approach.

manually maintained metadata in modularity-modules

I don't even have definitions of modules there.

switch dependency-reports-scripts over to using fedmod as well

Neither I do use dependency-reports-scripts.

Generally, this thing is not easy to implement. But I would like to see PR which implements this.

@ignatenkobrain Yeah, I remembered as I was writing the second comment that the "module awareness" part was exactly the bit I'd needed to add to fedmod in https://pagure.io/modularity/fedmod/pull-request/11#request_diff

So the real structure in the dependency-reports-scripts is that they depend on depchase to deal with RPMs, and modularity-modules to deal with modules, which is somewhat similar to how fedmod is structured internally (where _fedmod._depchase remains blissfully unaware of the existence of modules).

Either way though, now that fedmod is kinda-sorta working, it makes sense to work out what we need at a CLI level to allow dependency-reports to switch over to using real metadata from Fedora's Module Build Server, instead of the README files in modularity-modules.

https://pagure.io/modularity/fedmod/issue/36 is a separate issue to cover handling dependency resolution for multiple architectures

The core functionality for this new subcommand already exists in https://pagure.io/modularity/fedmod/blob/master/f/src/_fedmod/module_generator.py#_31

The changes that would be needed to handle the new subcommand would be:

  1. Factor it out into a new internal API (independent of the module generator) that accepted a list of packages, and returned a dictionary describing the discovered dependencies
  2. Doing a better job internally of tracking why a particular dependency was added
  3. Fixing https://pagure.io/modularity/fedmod/issue/34 to handle multiple modules publishing the same RPM

Another possibility to consider here would be to add a subcommand that's the equivalent of https://github.com/fedora-modularity/dependency-report-scripts/blob/master/regenerate_everything.sh and emits a full directory tree with dependency reports for all the defined modules, as well as assorted overview details (this would be in addition to the dependencies subcommand, rather than instead of).

@otaylor's work on adding JSON output support to fedmod resolve-deps is potentially relevant here: https://pagure.io/modularity/fedmod/pull-request/47

The use case for the dependency-report scripts doesn't exist anymore, those are considered obsolete.
Nonetheless it would be useful to have this functionality later on when we have a huge # of modules depending on each other instead of the 'flat' F28 system where everything depends on platform only.
This is low priority atm, but should no be closed yet.

Metadata Update from @karsten:
- Issue close_status updated to: CANCELLED

2 years ago

Metadata Update from @karsten:
- Issue status updated to: Open (was: Closed)

2 years ago

Login to comment on this ticket.