Collecting links to relevant resources:

• @james's F27 Modular Server Docker images: docker run --rm -it jamesantill/boltron-27 bash
• Repo for that image: https://github.com/container-images/boltron-27
• Module listing script: https://github.com/container-images/boltron-27/blob/master/list-modules-py3.py
• DNF's modularity branch: https://github.com/rpm-software-management/dnf/tree/wip/modularity
• DNF module metadata loader: https://github.com/rpm-software-management/dnf/blob/wip/modularity/dnf/module/metadata_loader.py

Metadata for F27 Modular Bikeshed repo: https://dl.fedoraproject.org/pub/fedora/linux/modular/development/bikeshed/Server/x86_64/os/repodata/

(Note: the module metadata isn't currently given for SRPMs, which makes sense, since modules track how things are built and delivered, not where their source code comes from)

Critical snippet from the DNF code:

with gzip.open(repo_module_metadata_archive, "r") as modules_yaml_gz:
    modules_yaml = modules_yaml_gz.read()
module_metadata = modulemd.loads_all(modules_yaml)

ModuleMD's docs are currently rather thin, but checking the source code shows the result of the above will be a list of module metadata files: https://pagure.io/modulemd/blob/master/f/modulemd/init.py#_73

https://github.com/rpm-software-management/dnf/blob/wip/modularity/dnf/module/repo_module_dict.py#L193 (RepoModuleDict.install()) is then the starting point for where DNF figures out whether or not it can satisfy a module installation request.

For fedmod, we can probably do something simpler: use the repo level module metadata to build a reverse mapping from RPMs listed in module's public API definitions to the corresponding modules.

Then we would run through each dependency found via libsolv and:

1. Check if it was in the mapping, if so, add a dependency declaration on the relevant module
2. If it isn't in the mapping, add it to the module currently being defined as an internal implementation detail that isn't part of the public API

Hmm, thinking about it further, that simplistic approach won't work, since we need to cut out the entire dependency tree from below that component as well, and the initial dependency resolution doesn't currently keep track of the transitive dependency relationships that explain why a particular package ended up in the dependency set.

It may be that we actually need something like networkx in order to solve this properly, since this is a form of graph partitioning problem: https://en.wikipedia.org/wiki/Graph_partition

One simplifying point in our favour, is that we're not having to calculate the partitions: instead, we just want to work out what's left as reachable dependencies of the current module after we cut the dependency links hidden behind the public APIs of other modules.

We can probably ignore this problem for now though, as our main initial interest is in modules that depend directly on the platform module (and nothing else). Once that's working, then we can consider the harder case of depending on arbitrary other modules without mistakenly added additional copies of their private dependencies.

Looking in more detail at the module metadata in https://dl.fedoraproject.org/pub/fedora/linux/modular/development/bikeshed/Server/x86_64/os/repodata/, I think the naive approach of just collecting all the RPMs from the "artifacts.rpms" lists and using those to filter the components list for the generated modulemd will definitely be adequate for now.

It isn't entirely valid in a strict sense, since it may result in a latent defect where the generated module may be affected by updates in its dependencies that drop non-public RPMs from their output, but the long terms trend there is going to be towards relative thin clients with few dependencies talking to containerised backends that hide those dependencies in the first place.

However, the naive approach should be simple to implement, and we may even be able to include comments for each generated module dependency saying which public RPMs and which non-public RPMs in that module are part of this module's repoclosure..

First iteration has landed with all sorts of hardcoded assumptions: https://pagure.io/modularity/fedmod/pull-request/11

The one addition I want to make now is to handle the python 2 vs python 3 case, such that projects with python 3 dependencies necessarily pull in the right version of the stack. This means making the reverse lookup table generation aware of module output filters, as well as giving it the ability to handle ambiguous component resolutions.

After writing the above, I realised that the initial code already does the right thing here, since MBS generates the artifact list after applying the output filters.

I've also clarified the code to make it clear that only RPM names are used to calculate module dependencies, not SRPM names.

Closing this, as any further adjustments will be either by purely internal cleanups, or else fixes based on particular candidate modules giving nonsense answers.