#4 Make metadata used for modulemd generation configurable
Closed: RESOLVED 5 years ago Opened 6 years ago by ncoghlan.

fedmod is currently hardcoded to use F27 metadata as the basis for determining which modules are available as potential dependencies.

This should be made configurable, ideally in a way that's compatible with koji mock-config.


btw I forgot to mention... There are many packages which have things like

%ifnarch s390x
BuildRequires: xxx
%endif

in spec files and depends on the state of moon SRPM in koji will either have this BuildRequires or not have it. In baseruntime-package-lists we have some list of packages which we definitely know that those are doing this. In that case you need to download srpm and rebuild it in @srpm-buildroot for each architecture.

To avoid having modules like mariadb pull in the world as build dependencies, I'm going to make the current version of fedmod "bootstrap module aware" and have it pull in the module definition from https://src.fedoraproject.org/modules/bootstrap/blob/master/f/bootstrap.yaml

However, as @karsten pointed out, that's going to need an off switch once we start trying to replace the bootstrap module with properly defined modules for the various development, testing, and build tools.

Having spent some more time working on this, I'm going to declare koji mock-config compatibility a non-goal, at least for now.

Instead, I'd like to start out with just three defined metadata sets:

  • Fedora Rawhide
  • Fedora Bikeshed
  • Fedora 27

Initially, those would be defined directly in the fedmod source code, but would eventually be moved out to a config file format so folks can supply their own not-necessarily-public configuration settings.

The UX would be a new --dataset option for all commands (including fedmod fetch-metadata). For fetch-metadata, this option would specify both what metadata to collect, and where to cache it locally. For other commands, it would just specify the cached metadata to use.

Metadata Update from @ncoghlan:
- Issue marked as depending on: #49

6 years ago

Added a dependency on https://pagure.io/modularity/fedmod/issue/49 as it makes sense to switch over to the intended long term argument parsing solution before figuring out the internal architecture for managing the new --dataset option.

Metadata Update from @ncoghlan:
- Issue assigned to ncoghlan

6 years ago

Metadata Update from @ncoghlan:
- Issue marked as depending on: #34

6 years ago

Also added a dependency on https://pagure.io/modularity/fedmod/issue/34 since it makes sense to clean that up before making further changes.

Metadata Update from @ncoghlan:
- Issue marked as depending on: #57

6 years ago

The reverse lookup queries have been tidied up, but while working on that I was reminded of https://pagure.io/modularity/fedmod/issue/57 (to move the reverse lookup table creation to the fetch-metadata command), which also makes sense to do prior to adding multiple-data-set support.

(It will also make it easier to inject mock module definitions for testing purposes)

@asamalik https://pagure.io/modularity/fedmod/pull-request/59 starts down the path I'd now like to take in resolving this issue by better separating the metadata downloads from reading the metadata when the CLI runs.

The basic idea is that we'll end up having the general notion of a "dataset name" which will be supported for all commands. The command line option processing will take care of loading the requested data set before any other commands run (with a special case for fetch-metadata to skip that step).

Initially, fetch-metadata will only support the three predefined options above, with an architecture dependent suffix: --dataset fedora-27-{arch}, --dataset fedora-bikeshed-{arch}, --dataset fedora-rawhide-{arch}, with the default data set probably being Rawhide (so you generate modules against the latest components by default, and have to explicitly opt in to using something else).

Eventually I'd like to have fetch-metadata accept arbitrary mock config files as inputs, but that will require refactoring some of the other assumptions in the metadata handling (at the moment the code fairly pervasively assumes that there will be exactly four repositories - a base repo, and updates repo, and their source repo counterparts)

Metadata Update from @ncoghlan:
- Assignee reset

6 years ago

The ability to select F28, F29, Rawhide datasets (and arch) was contributed as #86. This still hard-codes the locations, but making this configurable is planned. Closing.

Metadata Update from @nphilipp:
- Issue close_status updated to: RESOLVED
- Issue status updated to: Closed (was: Open)

5 years ago

Login to comment on this ticket.