fedmod modularity tools

fedmod provides tools for working with Fedora's modulemd metadata format that aren't related to actually building them (for build commands, see fedpkg and mbs-build).

Currently, this consists of:

• fedmod <query-command>: simple repoquery-like commands providing operations like listing modules, resolving dependencies for packages, finding out where a certain package is, etc. See below for a full list of available query subcommands.
• fedmod lint: check modulemd documents for validity and flag errors
• fedmod fetch-metadata: download the package and module metadata needed to generate draft module definitions
• fedmod rpm2module: generates a draft modulemd file based on the given RPM name (multiple RPM names can be given, but the resulting draft module will lack any descriptive metadata in that case)
• fedmod rpm2flatpak: generates a draft modulemd file and container.yaml the given RPM name
• fedmod flatpak-report': generates a JSON report about the packages in the Flatpak runtime and not in the Flatpak runtime that would be required for turning the specified list of rpms into Flatpaks. This is used for maintaining the Fedora Flatpak runtime, but is likely not very useful otherwise.
• fedmod summarize-module: Prints a summary of available modules.

Using the repoquery-like commands

List all modules

Lists all modules available.

$ fedmod list-modules
module1
module2
...

List all modularized packages

Lists all packages that have been modularized. It can optionally list only duplicate packages, and show in which modules every package is.

$ fedmod list-rpms
pkg1
pkg2
...

$ fedmod list-rpms --duplicate-only
pkg2
...

$ fedmod list-rpms --list-modules
pkg1    (module1)
pkg2    (module2, module3)
...

Resolve package dependencies

Resolve package dependencies which is useful for creating new modules. User can also specify modular dependencies.

$ fedmod resolve-deps pkg
pkg2
pkg3
pkg4
...

$ fedmod resolve-deps -m host -m platform pkg
pkg3

Passing the --json option both requests additional information about dependencies (giving the SRPM name and runtime requirements for each RPM), and changes the output format to be a structured JSON list rather than just a simple list of names:

$ fedmod resolve-deps --json setup
[
    {
        "requires": {
            "system-release": [
                "fedora-release-27-1.noarch"
            ]
        },
        "rpm": "setup-2.10.10-1.fc27.noarch",
        "srpm": "setup-2.10.10-1.fc27.src"
    },
    {
        "requires": {
            "fedora-gpg-keys = 27-1": [
                "fedora-gpg-keys-27-1.noarch"
            ],
            "system-release(27)": [
                "fedora-release-27-1.noarch"
            ]
        },
        "rpm": "fedora-repos-27-1.noarch",
        "srpm": "fedora-repos-27-1.src"
    },
    {
        "requires": {},
        "rpm": "fedora-gpg-keys-27-1.noarch",
        "srpm": "fedora-repos-27-1.src"
    }
]

Find package in modules

Finds out whether a certain package has been modularized and in which module(s).

$fedmod where-is-package pkg
module1
module2

List packages of a module

Lists all packages in a given module. Can also list full NEVRAs.

$ fedmod module-packages module
pkg1
pkg2

$ fedmod module-packages --full-nevra module
pkg1-0:2.4.28-3.module_e7ab08d3.x86_64
pkg2-0:4.5.20-1.module_e7ab08d3.x86_64

Lookup RPMs from SRPM

Lists all the RPMs generated by the specified SRPM package name.

$ ./fedmod rpms-from-srpm spkg
pkg1
pkg2
pkg3
...

Lookup SRPM of RPM

Return the SRPM package name that generates the passed RPM

$ ./fedmod srpm-of-rpm pkg1
spkg

Modulemd creation

Before generating any draft modulemd files, first run the following command to fetch and locally cache the required metadata files:

$ fedmod fetch-metadata

fedmod rpm2module [RPM NAMES] will then create a modulemd file from the given package names and emit it on stdout. The YAML metadata can be written directly to a file instead by passing the `--output (or -o) option:

$ fedmod rpm2module -o graphite-web.yaml graphite-web

Only module level build dependencies are generated by default - there is no attempt to make the generated module definition self-hosting. If a self-hosting module is desired, then the --build-deps N option can be passed, where N is the number of levels of recursive build dependencies to attempt to include (this can quickly become unmanageable due to dependencies on build tools that themselves have complex build requirements, but are not yet part of a defined module)

The following metadata is currently used as input to the draft module generation process:

• Package dependency definitions are pulled from the regular Fedora release and updates repositories, with the metadata being downloaded for local use via the fedmod fetch-metadata command

• Descriptive metadata is taken from the system running fedmod. Due to this, fedmod currently only supports Fedora 26+. (This will be fixed to use the same repository metadata as is used for package dependency resolution)

Module dependencies currently err on the side of making the generated modules smaller by permitting generated modules to depend on packages that aren't listed as part of the public API of other modules. This reflects the fact that those transitive dependencies are typically the reason for the lower level modules appearing in the dependency set in the first place, as well as the fact that true dependency isolation will start being enforced once modules begin including opaque container images, such that only the client libraries are installed into shared environments.

Other limitations in generated modulemd files:

• components are only given a name and rationale, relying on the default settings for everything else

Contributing

If you want to contribute to fedmod, the recommended way is to fork the project on Pagure, develop your changes in a feature branch and submit a pull request against the upstream master branch. A member of the development team will then review the pull request and, when it's accepted, apply your changes.

Getting in touch

If you have any questions about contributing or issues, please contact a member of the development team on IRC: #fedora-modularity on irc.freenode.net

Rebase early, rebase often

As a general rule, contributions aren't merged into the upstream branch but applied on top ("rebased"). To make the reviewer's task easy and the review go as smooth as possible, please rebase your branch on top of the upstream branch it is based on (usually master) every now and then while you are developing your contribution and also when you submit the pull request and while it is reviewed. This is very appreciated as it ensures that your contribution applies cleanly always. Thanks!

Coding style

For our Python code base, regarding coding style we largely follow PEP 8 with a maximum line length of 100 characters for code (please wrap doc strings and comments at a "traditional" 72 characters regardless).

We make an exception regarding import order if a specific version or range of a module is required (e.g. for placing calls to gi.require_version() or pkg_resources.require() before the respective import statements). Please add a noqa comment so that flake8 doesn't flag them:

import gi
gi.require_version('Modulemd', '1.0')  # noqa: E402
from gi.repository import Modulemd

Preparing a pull request

Please ensure that all commits you submit in a pull request conform to the coding style outlined above and pass the test suite. If you fix such issues introduced by one PR commit in a later one, we prefer that they are merged by using git interactive rebase.

When reviewing, we run the flake8 tool (with the flake8-import-order plugin which is its own package) and the test suite on all submitted commits. We recommend doing likewise before submitting a PR to make the process as smooth as possible:

$ git rebase master --exec flake8
$ git rebase master --exec pytest

Either of these commands will stop the rebase process if it encounters problems to allow fixing it.

If you have problems with this, get in touch (see above) so we can help you with it.

Local development

Runtime dependencies

The preferred dependency management tool for development is pipenv:

$ pipenv --three --site-packages
$ PIP_IGNORE_INSTALLED=1 pipenv install --dev

The PIP_IGNORE_INSTALLED=1 setting means that everything available to pip will be installed into the virtual environment based on Pipfile.lock, and only components that aren't installable with pip will be used from the system Python installation.

Some dependencies aren't currently available from PyPI, and hence need to be installed system-wide:

$ sudo dnf install libmodulemd python3-gobject-base python3-solv

Additional development dependencies

pipenv itself isn't packaged for Fedora yet, so the recommended bootstrapping approach is to use the "pip script installer", pipsi:

$ sudo dnf install pipsi
$ pipsi install pew
$ pipsi install pipenv

This will create a pair of isolated virtual environments in your home directory specifically for pipenv and the tool it uses for virtual environment management, pew. These can later be updated to newer versions using pipsi:

$ pipsi upgrade pew
$ pipsi upgrade pipenv

(Note: the pipsi list command will list all packages installed via pipsi, and the commands they provide)

Running the development version

After setting up the pipenv environment, the development version can be run as follows

$ pipenv run fedmod fetch-metadata
$ pipenv run fedmod rpm2module graphite-web

Alternatively, start an interactive shell as described below for running the tests, and fedmod will refer to the development version.

Running the tests

After going through the environment setup steps above, start a shell that's correctly configured to run the tests with fedmod and all of its dependencies available:

$ pipenv shell

The metadata needed by the module generator tests can then be installed with fedmod itself:

$ fedmod fetch-metadata

The tests can then be run in the launched subshell with:

$ pytest tests

To test the package build process, tox is also supported:

$ tox -e py36

Reviewing project dependencies

To see the Python level dependencies graph:

$ pew toggleglobalsitepackages
$ pipenv graph
$ pew toggleglobalsitepackages

(If you don't turn off global site-packages access first, you'll get the dependency graph of all the installed system Python components as well)

Testing Fedora system package compatibility

While the default development environment is managed with pipenv for a more consistent cross-platform development experience, fedmod is intended to support installation as a system package in Fedora 26 and later.

A specific tox environment is provided to enable this testing:

$ tox -e system

The only component this installs into the environment is fedmod itself: all other dependencies must be available as Python 3 system packages.

Unlike the regular test environment, this environment also implicitly runs fedmod fetch-metadata in order to ensure that the metadata fetching operation also works correctly given only system packages as dependencies.

Publishing new releases

fedmod's RPMs are built with tito, but version tagging is handled with a helper script. To publish a new release, run:

$ ./tag-release.sh <X.Y.Z>
$ git push && git push --tags

PyPI releases

fedmod is also published to PyPI here: https://pypi.org/project/fedmod/

After releasing to COPR to ensure everything is properly tagged, a new PyPI release can be made by doing:

$ rm dist/*
$ python setup.py sdist bdist_wheel
$ twine upload dist/*

The dnf and solv dependencies unfortunately mean the PyPI release isn't particularly useful at this point (pipsi doesn't allow system level dependencies).