On Fedora 31 mock chroot fails to initialize with the following DNF error:

No available modular metadata for modular package 'module-build-macros-0.1-1.module_f30+7100+0fc9a5b6.noarch', it cannot be installed on the system

On Fedora 29 mock chroot initializes fine.

Fedora 29 builders have:

• dnf-4.2.5-5.fc29.noarch
• libdnf-0.31.0-10.fc29.x86_64

Fedora 31 builders have:

• dnf-4.2.15-2.fc31.noarch
• libdnf-0.37.2-2.fc31.x86_64

fail-safe mechanism of DNF 4.2.7 is causing this. Koji build repos lack modular metadata, but DNF sees that module-build-macros is a modular package and fails to install it.

I've temporary disabled F29 staging builders to ease reproducing this issue in staging Koji:

Mon Nov 25 13:35:06 2019 host configuration for buildvm-05.stg.phx2.fedoraproject.org altered by mizdebsk
    enabled: True -> False
Mon Nov 25 13:35:06 2019 host configuration for buildvm-03.stg.phx2.fedoraproject.org altered by mizdebsk
    enabled: True -> False
Mon Nov 25 13:35:24 2019 host configuration for buildvm-aarch64-01.stg.arm.fedoraproject.org altered by mizdebsk
    enabled: True -> False
Mon Nov 25 13:35:24 2019 host configuration for buildvm-armv7-01.stg.arm.fedoraproject.org altered by mizdebsk
    enabled: True -> False

The issue is reproducible in staging Koji, but only after I upgraded DNF to latest 4.2.15 as follows:

buildvm-01.stg.phx2.fedoraproject.org | CHANGED => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python3"
    }, 
    "changed": true, 
    "msg": "", 
    "rc": 0, 
    "results": [
        "Installed: dnf", 
        "Installed: sqlite-3.30.0-1.fc31.x86_64", 
        "Installed: yum-4.2.15-2.fc31.noarch", 
        "Installed: python3-hawkey-0.37.2-2.fc31.x86_64", 
        "Installed: libdnf-0.37.2-2.fc31.x86_64", 
        "Installed: python3-libdnf-0.37.2-2.fc31.x86_64", 
        "Installed: librepo-1.11.0-1.fc31.x86_64", 
        "Installed: dnf-4.2.15-2.fc31.noarch", 
        "Installed: dnf-automatic-4.2.15-2.fc31.noarch", 
        "Installed: dnf-data-4.2.15-2.fc31.noarch", 
        "Installed: python3-librepo-1.11.0-1.fc31.x86_64", 
        "Installed: python3-dnf-4.2.15-2.fc31.noarch", 
        "Removed: yum-4.2.9-5.fc31.noarch", 
        "Removed: libdnf-0.35.3-6.fc31.x86_64", 
        "Removed: dnf-4.2.9-5.fc31.noarch", 
        "Removed: librepo-1.10.5-1.fc31.x86_64", 
        "Removed: dnf-automatic-4.2.9-5.fc31.noarch", 
        "Removed: dnf-data-4.2.9-5.fc31.noarch", 
        "Removed: python3-dnf-4.2.9-5.fc31.noarch", 
        "Removed: python3-hawkey-0.35.3-6.fc31.x86_64", 
        "Removed: python3-libdnf-0.35.3-6.fc31.x86_64", 
        "Removed: python3-librepo-1.10.5-1.fc31.x86_64"
    ]
}

The same component build succeeded with older DNF 4.2.9-5

@mizdebsk we had the same problem for MBS local builds and the following PR resolved it:
https://pagure.io/fm-orchestrator/pull-request/1494

Is there a way that MBS could configure DNF in Koji to use module_hotfixes=true when resolving dependencies?

So it is because the repo generated by koji does not contain modularity metadata?

Here is what I do when I test build locally. After building module-build-macros, and then after the completion of each rank of components I run this script: https://github.com/mbooth101/modularity-misc/blob/master/update_repo.py

It regenerates the repo and updates the modulemd metadata to include all the latest RPM artifacts -- I'm surprised this is not what is happening in MBS/koji; I guess there is no mechanism for MBS to customise the intermediate repositories that are generated by koji?

So it is because the repo generated by koji does not contain modularity metadata?

Correct. Koji repos don't contain modular metadata.

@mprahl, @mizdebsk , we should set:

KOJI_TAG_EXTRA_OPTS = {
  "mock.package_manager": "dnf"
  "repo_include_all": True
  "mock.new_chroot": 0
  "mock.yum.module_hotfixes": 1
}

in the roles/mbs/common/templates/config.py and run the playbook. That should fix it.

Also note that if we ever start using modular RPMs in non-modular buildroots (for example when f31-build includes modular RPMs, because thy would be default for f31), then f31-build also needs to set mock.yum.module_hotfixes: 1.

I already tested module_hotfixes=1 and it didn't work. I will try again.

@jkaluza thanks. I'll give that a try.

Edit: Nevermind. It looks like @mizdebsk is already on it.

I've added KOJI_TAG_EXTRA_OPTS to staging MBS config and I'm running testmodule build to check whether it fixes the issue.

Thanks @jkaluza , it looks like module_hotfixes=1 fixed the issue in staging. I'll apply the fix in production too.

I've applied the fix proposed by @jkaluza in production. @mbooth, Please confirm whether the issue is fixed.

Hey all, it seems to be behaving for me now, many thanks.

My random failures are now limited to s390x as is tradition :-)

Thanks for confirmation. I'm closing this issue as fixed then.
I have re-enabled Fedora 29 builders in staging as investigation is over.