I converted perl-bootstrap:5.32 and perl:5.32 YAML files into modulemd-packager-v3 format. perl:5.32 build-requires perl-bootstrap:5.32. Both streams have 4 context. Each for a different platform (f35, f34, f33, eln). perl-bootstrap:5.32 was built successfully https://koji.fedoraproject.org/koji/packageinfo?packageID=24572:
perl-bootstrap-5.32-3520210617063415.f27b74a8 perl-bootstrap-5.32-3420210617063415.058368ca perl-bootstrap-5.32-3320210617063415.601d93de perl-bootstrap-5.32-20210617063415.8b234a03
Then I commenced building perl:5.32 with this result:
$ fedpkg module-build-watch $(seq 12239 12242) [Build #12239] perl-5.32-3420210616081115-866eb18a is in "failed" state. Koji tag: module-perl-5.32-3420210616081115-866eb18a Link: https://mbs.fedoraproject.org/module-build-service/2/module-builds/12239 Components: 110 done, 6 failed Reason: Component(s) perl-Module-CoreList, perl-experimental, perl-generators, perl-perlfaq, perl-version, perl failed to build. [Build #12240] perl-5.32-3320210616081115-94c872f1 is in "failed" state. Koji tag: module-perl-5.32-3320210616081115-94c872f1 Link: https://mbs.fedoraproject.org/module-build-service/2/module-builds/12240 Components: 110 done, 6 failed Reason: Component(s) perl-perlfaq, perl-version, perl-generators, perl-experimental, perl-Module-CoreList, perl failed to build. [Build #12241] perl-5.32-3520210616081115-ab7e7bfe is in "ready" state. Koji tag: module-perl-5.32-3520210616081115-ab7e7bfe Link: https://mbs.fedoraproject.org/module-build-service/2/module-builds/12241 Components: 116 done, 0 failed [Build #12242] perl-5.32-20210616081115-d5eb079e is in "ready" state. Koji tag: module-perl-5.32-20210616081115-d5eb079e Link: https://mbs.fedoraproject.org/module-build-service/2/module-builds/12242 Components: 116 done, 0 failed
Contexts for f35 and eln succeceed. Contexts for f34 and f33 failed. The reason for the failure is that MBS tagged perl-bootstrap:5.32 for platform:f35 into their build root. This is the bug I report here.
An example: A build for f34 context https://mbs.fedoraproject.org/module-build-service/2/module-builds/12239:
"buildrequires": { "perl-bootstrap": { "context": "f27b74a8", "filtered_rpms": [], "koji_tag": "module-perl-bootstrap-5.32-3520210617063415-f27b74a8", "ref": "affaf88e210fdb668d517242b57ed7b1d7bc8cc1", "stream": "5.32", "version": "3520210617063415" }, "platform": { "context": "00000000", "filtered_rpms": [], "koji_tag": "module-f34-build", "ref": "f34", "stream": "f34", "stream_collision_modules": "", "ursine_rpms": "", "version": "1" } },
As a result, Koji attempted to install f35 packages when building perl package for f34 https://koji.fedoraproject.org/koji/taskinfo?taskID=70301320:
DEBUG util.py:542: Executing command: ['/usr/bin/dnf', 'builddep', '--installroot', '/var/lib/mock/module-perl-5.32-3420210616081115-866eb18a-build-27867011-3734355/root/', '--setopt=install_weak_deps=0', '--disableplugin=local', '--disableplugin=spacewalk', '/var/lib/mock/module-perl-5.32-3420210616081115-866eb18a-build-27867011-3734355/root//builddir/build/SRPMS/perl-5.32.1-470.module_f34+12239+ea36e837.src.rpm', '--setopt=tsflags=nocontexts'] with env {'TERM': 'vt100', 'SHELL': '/bin/bash', 'HOME': '/var/lib/mock/module-perl-5.32-3420210616081115-866eb18a-build-27867011-3734355/root/installation-homedir', 'HOSTNAME': 'mock', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin', 'PROMPT_COMMAND': 'printf "\\033]0;<mock-chroot>\\007"', 'PS1': '<mock-chroot> \\s-\\v\\$ ', 'LANG': 'C.UTF-8', 'LC_MESSAGES': 'C.UTF-8', 'LD_PRELOAD': '/var/tmp/tmp.mock.h6t_cso5/$LIB/nosync.so'} and shell False [...] DEBUG util.py:446: Package tar-2:1.34-1.fc34.x86_64 is already installed. DEBUG util.py:444: Error: DEBUG util.py:444: Problem 1: conflicting requests DEBUG util.py:444: - nothing provides libc.so.6(GLIBC_2.34)(64bit) needed by perl-interpreter-4:5.32.1-470.module_f35+12238+8b40b997.x86_64 DEBUG util.py:444: Problem 2: package perl-generators-1.13-1.module_f35+12238+8b40b997.noarch requires perl(:MODULE_COMPAT_5.32.1), but none of the providers can be installed DEBUG util.py:444: - conflicting requests DEBUG util.py:444: - nothing provides libc.so.6(GLIBC_2.34)(64bit) needed by perl-libs-4:5.32.1-470.module_f35+12238+8b40b997.x86_64 DEBUG util.py:446: (try to add '--skip-broken' to skip uninstallable packages) DEBUG util.py:598: Child return code was: 1
I thought that the platforms were crossed also in a build for eln, but they weren't. A perl build for eln used perl-bootstrap for eln. I cannot see any pattern in the bug.
By the way, yesterday I build plenty of multi-context modules (e.g. perl-DBI:1.643) and I did not observed these issues.
The only difference I can found is that I converted and built them in a reverse order of dependencies (first perl-DBD-Pg, then perl-DBI). Today I built them in the natural order (first perl-bootstrap, then perl). All of the modules were converted at the same time. I.e. yesterday I built modulemd-packager-v3 modules which build-required stream-expanded modulemd-v2 modules. Today I build modulemd-packager-v3 modules which build-required modulemd-packager-v3 modules.
modulemd-v2 tasks produce modulemd-v2 builds with platform in requires https://kojipkgs.fedoraproject.org//packages/perl-bootstrap/5.32/3420210211113600.058368ca/files/module/modulemd.x86_64.txt, but modulemd-packager-v3 modules tasks produce modulemd-v2 builds without platform in requies and a static_context flag https://kojipkgs.fedoraproject.org//packages/perl-bootstrap/5.32/3420210617063415.058368ca/files/module/modulemd.x86_64.txt.
That could explain why building against static_context builds has difficulties to select the right platform.
I tried to work around this issue by run-requiring a specific platform in perl-bootstrap [https://mbs.fedoraproject.org/module-build-service/2/module-builds/12243] as well as in perl [https://mbs.fedoraproject.org/module-build-service/2/module-builds/12247] module and it won't help. It even did not appear in requires section of the output modulemd-v2 document stored in Koji.
Because I needed building the modules, I had to revert the conversion to modulemd-packager-v3 format. If you need the yaml files, you can found them in a dist-git history. Commit IDs are recorded in the module build tasks.
So far I've had no luck finding where in the code this is going wrong.
I'm also having trouble replicating the issue. For example, if I trigger a scratch build from the same ref as your original failure, the result seems to have the correct buildrequires. E.g
https://mbs.fedoraproject.org/module-build-service/2/module-builds/12647
My guess is that something about the exact state of the mbs db at the time was tickling a bug in dependency resolution.
If we can find a replicator, I should be able to debug it in detail now. Do you have an example where this replicates currently?
You cannot reproduce it because the bug does not depend on what you build, but on what the build build-requires. If you build perl, then you need perl-bootstrap which does not run-require any platform.
As I wrote I reverted the changes in perl-bootstrap:5.32 and perl:5.32. I also rebuilt them. As a result, the latest perl-bootstrap:5.32 run-requires platform now. Hence you cannot reproduce it with building perl:5.32.
You need to build the historical perl-bootstrap:5.32 in your staging environment and then build perl:5.32. That should reproduce it.
Or: I have not yet reverted e.g. perl-DBI. You can try building anything what build-requires it. E.g. perl-DBD-Pg. That should also reproduce it. I'm not here next week, so I won't touch the modules in Fedora. But then I will continue in the reverts because Fedora 35 was branched and we need to build all modules for Fedora 36.
Ok, I think you are right that the lack of a platform requires in the dependency is the root issue.
The MBS code explicitly removes this requires in the V3 case, in process_module_context_configuration(). This was part of the original V3 work from PR #1667. When I asked @mcurlej about it, he wrote:
process_module_context_configuration()
wrt removing platform from requires, this was not implemented in libmodulemd. libmodulemd right now has no active development as it is missing a developer. An issue was created and was closed, with the recommendation to do it in MBS. This is required by the DNF team as the virtual modules are not RPMs that can be installed on runtime, so they have no place in requires.
I believe the issue he refers to is this one -- https://github.com/fedora-modularity/libmodulemd/issues/547 I don't see any info in there about why DNF requires this, but I agree with sgallagher's comment that it seems like keeping the requires is the backwards compatible thing to do.
Commit 85e5b9c fixes this issue
Thank you for fixing it. I've just successfully built perl-bootstrap-5.32-3320210903064505.601d93de and perl-5.32-3320210903081051.94c872f1, both in from modulemd-packager-v3 input format with correctly produced a static_context modulemd-v2 document with a run-time dependency on a platform. E.g. perl-Digest component was not reused in perl:5.32 module and it got correct F33 perl-boostrap packages in its buildroot https://kojipkgs.fedoraproject.org//work/tasks/4437/75044437/root.log.
I will continue with migrating other modules to modulem-packager-v3. I believe this issue is properly fixed now.
Login to comment on this ticket.