#1592 Pungi fails to process module obsoletes for multiple streams
Closed: Fixed 6 months ago by lsedlar. Opened 9 months ago by ppisar.

There is a bug in pungi/phases/init.py:validate_module_defaults_or_obsoletes(). This piece of code attempt to detect multiple default documents for one module:

for module_name, defaults_or_obsoletes in seen.items():
        if len(defaults_or_obsoletes) > 1:
                "Module %s has multiple %s: %s"
                % (module_name, mmd_type, ", ".join(sorted(defaults_or_obsoletes)))

However, the same code processes obsolete documents and it reports an error if there are obsolete documents for more streams of the same module. That's is a legit use case. You may wont to have 3 postgresql streams in a repository:

postgresql:10 - obsoleted by 11
postgresql:11 - obsoleted by 12

and two of them obsoleted be the newer stream. Hence you need two obsolete documents for postgresql module.

If that happens, you can see failures like this with perl https://kojipkgs.fedoraproject.org/compose/updates/Fedora-Modular-34-updates-testing-20220303.0/logs/global/traceback.global.log

Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/pungi/scripts/pungi_koji.py", line 670, in cli_main
  File "/usr/lib/python3.9/site-packages/pungi/scripts/pungi_koji.py", line 347, in main
  File "/usr/lib/python3.9/site-packages/pungi/scripts/pungi_koji.py", line 496, in run_compose
  File "/usr/lib/python3.9/site-packages/pungi/phases/base.py", line 63, in start
  File "/usr/lib/python3.9/site-packages/pungi/phases/init.py", line 78, in run
  File "/usr/lib/python3.9/site-packages/pungi/phases/init.py", line 270, in validate_module_defaults_or_obsoletes
    % (module_name, mmd_type, ", ".join(sorted(defaults_or_obsoletes)))
TypeError: sequence item 0: expected str instance, Obsoletes found
Frame validate_module_defaults_or_obsoletes in /usr/lib/python3.9/site-packages/pungi/phases/init.py at line 270
     263             seen[module_name].add(defaults_or_obsoletes.get_default_stream())
     265     errors = []
     266     for module_name, defaults_or_obsoletes in seen.items():
     267         if len(defaults_or_obsoletes) > 1:
     268             errors.append(
     269                 "Module %s has multiple %s: %s"
-->  270                 % (module_name, mmd_type, ", ".join(sorted(defaults_or_obsoletes)))
     271             )
     273     if errors:
     274         raise RuntimeError(
     275             "There are duplicated module %s:\n%s" % (mmd_type, "\n".join(errors))
     276         )

The code needs to be fixed to accept multiple obsolete documents for a module. Don't forget that you can have even multiple obsoletes for the same module:stream differing only in a context.

I personally believe that this kind of check (detecting conflicting documents) pungi tries to achieve is wrong by design. libmodulemd in DNF handle the conflicts correctly. For obsoletes as well as for defaults.

Moreover there is a legit requirement to have more of the "conflicting" documents there. The can supersede each other by a different modification time. Either as a fix for a previous erroneous data, or an intentional change.

@mcurlej, @fivaldi , can you comment on the usefulness of the check? Can we remove it completely?

If you decide for removing the check, consider improving pungi/module_util.py:iter_module_defaults_or_obsoletes() called from there. iter_module_defaults_or_obsoletes() has a quadratic complexity. It could be linear.

And yet another use case for multiple defaults documents for the same module: module-d-packager-v3 format allows specifying a default profile for a stream. Once MBS is fixed https://pagure.io/fm-orchestrator/issue/1736, there will be regularly many documents like that coming from Koji to a repository.

Is this fixed with latest patch from @mkulik?

Metadata Update from @lsedlar:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

6 months ago

Metadata Update from @lsedlar:
- Issue set to the milestone: 4.3.5

6 months ago

Login to comment on this ticket.