#1213 Fix -debuginfo/-debugsource packages handling in KojiContentGenerator.
Closed 4 years ago by jkaluza. Opened 4 years ago by jkaluza.
jkaluza/fm-orchestrator debuginfo-fix  into  master

@@ -254,15 +254,25 @@ 

                  src_rpms[rpm["id"]] = rpm

              else:

                  binary_rpms[rpm["id"]] = rpm

+ 

          # Prepare the arguments for Koji multicall.

          # We will call session.getRPMHeaders(...) for each SRC RPM to get exclusivearch,

          # excludearch and license headers.

          multicall_kwargs = [{"rpmID": rpm_id,

                               "headers": ["exclusivearch", "excludearch", "license"]}

                              for rpm_id in src_rpms.keys()]

-         # For each binary RPM, we only care about the "license" header.

-         multicall_kwargs += [{"rpmID": rpm_id, "headers": ["license"]}

-                              for rpm_id in binary_rpms.keys()]

+ 

+         # For each binary RPM, we care about the "license" header.

+         # For -debuginfo and -debugsource RPMs, we care also about the "requires", because we

+         # need to be able to add -debuginfo/-debugsource packages to non-devel module if they

+         # are required by other -debuginfo/-debugsource packages.

+         for rpm_id, rpm in binary_rpms.items():

+             headers = ["license"]

+             if rpm["name"].endswith(("-debuginfo", "-debugsource")):

+                 headers.append("requires")

+             multicall_kwargs += [{"rpmID": rpm_id, "headers": headers}]

+ 

+         # Get the RPM headers.

          rpms_headers = koji_retrying_multicall_map(

              session, session.getRPMHeaders, list_of_kwargs=multicall_kwargs)

  
@@ -295,6 +305,7 @@ 

              rpm['srpm_nevra'] = build_id_to_srpm_nevra[rpm["build_id"]]

              rpm['exclusivearch'] = build['exclusivearch']

              rpm['excludearch'] = build['excludearch']

+             rpm['requires'] = headers['requires'] if 'requires' in headers else []

  

          return rpms

  
@@ -585,20 +596,58 @@ 

  

          # Grouping of "main" RPM with the corresponding debuginfo and debugsource RPMs

          grouped_rpms = defaultdict(list)

+         # Source RPM name to nevra

          source_rpms = {}

+         # Source RPM name to included -debugsource NEVRA.

+         included_debugsource_nevras = {}

+         # Name of source RPM to include in -devel module to nevra.

          non_devel_source_rpms = {}

+         # Debuginfo/Debugsource RPM name to RPM dict.

+         debuginfo_debugsource_rpms = {}

          for nevra, rpm in self.rpms_dict.items():

+             if rpm["name"].endswith(("-debuginfo", "-debugsource")):

+                 debuginfo_debugsource_rpms[rpm["name"]] = rpm

              name = strip_suffixes(rpm["name"], group_suffixes)

              grouped_rpms[name].append((nevra, rpm))

              if rpm["arch"] == "src":

                  source_rpms[name] = nevra

  

+         # Add the required -debuginfo and -debugsource packages to `grouped_rpms` too.

+         # For example:

+         #  - The `grouped_rpms["python3-pymongo"]` contains "python3-pymongo" and

+         #    "python3-pymongo-debuginfo".

+         #  - The "python3-pymongo-debuginfo" requires "python-pymongo-debuginfo".

+         #  - The goal of this code is therefore to add "python-pymongo-debuginfo"

+         #    to grouped_rpms["python3-pymongo"], so when python3-pymongo is added

+         #    to RPM list, we add also both python3-pymongo-debuginfo and

+         #    python-pymongo-debuginfo.

+         # Iterate over each -debuginfo/-debugsource RPM and find out if it requires

+         # some other -debuginfo/-debugsource. If it does, add it to right `grouped_rpms`

+         # list.

+         for dbg_name, dbg_rpm in debuginfo_debugsource_rpms.items():

+             # The `dbg_name` is for example "python3-pymongo-debuginfo".

+             if not dbg_rpm["requires"]:

+                 continue

+ 

+             name = strip_suffixes(dbg_name, group_suffixes)

+             for required_dep in dbg_rpm["requires"]:

+                 # The `required_dep` is for example "python-pymongo-debuginfo(aarch-64)".

+                 # We are interested only in a name without arch, so remove the parts after

+                 # bracket.

+                 required_name = required_dep.split("(")[0]

+                 if required_name not in debuginfo_debugsource_rpms:

+                     continue

+                 # Get the required RPM data and add it to grouped_rpms belonging to original

+                 # debug RPM.

+                 required_rpm = debuginfo_debugsource_rpms[required_name]

+                 required_nevra = kobo.rpmlib.make_nvra(required_rpm, force_epoch=True)

+                 grouped_rpms[name].append((required_nevra, required_rpm))

+ 

          # Check each RPM in `self.rpms_dict` to find out if it can be included in mmd

          # for this architecture.

          for nevra, rpm in self.rpms_dict.items():

-             # Filter out debug and source RPMs, these will later be included if

-             # the "main" RPM is included

-             if rpm["name"].endswith(group_suffixes) or rpm["arch"] == "src":

+             # Filter out debug RPMs, these will later be included if the "main" RPM is included.

+             if rpm["name"].endswith("-debuginfo") or rpm["arch"] == "src":

                  continue

  

              # Filter out RPMs which will never end up in final modulemd:
@@ -618,6 +667,25 @@ 

  

              should_include = self._should_include_rpm(rpm, mmd, arch, multilib_arches)

  

+             # It's complicated with -debugsource RPMs... There are two cases:

+             # a) The -debugsource package is generated for .src.rpm.

+             #    In this case, we need to include this -debugsource package in the final RPM

+             #    list *only if* the .src.rpm is included in the final RPM list.

+             #    But we don't know this yet, so mark this by adding -debugsource into the

+             #    `included_debugsource_nevras` and add it to `rpm_artifacts` later together

+             #    with the source RPM.

+             # b) The -debugsource package is generated for real non-src RPM.

+             #    In this case, we need to include this -debugsource package in the final RPM

+             #    list *only if* the main non-src RPM is in the list. Luckily for us, the

+             #    -debugsource package is in the `grouped_rpms`, so no extra handling

+             #    is needed here.

+             if rpm["name"].endswith("-debugsource"):

+                 if should_include:

+                     name = strip_suffixes(rpm["name"], ["-debugsource"])

+                     if rpm["srpm_name"] == name:

+                         included_debugsource_nevras[name] = nevra

+                 continue

+ 

              # A source RPM should be included in a -devel module only if all the

              # RPMs built from this source RPM are included in a -devel module.

              # The list of source RPMs in non-devel module is tracked in
@@ -647,12 +715,20 @@ 

                  if license:

                      rpm_licenses.add(license)

  

+         # Get the list of source RPM nevras to include in a final RPM list.

          if self.devel:

-             for source_nevra in set(source_rpms.values()) - set(non_devel_source_rpms.values()):

-                 rpm_artifacts.add(source_nevra)

+             source_nevras = set(source_rpms.values()) - set(non_devel_source_rpms.values())

          else:

-             for source_nevra in non_devel_source_rpms.values():

-                 rpm_artifacts.add(source_nevra)

+             source_nevras = non_devel_source_rpms.values()

+ 

+         # Add the source RPMs.

+         for source_nevra in source_nevras:

+             rpm_artifacts.add(source_nevra)

+ 

+             # If there is also -debugsource to be included, add it.

+             source_rpm = self.rpms_dict[source_nevra]

+             if source_rpm["name"] in included_debugsource_nevras:

+                 rpm_artifacts.add(included_debugsource_nevras[source_rpm["name"]])

  

          mmd.set_content_licenses(rpm_licenses)

          mmd.set_rpm_artifacts(rpm_artifacts)

file modified
+173 -1
@@ -502,7 +502,7 @@ 

  

      def _add_test_rpm(self, nevra, srpm_nevra, multilib=None,

                        koji_srpm_nevra=None, excludearch=None, exclusivearch=None,

-                       license=None):

+                       license=None, requires=None):

          """

          Helper method to add test RPM to ModuleBuild used by KojiContentGenerator

          and also to Koji tag used to generate the Content Generator build.
@@ -518,6 +518,7 @@ 

          :param list excludearch: List of architectures this package is excluded from.

          :param list exclusivearch: List of architectures this package is exclusive for.

          :param str license: License of this RPM.

+         :param list requires: List of requires.

          """

          srpm_name = kobo.rpmlib.parse_nvra(srpm_nevra)["name"]

  
@@ -532,6 +533,7 @@ 

          parsed_nevra["excludearch"] = excludearch or []

          parsed_nevra["exclusivearch"] = exclusivearch or []

          parsed_nevra["license"] = license or ""

+         parsed_nevra["requires"] = requires or []

          self.cg.rpms.append(parsed_nevra)

          self.cg.rpms_dict[nevra] = parsed_nevra

  
@@ -888,3 +890,173 @@ 

              side_effect=koji.GenericError('Build already exists asdv'))

          self.cg.koji_import()

          tagger.assert_called()

+ 

+     def test_fill_in_rpms_list_debuginfo_deps(self):

+         """

+         Tests that -debuginfo RPM required by other -debuginfo RPM is included in a RPM list.

+ 

+         The python3-pymongo has matching python3-pymongo-debuginfo RPM which requires

+         python-pymongo-debuginfo RPM. All of them should appear in RPM list

+         """

+         self._add_test_rpm(

+             "python-pymongo-debuginfo-3.6.1-9.module+f29.1.0+2993+d789589b.x86_64",

+             "python-pymongo-3.6.1-9.module+f29.1.0+2993+d789589b.src",

+             requires=["rpmlib(CompressedFileNames)", "rpmlib(FileDigests)",

+                       "rpmlib(PayloadFilesHavePrefix)", "rpmlib(PayloadIsXz)"])

+ 

+         self._add_test_rpm(

+             "python3-pymongo-debuginfo-3.6.1-9.module+f29.1.0+2993+d789589b.x86_64",

+             "python-pymongo-3.6.1-9.module+f29.1.0+2993+d789589b.src",

+             requires=["python-pymongo-debuginfo(x86_64)", "rpmlib(CompressedFileNames)",

+                       "rpmlib(FileDigests)", "rpmlib(PayloadFilesHavePrefix)",

+                       "rpmlib(PayloadIsXz)"])

+ 

+         self._add_test_rpm(

+             "python3-pymongo-3.6.1-9.module+f29.1.0+2993+d789589b.x86_64",

+             "python-pymongo-3.6.1-9.module+f29.1.0+2993+d789589b.src")

+ 

+         self._add_test_rpm(

+             "python-pymongo-3.6.1-9.module+f29.1.0+2993+d789589b.src",

+             "python-pymongo-3.6.1-9.module+f29.1.0+2993+d789589b.src")

+ 

+         mmd = self.cg.module.mmd()

+         mmd = self.cg._fill_in_rpms_list(mmd, "x86_64")

+ 

+         assert set(mmd.get_rpm_artifacts().get()) == set([

+             'python-pymongo-debuginfo-0:3.6.1-9.module+f29.1.0+2993+d789589b.x86_64',

+             'python3-pymongo-debuginfo-3.6.1-9.module+f29.1.0+2993+d789589b.x86_64',

+             'python-pymongo-3.6.1-9.module+f29.1.0+2993+d789589b.src',

+             'python3-pymongo-3.6.1-9.module+f29.1.0+2993+d789589b.x86_64'])

+ 

+     def test_fill_in_rpms_list_debuginfo_deps_psycopg2(self):

+         """

+         Tests that -debuginfo RPM required by other -debuginfo RPM is included in a RPM list

+         with the psycopg2 RPM test-case, because psycopg2 RPMs are built in kind of special

+         way...

+         """

+         self._add_test_rpm(

+             "python2-psycopg2-debuginfo-2.7.5-7.module+f29.0.0+2961+596d0223.x86_64",

+             "python-psycopg2-2.7.5-7.module+f29.0.0+2961+596d0223.src",

+             requires=["python-psycopg2-debuginfo(x86-64)"])

+ 

+         self._add_test_rpm(

+             "python2-psycopg2-debug-debuginfo-2.7.5-7.module+f29.0.0+2961+596d0223.x86_64",

+             "python-psycopg2-2.7.5-7.module+f29.0.0+2961+596d0223.src",

+             requires=["python-psycopg2-debuginfo(x86-64)"])

+ 

+         self._add_test_rpm(

+             "python-psycopg2-debugsource-2.7.5-7.module+f29.0.0+2961+596d0223.x86_64",

+             "python-psycopg2-2.7.5-7.module+f29.0.0+2961+596d0223.src")

+ 

+         self._add_test_rpm(

+             "python-psycopg2-debuginfo-2.7.5-7.module+f29.0.0+2961+596d0223.x86_64",

+             "python-psycopg2-2.7.5-7.module+f29.0.0+2961+596d0223.src")

+ 

+         self._add_test_rpm(

+             "python2-psycopg2-tests-2.7.5-7.module+f29.0.0+2961+596d0223.x86_64",

+             "python-psycopg2-2.7.5-7.module+f29.0.0+2961+596d0223.src")

+ 

+         self._add_test_rpm(

+             "python2-psycopg2-debug-2.7.5-7.module+f29.0.0+2961+596d0223.x86_64",

+             "python-psycopg2-2.7.5-7.module+f29.0.0+2961+596d0223.src")

+ 

+         self._add_test_rpm(

+             "python2-psycopg2-2.7.5-7.module+f29.0.0+2961+596d0223.x86_64",

+             "python-psycopg2-2.7.5-7.module+f29.0.0+2961+596d0223.src")

+ 

+         self._add_test_rpm(

+             "python-psycopg2-2.7.5-7.module+f29.0.0+2961+596d0223.src",

+             "python-psycopg2-2.7.5-7.module+f29.0.0+2961+596d0223.src")

+ 

+         mmd = self.cg.module.mmd()

+         mmd = self.cg._fill_in_rpms_list(mmd, "x86_64")

+ 

+         assert set(mmd.get_rpm_artifacts().get()) == set([

+             "python2-psycopg2-debuginfo-2.7.5-7.module+f29.0.0+2961+596d0223.x86_64",

+             "python2-psycopg2-debug-debuginfo-2.7.5-7.module+f29.0.0+2961+596d0223.x86_64",

+             "python-psycopg2-debugsource-2.7.5-7.module+f29.0.0+2961+596d0223.x86_64",

+             "python-psycopg2-debuginfo-0:2.7.5-7.module+f29.0.0+2961+596d0223.x86_64",

+             "python2-psycopg2-tests-2.7.5-7.module+f29.0.0+2961+596d0223.x86_64",

+             "python2-psycopg2-debug-2.7.5-7.module+f29.0.0+2961+596d0223.x86_64",

+             "python2-psycopg2-2.7.5-7.module+f29.0.0+2961+596d0223.x86_64",

+             "python-psycopg2-2.7.5-7.module+f29.0.0+2961+596d0223.src"])

+ 

+     def test_fill_in_rpms_list_debugsource_for_non_srpm(self):

+         self._add_test_rpm(

+             "python2-psycopg2-debugsource-2.7.5-7.module+f29.0.0+2961+596d0223.x86_64",

+             "python-psycopg2-2.7.5-7.module+f29.0.0+2961+596d0223.src")

+ 

+         self._add_test_rpm(

+             "python2-psycopg2-2.7.5-7.module+f29.0.0+2961+596d0223.x86_64",

+             "python-psycopg2-2.7.5-7.module+f29.0.0+2961+596d0223.src")

+ 

+         self._add_test_rpm(

+             "python-psycopg2-2.7.5-7.module+f29.0.0+2961+596d0223.src",

+             "python-psycopg2-2.7.5-7.module+f29.0.0+2961+596d0223.src")

+ 

+         mmd = self.cg.module.mmd()

+         mmd = self.cg._fill_in_rpms_list(mmd, "x86_64")

+ 

+         assert set(mmd.get_rpm_artifacts().get()) == set([

+             "python2-psycopg2-debugsource-2.7.5-7.module+f29.0.0+2961+596d0223.x86_64",

+             "python2-psycopg2-2.7.5-7.module+f29.0.0+2961+596d0223.x86_64",

+             "python-psycopg2-2.7.5-7.module+f29.0.0+2961+596d0223.src"])

+ 

+     def test_fill_in_rpms_list_debuginfo_deps_glibc(self):

+         self._add_test_rpm(

+             "glibc-common-2.29.9000-16.fc31.x86_64",

+             "glibc-2.29.9000-16.fc31.src")

+ 

+         self._add_test_rpm(

+             "glibc-2.29.9000-16.fc31.x86_64",

+             "glibc-2.29.9000-16.fc31.src")

+ 

+         self._add_test_rpm(

+             "glibc-debuginfo-common-2.29.9000-16.fc31.x86_64",

+             "glibc-2.29.9000-16.fc31.src")

+ 

+         self._add_test_rpm(

+             "glibc-debuginfo-2.29.9000-16.fc31.x86_64",

+             "glibc-2.29.9000-16.fc31.src",

+             requires=["glibc-debuginfo-common"])

+ 

+         self._add_test_rpm(

+             "glibc-2.29.9000-16.fc31.src",

+             "glibc-2.29.9000-16.fc31.src")

+ 

+         mmd = self.cg.module.mmd()

+         mmd = self.cg._fill_in_rpms_list(mmd, "x86_64")

+ 

+         assert set(mmd.get_rpm_artifacts().get()) == set([

+             "glibc-common-2.29.9000-16.fc31.x86_64",

+             "glibc-2.29.9000-16.fc31.src",

+             "glibc-2.29.9000-16.fc31.x86_64",

+             "glibc-debuginfo-common-2.29.9000-16.fc31.x86_64",

+             "glibc-debuginfo-2.29.9000-16.fc31.x86_64"])

+ 

+     def test_fill_in_rpms_list_debuginfo_deps_kernel(self):

+         self._add_test_rpm(

+             "kernel-debuginfo-common-aarch64-5.0.9-301.fc30.aarch64",

+             "kernel-5.0.9-301.fc30.src")

+ 

+         self._add_test_rpm(

+             "kernel-debuginfo-5.0.9-301.fc30.aarch64",

+             "kernel-5.0.9-301.fc30.src",

+             requires=["kernel-debuginfo-common-aarch64"])

+ 

+         self._add_test_rpm(

+             "kernel-5.0.9-301.fc30.aarch64",

+             "kernel-5.0.9-301.fc30.src")

+ 

+         self._add_test_rpm(

+             "kernel-5.0.9-301.fc30.src",

+             "kernel-5.0.9-301.fc30.src")

+ 

+         mmd = self.cg.module.mmd()

+         mmd = self.cg._fill_in_rpms_list(mmd, "aarch64")

+ 

+         assert set(mmd.get_rpm_artifacts().get()) == set([

+             "kernel-debuginfo-common-aarch64-5.0.9-301.fc30.aarch64",

+             "kernel-5.0.9-301.fc30.src",

+             "kernel-debuginfo-5.0.9-301.fc30.aarch64",

+             "kernel-5.0.9-301.fc30.aarch64"])

Our current code has following issues with -debuginfo/-debugsource handling:

  • The foo-debuginfo can require bar-debuginfo and we need to add
    bar-debuginfo to the final MMD file even if bar is not added there.
    This is typical situation for python-foo-debuginfo which is required
    by python2-foo-debuginfo and python3-foo-debuginfo.
  • The python-foo-debugsource needs to be added to final MMD file even
    if there is no python-foo binary RPM, but only python-foo SRPM.

In this commit, this is fixed following way:

  • For -debuginfo and -debugsource RPMs, we ask Koji for their list of 'requires'.
  • When constructing the list of grouped_rpms (which normally includes just the
    foo, foo-debuginfo and foo-debugsource packages), we also add the packages
    required by the -debuginfo and -debugsource. This ensures that they are added
    into the final MMD list together.
  • The -debugsource subpackage with the name matching the SRPM name of its source
    package is marked to be included in a final MMD list together with SRPM. If the
    -debugsource package name does not match the SPRM name, it is the -debugsource
    built for normal binary RPM and this is case we are already handling even with
    the current code.

Why do we do this in first place?

To generate the modulemd file in Koji which contains the right RPM artifacts, so other tools know what is part of a module and what not (otherwise there is no way to get the list of resulting RPMs for each architecture)

Why not to just go through list of builds and find all RPMs from there? Trying to guess names is really bad (e.g. kernel and glibc have their own names).

Also doing dependency solving of RPMs (in a wrong way again), seems wrong to me.

Thinking about this more, you should just get RPMTAG_SOURCERPM and group them by that key.

@ignatenkobrain, the general issue here is that there are two module builds - "foo" and "foo-devel".

When some RPM is included in "foo" module, it's .src.rpm and -debuginfo/-debugsource must also be included in "foo" module", but otherwise, the .src.rpm and -debuginfo/-debugsource should be included in "foo-devel" module. The code which does that in MBS is mostly copied from Pungi which was doing this before.

I can group the RPMs bny the SOURCERPM, but I still need to do the basic deps resolving between debuginfo packages to include the right packages in a "foo" and "foo-devel" I'm afraid.

Yes, you need to group them by src.rpm, but you should not resolve any dependencies I think. Just include whatever matches into devel. If you can describe some example, I will try to help.

Hm, I can probably group by src.rpm and in case that "srpm" RPM built form that .src.rpm is included in "foo", I will also include all the -debuginfo/-debugsource RPMs buit from that .src.rpm. It would "mostly" work.

The issue is that there would be -debuginfo packages for subpackage which might not be included in "foo" at all, because they are filtered out in modulemd.

1 new commit added

  • Add tests for kernel and glibc to be sure we include all the debuginfo packages.
4 years ago

The issue is that there would be -debuginfo packages for subpackage which might not be included in "foo" at all, because they are filtered out in modulemd.

And why is that a problem? there are no -debuginfo packages for subpackage? debuginfo gets created for one SOURCERPM. If somebody filters out -debuginfo, he wants to filter it out entirely.

probably show me testcase and I can help you with this.

Discussed with @ignatenkobrain, I will rework that today in the afternoon.

Pull-Request has been closed by jkaluza

4 years ago