#1552 Fix the provides of base modules
Merged 4 years ago by mprahl. Opened 4 years ago by breilly.
breilly/fm-orchestrator basestreamver-1334  into  master

@@ -228,9 +228,14 @@ 

  

          - module(platform:el8.1.0) = 80100 - Modules can require specific platform stream.

          - module(platform:el8) = 80100 - Module can also require just platform:el8.

+ 

+         :return: A boolean that is True if a provides for the stream version was added to the input

+             solvable.

          """

+         base_stream_ver = False

+ 

          if mmd.get_module_name() not in conf.base_module_names:

-             return

+             return base_stream_ver

  

          # When depsolving, we will need to follow specific rules to choose the right base

          # module, like sorting the base modules sharing the same virtual streams based on
@@ -240,13 +245,14 @@ 

          # to Provides.

          stream_version = ModuleBuild.get_stream_version(mmd.get_stream_name(), right_pad=False)

          if stream_version:

+             base_stream_ver = True

              self.solvable_provides(

                  solvable, mmd.get_module_name(), mmd.get_stream_name(), str(stream_version))

  

          xmd = mmd.get_xmd()

          # Return in case virtual_streams are not set for this mmd.

          if not xmd.get("mbs", {}).get("virtual_streams"):

-             return

+             return base_stream_ver

  

          version = stream_version or mmd.get_version()

          # For each virtual stream, add
@@ -254,6 +260,8 @@ 

          for stream in xmd["mbs"]["virtual_streams"]:

              self.solvable_provides(solvable, mmd.get_module_name(), stream, str(version))

  

+         return base_stream_ver

+ 

      def _get_base_module_stream_overrides(self, mmd):

          """

          Checks the xmd["mbs"]["buildrequires"] and returns the dict containing
@@ -326,11 +334,14 @@ 

              # no particular stream is used - for example when buildrequiring

              # "gtk: []"

              self.solvable_provides(solvable, n)

+ 

+             base_stream_ver = self._add_base_module_provides(solvable, mmd)

+ 

              # Add "Provides: module(name:stream) = version", so we can find buildrequired

              # modules when "gtk:[1]" is used and also choose the latest version.

-             self.solvable_provides(solvable, n, s, str(v))

- 

-             self._add_base_module_provides(solvable, mmd)

+             # Skipped if this is a base module with a stream version defined.

+             if not base_stream_ver:

+                 self.solvable_provides(solvable, n, s, str(v))

  

              base_module_stream_overrides = self._get_base_module_stream_overrides(mmd)

              # Fill in the "Requires" of this module, so we can track its dependencies

@@ -25,6 +25,7 @@ 

  

  import collections

  import pytest

+ import solv

  

  from module_build_service.mmd_resolver import MMDResolver

  from module_build_service import Modulemd
@@ -400,3 +401,35 @@ 

          }

  

          assert expanded == expected

+ 

+     @pytest.mark.parametrize(

+         "nsvc, requires, expected",

+         (

+             ("platform:f28:0:c0", {}, True),

+             ("platform:latest:5:c8", {}, False),

+             ("gtk:3:0:c8", {"platform": ["f28"]}, False)

+         ),

+     )

+     def test_base_module_stream_version(self, nsvc, requires, expected):

+         """

+         Tests that add_base_module_provides returns True for base modules with stream versions

+         """

+         mmd = self._make_mmd(nsvc, requires)

+         solvable = self.mmd_resolver.available_repo.add_solvable()

+         solvable.name = nsvc

+         solvable.evr = str(mmd.get_version())

+         solvable.arch = "x86_64"

+         assert self.mmd_resolver._add_base_module_provides(solvable, mmd) is expected

+ 

+     @pytest.mark.parametrize(

+         "nsvc, expected",

+         (

+             ("platform:f28:3:c0", {"module(platform)", "module(platform:f28) = 28.0"}),

+             ("platform:latest:5:c8", {"module(platform)", "module(platform:latest) = 5"}),

+         ),

+     )

+     def test_base_module_provides(self, nsvc, expected):

+         self.mmd_resolver.add_modules(self._make_mmd(nsvc, {}))

+         ns = nsvc.rsplit(":", 2)[0]

+         provides = self.mmd_resolver.solvables[ns][0].lookup_deparray(solv.SOLVABLE_PROVIDES)

+         assert {str(provide) for provide in provides} == expected

Build #705 failed (commit: c10b5be056a2f245f1d4ec67ef7082ebed745020).
Rebase or make new commits to rebuild.

Build #706 failed (commit: c10b5be056a2f245f1d4ec67ef7082ebed745020).
Rebase or make new commits to rebuild.

Rather than repeat this logic, how about calling self._add_base_module_provides(solvable, mmd) before self.solvable_provides(solvable, n, s, str(v)) and have self._add_base_module_provides(solvable, mmd) return a boolean denoting if the stream version was added as a provide?

rebased onto ca22632f1f192c62f9aa465a29e1680ccf9aee76

4 years ago

Build #710 failed (commit: ca22632f1f192c62f9aa465a29e1680ccf9aee76).
Rebase or make new commits to rebuild.

@breilly this looks great. Could explain the return value in the docstring for _add_base_module_provides?

Afterwards, this is good to merge.

rebased onto eb077643ccfe2367c00e495100f936e101cb0b47

4 years ago

Build #711 failed (commit: eb077643ccfe2367c00e495100f936e101cb0b47).
Rebase or make new commits to rebuild.

I know this is nitpicking, but the purpose of the returned boolean is not dictated by this method, that is determined whatever code uses that boolean. I would suggest just stating what the boolean is for.

For instance, you could say: A boolean that is True if a provides for the stream version was added to the input solvable

Seems existing tests does not catch the issue reported in #1334. @breilly could you please add a test for this fix?

rebased onto df319e1

4 years ago

Build #721 failed (commit: 8b854a9c12861bacd79676d615c5ec3e8a11c90a).
Rebase or make new commits to rebuild.

@breilly how about another test like this to verify that the correct provides were added to the solvable?

@pytest.mark.parametrize(
    "nsvc, expected",
    (
        ("platform:f28:3:c0", {"module(platform)", "module(platform:f28) = 28.0"}),
        ("platform:latest:5:c8", {"module(platform)", "module(platform:latest) = 5"}),
    ),
)
def test_base_module_provides(self, nsvc, expected):
    self.mmd_resolver.add_modules(self._make_mmd(nsvc, {}))
    ns = nsvc.rsplit(":", 2)[0]
    provides = self.mmd_resolver.solvables[ns][0].lookup_deparray(solv.SOLVABLE_PROVIDES)
    assert {str(provide) for provide in provides} == expected

2 new commits added

  • Added unit tests for add_base_module_provides
  • Fix the provides of base modules when it has a stream version
4 years ago

Thanks @mprahl, was having difficulty checking the provides directly but that works. Updated.

Suggest to rename n -> nsvc and req -> requires.

In MBS code, name requires represents the runtime requires of a modulemd generally, and name buildrequires is for the buildtime requires specifically.

The test name could be more specific here. Maybe replaced with def test_provide_base_module_stream_version. Feel free to reword and choose another one that can express the test purpose directly.

Similar with above comment. What is the purpose of this test? It calls add_modules rather than _add_base_module_provides. It'd be better express the test purpose clearly, for example by the name, as much as possible.

If you want to test _add_base_module_provides only, the test should just call it and verify the result and relative behaviors.

This specific test is to verify that add_modules adds the correct provides to the solvable.

This specific test is to verify that add_modules adds the correct provides to the solvable.

Should it be possible to merge test_add_base_module_provides and test_base_module_provides into one, which calls self.mmd_resolver.add_modules, with fixture data specific to different cases?

BTW, I feel _add_base_module_provides should be split into two functions for the purpose of getting result of whether a base module is added as a provides when it has stream version.

Currently, _add_base_module_provides adds a base module as provides in two conditions, 1) if it has stream version and 2) if it has virtual streams in xmd["mbs"]["virtual_streams"]. However, in this PR, the function returns a value to indicate True or False of the part of the operations. It looks a little strange. So, perhaps we can split the function into two, for example,

  • provide_base_module_if_has_stream_version
  • provide_base_module_with_virtual_streams

then, we can determine whether to add base module as a provides with its n, s, str(v) only with the return value from the first one.

What do you think?

Based on the feedback from @cqi above, seems like we should at least make the following easy changes to the new unit tests:

  • rename those parametrized variables as suggested
  • make the test name more specific
  • use get_nsvc_as_string instead of calculating ourselves

I don't think we need to refactor _add_base_module_provides any further at this time (i.e. the split suggested), but it might make sense to look into this separately.

As for merging the two new unit tests, I'm mostly ambivalent. I'd tend toward leaving them as-is unless there is a real problem with the way they are set up.

@breilly could we get those three easy bits applied at least?

@cqi if I'm off the mark about skipping/deferring the rest, please correct me

2 new commits added

  • Added unit tests for add_base_module_provides
  • Fix the provides of base modules when it has a stream version
4 years ago

Could you just use the nsvc directly?

You can use mmd.get_version() here.

Can you use is instead of ==?

@breilly a few minor things to change in test. Once addressed, this looks good to merge.

2 new commits added

  • Added unit tests for add_base_module_provides
  • Fix the provides of base modules when it has a stream version
4 years ago

Commit 66a2362 fixes this pull-request

Pull-Request has been merged by mprahl

4 years ago

Pull-Request has been merged by mprahl

4 years ago

Build #732 failed (commit: c2561da).
Rebase or make new commits to rebuild.