#1256 Modify ModuleBuild._add_virtual_streams_filter to use a subquery to better support Postgres
Merged 4 years ago by mprahl. Opened 4 years ago by mprahl.

@@ -419,11 +419,21 @@ 

          if not virtual_streams:

              return query

  

-         return (

-             query.join(VirtualStream, ModuleBuild.virtual_streams)

+         # Create a subquery that filters down all the module builds that contain the virtual

+         # streams. Using distinct is necessary since a module build may contain multiple virtual

+         # streams that are desired.

+         modules_with_virtual_streams = (

+             session.query(ModuleBuild)

+             .join(VirtualStream, ModuleBuild.virtual_streams)

              .filter(VirtualStream.name.in_(virtual_streams))

+             .order_by(ModuleBuild.id)

              .distinct(ModuleBuild.id)

-         )

+         ).subquery()

+ 

+         # Join the original query with the subquery so that only module builds with the desired

+         # virtual streams remain

+         return query.join(

+             modules_with_virtual_streams, ModuleBuild.id == modules_with_virtual_streams.c.id)

  

      @staticmethod

      def get_last_builds_in_stream_version_lte(session, name, stream_version, virtual_streams=None):
@@ -444,7 +454,7 @@ 

              session.query(ModuleBuild)

              .filter(ModuleBuild.name == name)

              .filter(ModuleBuild.state == BUILD_STATES["ready"])

-             .order_by(ModuleBuild.version.desc())

+             .order_by(sqlalchemy.cast(ModuleBuild.version, db.BigInteger).desc())

          )

  

          query = ModuleBuild._add_stream_version_lte_filter(session, query, stream_version)

@@ -0,0 +1,93 @@ 

+ // Build an empty module that buildrequires a virtual stream

+ 

+ def runTests() {

+   def clientcert = ca.get_ssl_cert("mbs-${TEST_ID}-koji-admin")

+   koji.setConfig("https://koji-${TEST_ID}-hub/kojihub", "https://koji-${TEST_ID}-hub/kojifiles",

+                  clientcert.cert, clientcert.key, ca.get_ca_cert().cert)

+   def tags = koji.callMethod("listTags")

+   if (!tags.any { it.name == "module-f28" }) {

+     koji.addTag("module-f28")

+   }

+   if (!tags.any { it.name == "module-f28-build" }) {

+     koji.addTag("module-f28-build", "--parent=module-f28", "--arches=x86_64")

+   }

+   try {

+     // There's currently no way to query whether a given user has CG access, so just add it

+     // and hope no one else has already done it.

+     koji.runCmd("grant-cg-access", "mbs-${TEST_ID}-koji-admin", "module-build-service", "--new")

+   } catch (ex) {

+     echo "Granting cg-access to mbs-${TEST_ID}-koji-admin failed, assuming it was already provided in a previous test"

+   }

+ 

+   if (!koji.callMethod("listBTypes").any { it.name == "module" }) {

+     koji.callMethodLogin("addBType", "module")

+   }

+ 

+   def testmodule = """

+   document: modulemd

+   version: 1

+   data:

+     summary: A test module in all its beautiful beauty

+     description: This module buildrequires a virtual stream of the platform module

+     name: testmodule

+     stream: buildrequire_virtual_stream

+     license:

+       module: [ MIT ]

+     dependencies:

+       buildrequires:

+           platform: fedora

+       requires:

+           platform: fedora

+     references:

+       community: https://docs.pagure.org/modularity/

+       documentation: https://fedoraproject.org/wiki/Fedora_Packaging_Guidelines_for_Modules

+   """

+ 

+   def buildparams = """

+         {"modulemd": "${testmodule}",

+          "owner":  "mbs-${TEST_ID}-koji-admin"}

+       """

+   def resp = httpRequest(

+         httpMode: "POST",

+         url: "https://mbs-${TEST_ID}-frontend/module-build-service/1/module-builds/",

+         acceptType: "APPLICATION_JSON",

+         contentType: "APPLICATION_JSON",

+         requestBody: buildparams,

+         ignoreSslErrors: true,

+       )

+   if (resp.status != 201) {

+     echo "Response code was ${resp.status}, output was ${resp.content}"

+     error "POST response code was ${resp.status}, not 201"

+   }

+   def buildinfo = readJSON(text: resp.content)

+   timeout(10) {

+     waitUntil {

+       resp = httpRequest(

+         url: "https://mbs-${TEST_ID}-frontend/module-build-service/1/module-builds/${buildinfo.id}",

+         ignoreSslErrors: true,

+       )

+       if (resp.status != 200) {

+         echo "Response code was ${resp.status}, output was ${resp.content}"

+         error "GET response code was ${resp.status}, not 200"

+       }

+       def modinfo = readJSON(text: resp.content)

+       if (modinfo.state_name == "failed") {

+         error "Module ${modinfo.id} (${modinfo.name}) is in the ${modinfo.state_name} state"

+       } else if (modinfo.state_name != "ready") {

+         echo "Module ${modinfo.id} (${modinfo.name}) is in the ${modinfo.state_name} state, not ready"

+         return false

+       }

+       def br_platform_stream = modinfo.buildrequires.platform.stream

+       if (br_platform_stream != "f28") {

+         echo "Module ${modinfo.id} (${modinfo.name}) buildrequires platform:${br_platform_stream}, \

+           but it should buildrequire platform:f28"

+         return false

+       }

+ 

+       echo "All checks passed"

+       return true

+     }

+   }

+ }

+ 

+ return this;

@@ -11,12 +11,18 @@ 

    if (!tags.any { it.name == "module-f28-build" }) {

      koji.addTag("module-f28-build", "--parent=module-f28", "--arches=x86_64")

    }

-   // There's currently no way to query whether a given user has CG access, so just add it

-   // and hope no one else has already done it.

-   koji.runCmd("grant-cg-access", "mbs-${TEST_ID}-koji-admin", "module-build-service", "--new")

+   try {

+     // There's currently no way to query whether a given user has CG access, so just add it

+     // and hope no one else has already done it.

+     koji.runCmd("grant-cg-access", "mbs-${TEST_ID}-koji-admin", "module-build-service", "--new")

+   } catch (ex) {

+     echo "Granting cg-access to mbs-${TEST_ID}-koji-admin failed, assuming it was already provided in a previous test"

+   }

+ 

    if (!koji.callMethod("listBTypes").any { it.name == "module" }) {

      koji.callMethodLogin("addBType", "module")

    }

+ 

    def buildparams = """

          {"scmurl": "https://src.fedoraproject.org/forks/mikeb/modules/testmodule.git?#8b3fb16160f899ce10905faf570f110d52b91154",

           "branch": "empty-f28",

@@ -2429,7 +2429,6 @@ 

          )

          rv = self.client.post(post_url, data=json.dumps({"branch": "master", "scmurl": scm_url}))

          data = json.loads(rv.data)

-         print(data)

  

          mmd = load_mmd(data[0]["modulemd"])

          assert len(mmd.get_dependencies()) == 1

The old way performed a DISTINCT (module_builds.id) on the original query passed in to ModuleBuild._add_virtual_streams_filter, but this caused issues when the original query was ordered by something other than ID on Postgres databases. This new approach uses a subquery to filter that module builds with the desired virtual streams, and then joins this subquery to the original query.

This also casts the version to an integer for proper sorting in get_last_builds_in_stream_version_lte.

2 new commits added

  • Add an integration test for buildrequiring a virtual stream
  • Add a try/catch around the grant-cg-access code in the integration tests
4 years ago

rebased onto beb38b0

4 years ago

@mikeb do you know why the added C3I test wasn't executed? It seems that the two test cases it ran were the existing ones.

The way the pipeline jobs are configured, the integration tests are always run from the head of the master branch. It's something of a security feature. This was discussed here:

http://post-office.corp.redhat.com/archives/pnt-devops-cicd-discuss/2019-February/msg00019.html

In this case, I think it would be fine to take the definition of the pipelines from the master branch, but the tests themselves from the PR branch. That would require disabling the default checkout of the source on the agent, and instead doing a manual checkout in the Prepare step, similar to what's done in mbs-build.Jenkinsfile. This would allow the test cases to evolve with the code. @rayson What do you think?

+1 if the tests on postgres works.

@jkaluza it worked in stage using Postgres. I'll merge this and see if the C3I tests pass before releasing the 2.20.0-2 RPM with this patch.

Pull-Request has been merged by mprahl

4 years ago