#506 Track artifacts coming from koji itself
Merged 2 years ago by mikem. Opened 3 years ago by tkopecek.
tkopecek/koji mw-tracking  into  master

@@ -108,6 +108,14 @@ 

     -  checksum: The checksum of the file.

     -  checksum\_type: The checksum type used.

  

+ -  For maps where **type = kojifile**, the following fields will be present:

+    -  filename: The name of the file.

+    -  filesize: The size of the file.

+    -  checksum: The checksum of the file.

+    -  checksum\_type: The checksum type used.

+    -  nvr: Build nvr from which this file origins.

+    -  archive\_id: ID of archive from specified build.

+ 

  -  The format may be extended with other types in the future.

  -  extra: A map containing information specific to the Content Generator

     that produced the files to import. For OSBS, the extra map should

file modified
+28

@@ -5331,6 +5331,10 @@ 

                  match = self.match_file(comp)

                  if match:

                      files.append(match)

+             elif comp['type'] == 'kojifile':

+                 match = self.match_kojifile(comp)

+                 if match:

+                     files.append(match)

              else:

                  raise koji.GenericError("Unknown component type: %(type)s" % comp)

          return rpms, files

@@ -5382,6 +5386,30 @@ 

          return None

          #raise koji.GenericError("No match: %(filename)s (size %(filesize)s, sum %(checksum)s" % comp)

  

+     def match_kojifile(self, comp):

+         # kojifile is identified by nvr + archive_id + filename

+         # additional checks by checksum

+         assert(comp['type'] == 'kojifile')

+         build = get_build(comp['nvr'])

+         if build is None:

+             logger.error("No match: NVR: %(nvr)" % comp)

+             return None

+         try:

+             archive = list_archives(buildID = build['id'],

+                                     archiveID = comp['archive_id'],

+                                     filename=comp['filename'],

+                                     size=comp['filesize'])[0]

+         except IndexError:

+             logger.error("No match: NVR: %(nvr), Archive: %(archive_id)s" % comp)

+             return None

+         if archive['checksum_type'] != koji.CHECKSUM_TYPES[comp['checksum_type']]:

+             logger.error("Failed to match archive %(filename)s, unsupported checksum type: %(checksum_type)s" % archive)

+         elif archive['checksum'] != comp['checksum']:

+             logger.error("Failed to match archive %(filename)s (size %(filesize)s, sum %(checksum)s", comp)

+         else:

+             return archive

+         return None

+ 

  

      def prep_outputs(self):

          metadata = self.metadata

@@ -52,7 +52,15 @@ 

                                   "filename": "jboss-eap-6.3.3-full-build.zip",

                                   "filesize": 12345678,

                                   "checksum": "5ec2f29c4e1c2e2aa6552836e236a158",

-                                  "checksum_type": "md5"}],

+                                  "checksum_type": "md5"},

+                                 {"type": "kojifile",

+                                  "filename": "jboss-eap-6.4.10-1.win6.src.zip",

+                                  "filesize": 21670719,

+                                  "checksum": "215cf04db6bdbefb9644f001995bd550",

+                                  "checksum_type": "md5",

+                                  "archive_id": 1,

+                                  "nvr": "jboss-eap-6.4.10-1.win6"}

+                  ],

                   "extra": {"osbs": {"build_id": 12345,

                                      "builder_image_id": 67890}}

                   }],

no initial comment

I'm not convinced that a separate toplevel "kojifile" type in the cg schema metadata is the right path, but otoh the "file" type is kind of a mess, so maybe it is best to distinguish them strongly.

Since we are adding a new toplevel component type, that is by definition intended to match a koji file, then we should actually raise errors if there are problems. E.g.

  • if there is no match
  • if the archive specified by archive_id does not match the nvr+filename given
  • if the checksum does not match
  • if the size does not match

Given that we are overspecifying the archive, I wonder how much of this data we should strictly require. Technically, archive_id is sufficient. It is nice to have the extra data for sanity checks, but I'm not sure if we need to require quite all the fields that are listed in the doc.

ok, I'll merge that version

Commit 8ac8167 fixes this pull-request

Pull-Request has been merged by mikem@redhat.com

2 years ago