#1058 Make pungi build ISO reproducibly
Merged 6 years ago by lsedlar. Opened 6 years ago by marmarek.
marmarek/pungi reproducible-build  into  master

Set repodata mtime to SOURCE_DATE_EPOCH
Marek Marczykowski-Górecki • 6 years ago  
Make sure .treeinfo file is sorted
Marek Marczykowski-Górecki • 6 years ago  
Use constant MBR ID for isohybrid
Marek Marczykowski-Górecki • 6 years ago  
Use xorriso instead of genisoimage
Marek Marczykowski-Górecki • 6 years ago  
Use $SOURCE_DATE_EPOCH (if set) in discinfo file
Marek Marczykowski-Górecki • 6 years ago  
file modified
+3 -1
@@ -21,6 +21,7 @@ 

  BuildRequires:  python2-six

  BuildRequires:  python2-multilib

  BuildRequires:  python2-dogpile-cache

+ BuildRequires:  python2-dict-sorted

  

  Requires:       createrepo >= 0.4.11

  Requires:       yum => 3.4.3-28
@@ -40,7 +41,7 @@ 

  Requires:       cvs

  Requires:       yum-utils

  Requires:       isomd5sum

- Requires:       genisoimage

+ Requires:       xorriso

  Requires:       gettext

  Requires:       syslinux

  Requires:       git
@@ -51,6 +52,7 @@ 

  Requires:       python2-libcomps

  Requires:       python2-six

  Requires:       python2-dogpile-cache

+ Requires:       python2-dict-sorted

  

  BuildArch:      noarch

  

@@ -32,6 +32,7 @@ 

  )

  

  

+ import os

  import time

  

  
@@ -43,7 +44,7 @@ 

      if not isinstance(disc_numbers, list):

          raise TypeError("Invalid type: disc_numbers type is %s; expected: <list>" % type(disc_numbers))

      if not timestamp:

-         timestamp = "%f" % time.time()

+         timestamp = os.environ.get('SOURCE_DATE_EPOCH', "%f" % time.time())

      with open(file_path, "w") as f:

          f.write("%s\n" % timestamp)

          f.write("%s\n" % description)

file modified
+10 -1
@@ -26,6 +26,7 @@ 

  import subprocess

  import createrepo

  import ConfigParser

+ from sdict import AlphaSortedDict

  from fnmatch import fnmatch

  

  import arch as arch_module
@@ -89,6 +90,10 @@ 

  class MyConfigParser(ConfigParser.ConfigParser):

      """A subclass of ConfigParser which does not lowercase options"""

  

+     def __init__(self, *args, **kwargs):

+         kwargs['dict_type'] = AlphaSortedDict

+         ConfigParser.ConfigParser.__init__(self, *args, **kwargs)

+ 

      def optionxform(self, optionstr):

          return optionstr

  
@@ -1435,6 +1440,9 @@ 

              conf.baseurl = baseurl

          if compress_type:

              conf.compress_type = compress_type

+         if 'SOURCE_DATE_EPOCH' in os.environ:

+             conf.revision = os.environ['SOURCE_DATE_EPOCH']

+             conf.clamp_mtime_to = int(os.environ['SOURCE_DATE_EPOCH'])

This is backwards compatible, right? If createrepo does not have the patch, it will be effectively a no-op?

Yes, exactly.
There is also a discussion regarding linked createrepo patch, if the final version would use a different option, I'll update it here too.

          repomatic = createrepo.MetaDataGenerator(conf)

          self.logger.info('Making repodata')

          repomatic.doPkgMetadata()
@@ -1740,7 +1748,7 @@ 

                             clean=True) # This is risky...

  

          # setup the base command

-         mkisofs = ['/usr/bin/mkisofs']

+         mkisofs = ['/usr/bin/xorriso', '-as', 'mkisofs']

          mkisofs.extend(['-v', '-U', '-J', '--joliet-long', '-R', '-T', '-m', 'repoview', '-m', 'boot.iso']) # common mkisofs flags

  

          x86bootargs = ['-b', 'isolinux/isolinux.bin', '-c', 'isolinux/boot.cat', 
@@ -1762,6 +1770,7 @@ 

          ppcbootargs.append('-hfs-bless') # must be last

  

          isohybrid = ['/usr/bin/isohybrid']

+         isohybrid.extend(['--id', '42'])

  

          # Check the size of the tree

          # This size checking method may be bunk, accepting patches...

file modified
+1
@@ -63,6 +63,7 @@ 

          "productmd",

          "six",

          'dogpile.cache',

+         'dict.sorted',

          ],

      tests_require = [

          "mock",

This PR is a part of the “reproducible builds” effort 1.

The attached patches replaces timestamps with SOURCE_DATE_EPOCH2 variable (if set) to ensure reproducible result. Also .treeinfo file is made deterministic and genisoimage is replaced with xorriso which already support generating reproducible images.
The top-most commit depends on https://github.com/rpm-software-management/createrepo/pull/9. If you prefer, I can drop it from this PR and open new one when it gets merged.
To make the ISO image fully reproducible, similar changes are needed in lorax, but there are no direct dependencies between those two patch series.

With this all applied, pungi given the same set of input rpm packages (and build options + SOURCE_DATE_EPOCH variable) will build identical ISO image.

rebased onto a0f992d6faca8a837b1d25e5e54ea40e46765bd2

6 years ago

5 new commits added

  • Set repodata mtime to SOURCE_DATE_EPOCH
  • Make sure .treeinfo file is sorted
  • Use constant MBR ID for isohybrid
  • Use xorriso instead of genisoimage
  • Use $SOURCE_DATE_EPOCH (if set) in discinfo file
6 years ago

In general reproducibility is a good thing.

What exactly are you trying to achieve here though? All changes here affect pungi.gather module, which is backing pungi command that is most likely going to be dropped (since it depends on yum which is not compatible with Python 3 and so far there was no interested to port it to DNF). Also it is not used by rel-eng for anything official in Fedora anymore.

I think I've heard it some years ago and it's still there...
Anyway, I've done this for Qubes OS, which still use pungi command.
When it will be migrated to whatever new tool is (pungi-koji?), I'll
happily port this set of patches to the new code, but right now I have
very little possibility to test it there.

How about one line timestamp = os.environ.get("SOURCE_DATE_EPOCH", "%f" % time.time())?

This is backwards compatible, right? If createrepo does not have the patch, it will be effectively a no-op?

I see. In that case you are probably fine, the code will remain in this repo for the foreseeable future, only it won't be packaged in Fedora eventually.

I updated the Jenkins job to install sdict module.

As I don't have experience with xorriso, I assume with the -as mkisofs it should work as the emulated command, right?

Yes, exactly.
There is also a discussion regarding linked createrepo patch, if the final version would use a different option, I'll update it here too.

I updated the Jenkins job to install sdict module.

I'll push that timestamp = os.environ.get change to trigger new build.

As I don't have experience with xorriso, I assume with the -as mkisofs it should work as the emulated command, right?

Yes, command line options are compatible in this mode.

rebased onto 41e144e

6 years ago

Thanks for the patch. I'll merge it now, and I plan a release later this week. If the createrepo patch changes, further patches would be welcome.

Pull-Request has been merged by lsedlar

6 years ago