#1184 Several multlib fixes
Merged 4 years ago by praiskup. Opened 4 years ago by praiskup.
Unknown source multilib-fixes  into  master

file modified
+28 -11
@@ -217,25 +217,42 @@

  Multilib

  --------

  

- In Copr you cannot build multilib packages, i.e. build a i386 package in a x86_64 chroot. If you need to use multilib packages you will need to specify both repos on your x86_64 system. An example of the rhughes/f20-gnome-3-12 project follows:

+ In Copr, you cannot build an i386 package into x86_64 repository (also known as

+ multilib package) like e.g. in Koji.  You can though build for both

+ multilib-pair chroots (e.g. ``fedora-31-x86_64`` and ``fedora-31-i386``)

+ separately, and users can enable both multilib-pair repositories - so in turn

+ all built 32bit and 64bit packages will be available concurrently.

  

- Install the repo file for this Copr by copying it to /etc/yum.repos.d and run yum update. If you have a multilib system (i.e. are running x86_64 but have i686 packages installed for flash / steam) then you'll need to modify the .repo file to include both i386 and x86_64 sources, e.g.::

+ If you want to automatize this, specify that your project is supposed to be

+ "multilib capable".  Either in commandline::

  

-     $ cat rhughes-f20-gnome-3-12.repo

-     [rhughes-f20-gnome-3-12-i386]

-     name=Copr repo for f20-gnome-3-12 owned by rhughes (i386)

-     baseurl=http://copr-be.cloud.fedoraproject.org/results/rhughes/f20-gnome-3-12/fedora-$releasever-i386/

+     copr create --multilib=on [other options]

+ 

+ or by checkbox on ``Project -> Settings`` web-UI page.

+ 

+ When (a) this feature is enabled for project and (b) the project also contains

+ multilib-pair chroots, the relevant copr web-UI project page will also provide

+ multilib repo files button (aside the normal one) so user can pick those.  On

+ top of that, ``dnf copr enable <owner>/<project>`` installs the multilib

+ repofile automatically instead of the normal one on multilib capable system.

+ 

+ Users can also manually install the multilib repofiles on multilib capable

+ system regardless of the project settings, those repofile can e.g. look like::

+ 

+     $ cat /etc/yum.repos.d/rhughes-f20-gnome-3-12.repo

+     [copr:copr.fedorainfracloud.org:rhughes:gnome-3-12]

+     name=Copr repo for f20-gnome-3-12 owned by rhughes

+     baseurl=http://copr-be.cloud.fedoraproject.org/results/rhughes/f20-gnome-3-12/fedora-$releasever-$basearch/

      skip_if_unavailable=True

      gpgcheck=0

-     cost=900

      enabled=1

  

-     [rhughes-f20-gnome-3-12-x86_64]

-     name=Copr repo for f20-gnome-3-12 owned by rhughes (x86_64)

-     baseurl=http://copr-be.cloud.fedoraproject.org/results/rhughes/f20-gnome-3-12/fedora-$releasever-x86_64/

+     [copr:copr.fedorainfracloud.org:rhughes:gnome-3-12:ml]

+     name=Copr repo for f20-gnome-3-12 owned by rhughes (i386)

+     baseurl=http://copr-be.cloud.fedoraproject.org/results/rhughes/f20-gnome-3-12/fedora-$releasever-i386/

      skip_if_unavailable=True

      gpgcheck=0

-     cost=800

+     cost=1100

      enabled=1

  

  Status Badges

@@ -1,11 +1,15 @@

  [{{ repo_id }}]

  name=Copr repo for {{ copr_dir.name }} owned by {{ copr_dir.copr.owner_name }}

+ {{- " (" + arch + ")" if arch }}

  baseurl={{ url | fix_url_https_backend }}

  type=rpm-md

  skip_if_unavailable=True

  gpgcheck={{ config.REPO_GPGCHECK | default("1")}}

  gpgkey={{ pubkey_url | fix_url_https_backend  }}

  repo_gpgcheck=0

+ {% if cost %}

+ cost={{ cost }}

+ {% endif %}

  enabled=1

  enabled_metadata=1

  {% if copr_dir.copr.module_hotfixes %}

@@ -723,7 +723,6 @@

  @coprs_ns.route("/g/<group_name>/<copr_dirname>/repo/<name_release>/", defaults={"repofile": None})

  @coprs_ns.route("/g/<group_name>/<copr_dirname>/repo/<name_release>/<repofile>")

  @req_with_copr_dir

- @cache.memoize(timeout=5*60)

  def generate_repo_file(copr_dir, name_release, repofile):

      """ Generate repo file for a given repo name.

          Reponame = username-coprname """
@@ -732,7 +731,7 @@

      return render_generate_repo_file(copr_dir, name_release, arch)

  

  

- def render_repo_template(copr_dir, mock_chroot, arch=None):

+ def render_repo_template(copr_dir, mock_chroot, arch=None, cost=None):

      repo_id = "copr:{0}:{1}:{2}{3}".format(

          app.config["PUBLIC_COPR_HOSTNAME"].split(":")[0],

          copr_dir.copr.owner_name.replace("@", "group_"),
@@ -744,9 +743,10 @@

      pubkey_url = urljoin(url, "pubkey.gpg")

      return flask.render_template("coprs/copr_dir.repo", copr_dir=copr_dir,

                                   url=repo_url, pubkey_url=pubkey_url,

-                                  repo_id=repo_id) + "\n"

+                                  repo_id=repo_id, arch=arch, cost=cost) + "\n"

  

  

+ @cache.memoize(timeout=5*60)

  def render_generate_repo_file(copr_dir, name_release, arch=None):

      copr = copr_dir.copr

  
@@ -767,14 +767,21 @@

          raise ObjectNotFound("Chroot {} does not exist in {}".format(

              searched_chroot, copr.full_name))

  

+     # append multilib counterpart repo only upon explicit request (ach != None),

+     # and only if the chroot actually is multilib capable

+     multilib_on = (arch and

+                    copr.multilib and

+                    mock_chroot in copr.active_multilib_chroots)

+ 

      # normal, arch agnostic repofile

      response_content = render_repo_template(copr_dir, mock_chroot)

  

-     # append multilib counterpart repo only upon explicit request (ach != None),

-     # and only if the chroot actually is multilib capable

-     copr = copr_dir.copr

-     if arch and copr.multilib and mock_chroot in copr.active_multilib_chroots:

-         response_content += "\n" + render_repo_template(copr_dir, mock_chroot, 'i386')

+     if multilib_on:

+         # slightly lower cost than the default dnf cost=1000

+         response_content += "\n" + render_repo_template(

+             copr_dir, mock_chroot,

+             models.MockChroot.multilib_pairs[mock_chroot.arch],

+             cost=1100)

  

      response = flask.make_response(response_content)

  

@@ -703,7 +703,7 @@

                                 f_mock_chroots_many, f_custom_builds, f_db):

  

          r_non_ml_chroot = self.tc.get(

-             "/coprs/{0}/{1}/repo/fedora-18/some.repo&arch=x86_64".format(

+             "/coprs/{0}/{1}/repo/fedora-18/some.repo?arch=x86_64".format(

                  self.u1.name, self.c1.name))

  

          for f_version in range(19, 24):
@@ -711,13 +711,15 @@

                  # with disabled multilib there's no change between fedora repos,

                  # no matter what the version or architecture is

                  r_ml_chroot = self.tc.get(

-                     "/coprs/{0}/{1}/repo/fedora-{2}/some.repo&arch={3}".format(

+                     "/coprs/{0}/{1}/repo/fedora-{2}/some.repo?arch={3}".format(

                          self.u1.name, self.c1.name, f_version, arch))

                  assert r_ml_chroot.data == r_non_ml_chroot.data

  

          self.c1.multilib = True

          self.db.session.commit()

  

+         cache.clear() # f18 repofile is cached

+ 

          # The project is now multilib, but f18 chroot doesn't have i386

          # countepart in c1

  
@@ -738,28 +740,48 @@

              assert r_ml_chroot.data == r_ml_first_chroot.data

              assert r_ml_chroot.data != r_non_ml_chroot.data

  

+             # and the non-ml variants need to match non-ml chroot f18

+             # (this also checks that we don't cache 'some.repo' requests with

+             # 'some.repo&arch=...')

+             r_non_ml_repofile = self.tc.get(

+                 "/coprs/{0}/{1}/repo/fedora-{2}/some.repo".format(

+                     self.u1.name, self.c1.name, f_version))

+             assert r_non_ml_repofile.data == r_non_ml_chroot.data

+ 

          def parse_repofile(string):

              lines = string.split('\n')

-             repoids = [x.strip('[]') for x in lines if re.match(r'^\[.*\]$', x)]

-             baseurls = [x.split('=')[1] for x in lines if re.match(r'^baseurl=.*', x)]

-             gpgkeys = [x.split('=')[1] for x in lines if re.match(r'^gpgkey=.*', x)]

-             return repoids, baseurls, gpgkeys

+             def get_params(name, lines):

+                 return [x.split('=')[1] for x in lines

+                         if re.match(r'^{}=.*'.format(name), x)]

+             return (

+                 [x.strip('[]') for x in lines if re.match(r'^\[.*\]$', x)],

+                 get_params('baseurl', lines),

+                 get_params('gpgkey', lines),

+                 get_params('name', lines),

+                 get_params('cost', lines),

+             )

  

          non_ml_repofile = r_non_ml_chroot.data.decode('utf-8')

          ml_repofile = r_ml_first_chroot.data.decode('utf-8')

  

-         repoids, baseurls, gpgkeys = parse_repofile(non_ml_repofile)

+         repoids, baseurls, gpgkeys, _, costs = parse_repofile(non_ml_repofile)

          assert len(repoids) == len(baseurls) == len(gpgkeys) == 1

+         assert len(costs) == 0

  

          normal_gpgkey = gpgkeys[0]

          normal_repoid = repoids[0]

          normal_baseurl = baseurls[0]

  

-         repoids, baseurls, gpgkeys = parse_repofile(ml_repofile)

+         repoids, baseurls, gpgkeys, names, costs = parse_repofile(ml_repofile)

          assert len(repoids) == len(baseurls) == len(gpgkeys) == 2

+         assert len(costs) == 1

+         assert costs[0] == '1100'

  

          assert normal_repoid == repoids[0]

          assert normal_repoid + ':ml' == repoids[1]

+         assert 'x86_64' not in names[0]

+         assert '(i386)' not in names[0]

+         assert '(i386)' in names[1]

          assert gpgkeys[0] == gpgkeys[1] == normal_gpgkey

          assert normal_baseurl == baseurls[0]

          assert normal_baseurl.rsplit('-', 1)[0] == baseurls[1].rsplit('-', 1)[0]

Please see commits for comments. FTR, I noticed the problems while
documenting the new feature.

rebased onto d90033f7e91207100ebfb188176924aefe04eaf3

4 years ago

rebased onto f24ef2e34bf8f7a934ae2c2126520939014794d8

4 years ago

Metadata Update from @praiskup:
- Request assigned

4 years ago

In "build a i386 package", there should "an" instead of "a"

Also, the sentence is super hard to interpret in your (well at least mine :D) head because of the .e.g. and i.e. right next to each other.
I would maybe write it from the other end. Something like:
'In Copr, you cannot build an i386 package into x86_64 repository (also known as multilib package) like e.g. in Koji.'
What do you think? Do you find it better to understand? We can also keep the variant proposed by you, I have no problem with that.

Personally, I would go with

"multilib capable". Either in commandline::

I made just a few suggestions, but nothing really that important. Otherwise LGTM.

Just FTR, I've been playing with the render_generate_repo_file function and was confused what am I doing wrong and where are my changes ... turned out it was the caching. I think we should configure it somehow, to disable the caching in a development environment.

rebased onto d43c84ac655111dd856d93af94dcc828721dbe6b

4 years ago

I made just a few suggestions, but nothing really that important.

Thank you! I've used your notes.

I think we should configure it somehow, to disable the caching in a development environment.

It's hard to tell ... it's often useful to debug whether caching works..

I only touched the documentation since the last LGTM, so merging.

rebased onto 7f727c3

4 years ago

Thank you for the review!

Commit 7f727c3 fixes this pull-request

Pull-Request has been merged by praiskup

4 years ago

Pull-Request has been merged by praiskup

4 years ago