#1596 backend: get back to using standard createrepo_c command
Merged 3 years ago by praiskup. Opened 3 years ago by frostyx.
copr/ frostyx/copr createrepo-c-for-modules  into  master

file modified
+2 -2
@@ -21,7 +21,7 @@ 

  

  BuildArch:  noarch

  BuildRequires: asciidoc

- BuildRequires: createrepo_c

+ BuildRequires: createrepo_c >= 0.16.1

  BuildRequires: libappstream-glib-builder

  BuildRequires: libxslt

  BuildRequires: redis
@@ -56,7 +56,7 @@ 

  

  Requires:   (copr-selinux if selinux-policy-targeted)

  Requires:   ansible

- Requires:   createrepo_c

+ Requires:   createrepo_c >= 0.16.1

  Requires:   crontabs

  Requires:   gawk

  Requires:   libappstream-glib-builder

file modified
+1 -1
@@ -130,7 +130,7 @@ 

  

  

  def run_createrepo(opts):

-     createrepo_cmd = ['/usr/bin/createrepo_mod', opts.directory, '--database', '--ignore-lock',

+     createrepo_cmd = ['/usr/bin/createrepo_c', opts.directory, '--database', '--ignore-lock',

                        '--local-sqlite', '--cachedir', '/tmp/', '--workers', '8']

  

      if "epel-5" in opts.directory or "rhel-5" in opts.directory:

As modulemd-tools documentation suggests, the createrepo_mod tool is
needed only on older systems. The modularity support is built into the
createrepo_c tool itself since version 0.16.1 (which is available
for F32+)

Hmpfs, in our case we speak about this code:
https://github.com/rpm-software-management/createrepo_c/blob/59b508f872d5200bc6b04c32de1793055ec07e4f/src/createrepo_c.c#L257-L276

So I'm afraid it is not as easy as it looks, we still should put the modulemd file name into pkglist file if we want to use it.

This is interesting, yesterday I run beaker tests and the passed successfully. I even did

chmod -x /usr/bin/createrepo_mod # I want copr-backend to use createrepo_c only

to make sure, that createrepo_c does all the work.

Anyway, thank you for the link, I will examine that.

Maybe because we run full createrepo_c run for modules?

Indeed, for creating module repositories, we execute createrepo_c
like this

[2020-11-24 16:48:49,852][  INFO][PID:2958293][/usr/bin/copr-repo.pid-2958293][copr-repo:run_cmd:35] running: /usr/bin/createrepo_c /var/lib/copr/public_html/results/frostyx/module-coprtestmodule-beakertest-20201124161901/modules/fedora-32-x86_64+coprtestmodule-beakertest-20201124161901/latest/x86_64 --database --ignore-lock --local-sqlite --cachedir /tmp/ --workers 8

That's because we don't have modules in the main project repository
but create them in separate directories. This is a known issue #1576
that we want to fix. As a consequence, the createrepo_c needs to be
run from scratch to create the initial module repository, and for that
repository, it is never run again, so createrepo_c is never called
with --recycle-pkglist for modules.

Once the #1576 gets fixed, we will surely need to put the modulemd
file name in to the pkglist as you said but then the beaker tests are
going to fail and tell us to do that, so I would wait till then, so we
can be sure we implemented it properly. What do you think?

Regarding this PR and issue #1593 (which I forgot to mention in the
commit message), I am not sure whether it fixes the performance
penalty caused by os.walk() in modulemd-tools. Maybe there will be
the same penalty but caused by directory walk in createrepo_c
itself. Do you have the original reproducer, so we can measure the
difference? In any case, I would go with this PR since there is no
reason for us to continue using the createrepo_mod wrapper around
it.

We can also prioritize the #1576 if you want to.

Regarding this PR and issue #1593 (which I forgot to mention in the
commit message),

Yeah, please mention in the commit that you fix it.

I am not sure whether it fixes the performance
penalty caused by os.walk() in modulemd-tools.

I believe so. Since createrepo_c is not actively looking for the yaml
files with --recycle-pkglist (and friends). I tested on iucar/cran, and
the createrepo_c itself was really quick (a minute or so) .... the rest
of createrepo_mod logic took the rest of time (~25 minutes).

So I'm +1 here.

We should also add an explicit requirement on appropriate version of createrepo_c.

rebased onto d028c8c0daf3a3bf85885710f92f335f2c3e253c

3 years ago

Yeah, please mention in the commit that you fix it.

Done

We should also add an explicit requirement on appropriate version of createrepo_c.

Done

The CI is failing because there is not that new createrepo_c version for F31

rebased onto 56fd4c1

3 years ago

FTR, I disabled f31 in our copr/copr-dev and copr/copr projects; it doesn't make sense to test it because Bodhi doesn't allow us to ship F31 packages anymore.

Pull-Request has been merged by praiskup

3 years ago