#1461 backend: use createrepo_mod tool for generating module repodata
Merged 3 years ago by praiskup. Opened 3 years ago by frostyx.
copr/ frostyx/copr use-createrepo-mod  into  master

@@ -52,6 +52,7 @@ 

  BuildRequires: python3-setproctitle

  BuildRequires: python3-sphinx

  BuildRequires: python3-tabulate

+ BuildRequires: modulemd-tools >= 0.4

  

  Requires:   (copr-selinux if selinux-policy-targeted)

  Requires:   ansible
@@ -88,6 +89,7 @@ 

  Requires:   redis

  Requires:   rpm-sign

  Requires:   rsync

+ Requires:   modulemd-tools >= 0.4

  

  Requires(post): systemd

  Requires(preun): systemd

@@ -109,13 +109,6 @@ 

  {packages_dir}/repodata

  """

  

- INCLUDE_MODULES = \

-     """/usr/bin/modifyrepo_c \

- --mdtype modules \

- --compress-type gz \

- {packages_dir}/modules.yaml \

- {packages_dir}/repodata

- """

  

  def add_appdata(path, username, projectname, lock=None):

      out = ""
@@ -160,14 +153,6 @@ 

      return out

  

  

- def add_modules(path):

-     if os.path.exists(os.path.join(path, "modules.yaml")):

-         return run_cmd_unsafe(

-             INCLUDE_MODULES.format(packages_dir=path), os.path.join(path, "createrepo.lock")

-         )

-     return ""

- 

- 

  def createrepo(path, username, projectname, devel=False, base_url=None):

      """

      Creates repodata.  Depending on the "auto_createrepo" parameter it either
@@ -185,8 +170,7 @@ 

      if not devel:

          out_cr = createrepo_unsafe(path)

          out_ad = add_appdata(path, username, projectname)

-         out_md = add_modules(path)

-         return "\n".join([out_cr, out_ad, out_md])

+         return "\n".join([out_cr, out_ad])

  

      # Automatic createrepo disabled.  Even so, we still need to createrepo in

      # special "devel" directory so we can later build packages against it.

file modified
+2 -14
@@ -130,9 +130,8 @@ 

  

  

  def run_createrepo(opts):

-     createrepo_cmd = ['/usr/bin/createrepo_c', '--database', '--ignore-lock',

-                       '--local-sqlite', '--cachedir', '/tmp/', '--workers', '8',

-                       opts.directory]

+     createrepo_cmd = ['/usr/bin/createrepo_mod', opts.directory, '--database', '--ignore-lock',

+                       '--local-sqlite', '--cachedir', '/tmp/', '--workers', '8']

  

      if "epel-5" in opts.directory or "rhel-5" in opts.directory:

          # this is because rhel-5 doesn't know sha256
@@ -257,16 +256,6 @@ 

      # fix them, so that lighttpd could serve appdata dir

      run_cmd(['chmod', '-R', '+rX', os.path.join(path, 'appdata')], opts)

  

- def add_modular_metadata(opts):

-     if not os.path.exists(os.path.join(opts.directory, "modules.yaml")):

-         return

-     run_cmd(

-         ["/usr/bin/modifyrepo_c",

-          "--mdtype", "modules",

-          "--compress-type", "gz",

-          os.path.join(opts.directory, 'modules.yaml'),

-          os.path.join(opts.directory, 'repodata')], opts)

- 

  

  def delete_builds(opts):

      # To avoid race conditions, remove the directories _after_ we have
@@ -340,7 +329,6 @@ 

      # TODO: racy, these info aren't available for some time, once it is

      # possible we should move those two things before 'delete_builds' call.

      add_appdata(opts)

-     add_modular_metadata(opts)

  

      # while we still hold the lock, notify others we processed their task

      batch.commit()

Instead of knowing the specifics of module repositories format,
let's outsource it to a createrepo_mod tool from modulemd-tools
package.

I am not sure whether backend/copr_backend/createrepo.py is now
useless or not but I am changing it anyway.

It works for me locally. I wanted to deploy it on dev instance and run beaker tests against it, but they are so slow that tests fail because of timeouts. I didn't have a chance to fix it, so I am submitting this PR at least as a preview.

Also, I don't understand the current relationship between createrepo.py file and copr-repo script. It seems to duplicate a lot of code from createrepo.py.

are so slow that tests fail because of timeouts

Turn off messaging. That's the slow down, as copr continously tries to send messages on stg bus, which is shut-down.

rebased onto 7efe7ba0cb96a44087928e04866f338e70f1ca51

3 years ago

Turn off messaging

Thanks, that was a good hint, it helped.

I actually had some bugs in that code, that beaker tests discovered. I reworked it and they are now passing.

If the ultimate goal is to move the functionality directly to createrepo_c, shouldn't we unconditionally use createrepo_mod now? What happens if we call it on non-modular repo?

That's an interesting point of view, I haven't thought about that.

Sounds good to me, but I am gonna need to do a small change in the createrepo_mod code, I think it won't work right now.

rebased onto 1fd22cd823f85ea212073e649ea14da08c071900

3 years ago

Metadata Update from @frostyx:
- Pull-request tagged with: blocked

3 years ago

I updated the code as requested, PTAL.

I really like the change, it basically just deletes stuff. Beaker tests are mostly passing (some failures but IMHO unrelated).

I've set a blocked tag to this PR because it requires modulemd-tools >= 0.4 which is not released yet, so CI builds are gonna fail until then. Let's wait with merging this until this week's Copr release is done.

Just FTR, I released new modulemd-tools

https://github.com/rpm-software-management/modulemd-tools/releases/tag/modulemd-tools-0.4-1

but it is going to take a while before it gets to stable Fedora repositories.

Metadata Update from @frostyx:
- Pull-request untagged with: blocked

3 years ago

Not blocked anymore, PTAL

rebased onto e460e9d

3 years ago

rebased onto e460e9d

3 years ago

Pull-Request has been merged by praiskup

3 years ago