fedora-infrastructure

Clone
Source Code
GIT

#11176 Copr importing builds stuck (again)
Closed: Fixed a year ago by thl. Opened a year ago by thl.

Closed: Fixed
Reopen Issue

Hmm some tasks seem to be imported multiple times in parallel:

copr-di+  130440  0.1  0.2  91124  8008 ?        S    10:12   0:49 /usr/bin/python3 -sP /usr/bin/copr-distgit-process-import --daemon --build-id 5620608 --worker-id import_worker:5620608 [locked]
copr-di+  208578  0.4  0.2  91124 10456 ?        S    13:19   1:09 /usr/bin/python3 -sP /usr/bin/copr-distgit-process-import --daemon --build-id 5620607 --worker-id import_worker:5620607 [locked]
copr-di+  208638  0.4  0.2  91124 10400 ?        S    13:20   1:09 /usr/bin/python3 -sP /usr/bin/copr-distgit-process-import --daemon --build-id 5620616 --worker-id import_worker:5620616 [locked]
copr-di+  221157 23.0  0.2  91124 10732 ?        R    14:10  53:14 /usr/bin/python3 -sP /usr/bin/copr-distgit-process-import --daemon --build-id 5620616 --worker-id import_worker:5620616 [locking]
copr-di+  223719  0.2  0.3  91124 13408 ?        D    14:22   0:27 /usr/bin/python3 -sP /usr/bin/copr-distgit-process-import --daemon --build-id 5622294 --worker-id import_worker:5622294 [locked]
copr-di+  223864 19.9  0.2  91124 10848 ?        R    14:24  43:21 /usr/bin/python3 -sP /usr/bin/copr-distgit-process-import --daemon --build-id 5620610 --worker-id import_worker:5620610 [locking]
copr-di+  224412  0.4  0.2  91124 10884 ?        S    14:26   0:59 /usr/bin/python3 -sP /usr/bin/copr-distgit-process-import --daemon --build-id 5620610 --worker-id import_worker:5620610 [locked]
copr-di+  224551 20.6  0.2  91124 10768 ?        R    14:27  44:08 /usr/bin/python3 -sP /usr/bin/copr-distgit-process-import --daemon --build-id 5620610 --worker-id import_worker:5620610 [locking]
copr-di+  235295 18.2  0.2  91124 10744 ?        R    15:06  31:46 /usr/bin/python3 -sP /usr/bin/copr-distgit-process-import --daemon --build-id 5620610 --worker-id import_worker:5620610 [locking]
copr-di+  235300  0.6  0.2  91124 10880 ?        S    15:07   1:11 /usr/bin/python3 -sP /usr/bin/copr-distgit-process-import --daemon --build-id 5620613 --worker-id import_worker:5620613 [locked]
copr-di+  235742  0.7  0.2  91124 10976 ?        S    15:07   1:13 /usr/bin/python3 -sP /usr/bin/copr-distgit-process-import --daemon --build-id 5620615 --worker-id import_worker:5620615 [locked]
copr-di+  235996 16.1  0.2  91124 10800 ?        R    15:08  27:54 /usr/bin/python3 -sP /usr/bin/copr-distgit-process-import --daemon --build-id 5620610 --worker-id import_worker:5620610 [locking]
copr-di+  258313  5.8  0.4  91124 16596 ?        R    16:16   6:08 /usr/bin/python3 -sP /usr/bin/copr-distgit-process-import --daemon --build-id 5620610 --worker-id import_worker:5620610 [locking]
copr-di+  258340  1.1  0.5  91124 22744 ?        S    16:16   1:14 /usr/bin/python3 -sP /usr/bin/copr-distgit-process-import --daemon --build-id 5620614 --worker-id import_worker:5620614 [locked]
copr-di+  259463  1.0  0.2  91124 11340 ?        D    16:20   1:02 /usr/bin/python3 -sP /usr/bin/copr-distgit-process-import --daemon --build-id 5620614 --worker-id import_worker:5620614
copr-di+  259663  0.3  0.2  91124 10952 ?        S    16:22   0:23 /usr/bin/python3 -sP /usr/bin/copr-distgit-process-import --daemon --build-id 5622291 --worker-id import_worker:5622291 [locked]
copr-di+  259668  1.1  0.4  91124 16568 ?        D    16:22   1:09 /usr/bin/python3 -sP /usr/bin/copr-distgit-process-import --daemon --build-id 5620617 --worker-id import_worker:5620617 [locked]
copr-di+  259680  1.5  0.4  91252 16460 ?        R    16:22   1:33 /usr/bin/python3 -sP /usr/bin/copr-distgit-process-import --daemon --build-id 5620615 --worker-id import_worker:5620615 [locking]
copr-di+  261705  0.0  0.3  91124 15484 ?        S    16:29   0:05 /usr/bin/python3 -sP /usr/bin/copr-distgit-process-import --daemon --build-id 5624945 --worker-id import_worker:5624945 [locked]
copr-di+  262701 19.1  0.2  91124 10976 ?        R    16:33  16:53 /usr/bin/python3 -sP /usr/bin/copr-distgit-process-import --daemon --build-id 5624945 --worker-id import_worker:5624945 [locking]
copr-di+  267665  0.1  0.8  91124 33888 ?        D    17:11   0:04 /usr/bin/python3 -sP /usr/bin/copr-distgit-process-import --daemon --build-id 5625686 --worker-id import_worker:5625686
copr-di+  267974 10.1  0.5  91124 20656 ?        S    17:12   4:57 /usr/bin/python3 -sP /usr/bin/copr-distgit-process-import --daemon --build-id 5625564 --worker-id import_worker:5625564 [locked]
copr-di+  268437 11.4  1.0  91124 40064 ?        S    17:15   5:18 /usr/bin/python3 -sP /usr/bin/copr-distgit-process-import --daemon --build-id 5625686 --worker-id import_worker:5625686 [locked]
copr-di+  268981 14.8  1.4  91124 56724 ?        R    17:21   5:51 /usr/bin/python3 -sP /usr/bin/copr-distgit-process-import --daemon --build-id 5625686 --worker-id import_worker:5625686 [locking]
copr-di+  271034 12.5  1.5  91124 61740 ?        R    17:38   2:55 /usr/bin/python3 -sP /usr/bin/copr-distgit-process-import --daemon --build-id 5625728 --worker-id import_worker:5625728 [locking]
copr-di+  273327  0.0  2.2 147200 89864 ?        D    17:57   0:00 /usr/bin/python3 -sP /usr/bin/copr-distgit-process-import --daemon --build-id 5625730 --worker-id import_worker:5625730
copr-di+  273776  0.0  1.5  99072 60368 ?        D    18:00   0:00 /usr/bin/python3 -sP /usr/bin/copr-distgit-process-import --daemon --build-id 5625731 --worker-id import_worker:5625731

re-reported here for our team: https://github.com/fedora-copr/copr/issues/2571

Metadata Update from @praiskup:
- Issue tagged with: copr
a year ago

The issue should be worked-around. Some unfortunate peak happened today, some issues were identified - we need to get prepared for the next time. Thank you again for reporting.

Thank you again for reporting.

Thank you for looking into this!

The issue should be worked-around.

Great. I still have trouble with two builds, imports fail for some reason. Is there maybe something jammed in the git repos they are trying to import to?

https://copr.fedorainfracloud.org/coprs/g/kernel-vanilla/next/build/5627638/
https://copr.fedorainfracloud.org/coprs/g/kernel-vanilla/mainline-wo-mergew/build/5627645/

From the first:

IGNORING: Package module @kernel-vanilla/next/next-next-all already has a branch master
Branch already exists...continuing
repo_dir: /tmp/tmpjprw8v3d
clone the pkg repository into repo_dir directory
Cloning /var/lib/dist-git/git/@kernel-vanilla/next/next-next-all
Cloning into: /tmp/tmpjprw8v3d
Running: git clone /var/lib/dist-git/git/@kernel-vanilla/next/next-next-all --origin origin /tmp/tmpjprw8v3d
Failed to clone the Git repository and add files.
Releasing lock "/var/lib/dist-git/cache/lookaside/pkgs/@kernel-vanilla/next/next-next-all/import.lock"
Exception raised during package import.
Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/copr_dist_git/package_import.py", line 193, in import_package
    commands.clone(reponame, target=repo_dir)
  File "/usr/lib/python3.11/site-packages/pyrpkg/__init__.py", line 1619, in clone
    self._run_command(cmd, cwd=path)
  File "/usr/lib/python3.11/site-packages/pyrpkg/__init__.py", line 1297, in _run_command
    raise rpkgError('Failed to execute command.')
pyrpkg.errors.rpkgError: Failed to execute command.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/copr_dist_git/importer.py", line 87, in do_import
    result.update(import_package(
                  ^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/copr_dist_git/package_import.py", line 196, in import_package
    raise PackageImportException(str(e))
copr_dist_git.exceptions.PackageImportException: Failed to execute command.
sending a response for task {'build_id': 5627638}
Sending back: 
{"build_id": 5627638}
Starting new HTTPS connection (1): copr.fedorainfracloud.org:443
https://copr.fedorainfracloud.org:443 "POST /backend/import-completed/ HTTP/1.1" 200 17

The other has lots of git errors like

error: object file ./objects/f8/e9655f58e99ae2107518d2c1763fd92f29af4a is empty

I moved those git repos out from the path; and you should be able to re-import.
Can you please retry?

I moved those git repos out from the path; and you should be able to re-import.
Can you please retry?

Many thx for this. That seems to have helped in one of the two cases I had.

The other case fails differently now. Importing now succeeds, but all builders fail just seconds after they started. See https://copr.fedorainfracloud.org/coprs/g/kernel-vanilla/next/build/5629665/

To quote from the logs of one:

ERROR: Check-sum d41d8cd98f00b204e9800998ecf8427e is wrong, expected: 826278cc7723bfe175562301122da2c3


Traceback (most recent call last):
  File "/usr/bin/copr-rpmbuild", line 262, in build_rpm
    distgit.produce_sources()
  File "/usr/lib/python3.11/site-packages/copr_rpmbuild/providers/distgit.py", line 36, in produce_sources
    helpers.run_cmd(["copr-distgit-client", "sources"], cwd=self.clone_to)
  File "/usr/lib/python3.11/site-packages/copr_rpmbuild/helpers.py", line 76, in run_cmd
    raise RuntimeError(result.stderr)
RuntimeError: INFO: Reading stdout from command: git rev-parse --abbrev-ref HEAD

This is very weird. Seems like the rpkg import feature failed weirdly. Try:

$ git clone https://copr-dist-git.fedorainfracloud.org/git/@kernel-vanilla/next/next-next-all
$ cd next-next-all
$ copr-dist-git
...
ERROR: Check-sum d41d8cd98f00b204e9800998ecf8427e is wrong, expected: 826278cc7723bfe175562301122da2c3
$ grep 826278cc7723bfe175562301122da2c3 sources
826278cc7723bfe175562301122da2c3  linux-next-20230310.tar.xz
$ md5sum linux-next-20230310.tar.xz
d41d8cd98f00b204e9800998ecf8427e  linux-next-20230310.tar.xz

This seems like a separate bug in Copr or python-rpkg we should take
a look at. Is this reproducible?

Note the weird sources file contents, several artifacts have the same checksum d41d8cd98f00b204e9800998ecf8427e.

Is this reproducible?

seems so, but whatever, let's wait till tomorrow, then I'll likely build a newer upstream snapshot anyway, maybe things will sort themselves out on their own…

thx for your help so far!

Metadata Update from @thl:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)
a year ago

Is this reproducible?

seems so, but whatever, let's wait till tomorrow, then I'll likely build a newer upstream snapshot anyway, maybe things will sort themselves out on their own…

they did :-D

Hm. That problem with the wrong hash was so weird. If you reproduced
the issue in the future, please let us know.

Thank you for closing the issue, we track the rest against copr.

Login to comment on this ticket.

Metadata
None

copr

None
None
Needs Review
Powered by Pagure 5.13.3
Documentation • File an Issue • About • SSH Hostkey/Fingerprint
© Red Hat, Inc. and others.