koji

Clone

#2286 Builds keeps failing with "Build already in progress (task 45068044)" even though 45068044 is failed and canceled
Closed: Invalid 3 years ago by tkopecek. Opened 3 years ago by mohanboddu.

Closed: Invalid
Reopen Issue

koji task 45074737 failed with GenericError: Build already in progress (task 45068044), but the koji task 45068044 has already failed with RetryError: unable to retry call 12 (method host.initBuild) for session 101943963. I tried canceling the 45068044 tasks and tried resubmitting, but it still failed with the same error, the koji task is https://koji.fedoraproject.org/koji/taskinfo?taskID=45075124.

When I tried resubmitting 45068044 and it created https://koji.fedoraproject.org/koji/taskinfo?taskID=45075267 koji task which failed with a traceback.

Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/koji/daemon.py", line 1294, in runTask
    response = (handler.run(),)
  File "/usr/lib/python3.7/site-packages/koji/tasks.py", line 311, in run
    self.createWorkdir()
  File "/usr/lib/python3.7/site-packages/koji/tasks.py", line 337, in createWorkdir
    os.makedirs(self.workdir)
  File "/usr/lib64/python3.7/os.py", line 211, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/lib64/python3.7/os.py", line 221, in makedirs
    mkdir(name, mode)
OSError: [Errno 30] Read-only file system: '/var/tmp/koji/tasks/5267'

Not sure whats going on.

The task may be failed, but the build itself is still in the BUILDING state, and that keeps the NVR from being recycled. This is a very unusual case. Koji is very careful to mark a build failed when the task itself fails, but there are corner cases where this itself can go wrong.

This is such a corner case. The original task failed on the initBuild call with a retry error. It appears that:

  • the first call to initBuild succeeded, but
  • the builder did not receive the result of the first call and likely timed out (note the duration of the task), and
  • the retry failed because the first call succeeded in making a change
  • because the builder thinks the initBuild call failed, it doesn't have a build entry that it can fail, so it doesn't. However, unbeknownst to it the first call succeeded and the build was created in the BUILDING state.

In 15 years of working on Koji, I've never seen this case.
Edited 3 years ago by mikem

Metadata Update from @mikem:
- Custom field Size adjusted to None
3 years ago

However, you should be able to work around this easily. Simply cancel the build. E.g.

$ koji cancel-build gnome-flashback-3.36.3-4.fc32

This can be done by the build owner @atim, or a Koji admin.

If this does not resolve the issue, then you can always simply bump the release.

This is such a strange corner case that I don't think we can reasonably fix it, but the workarounds should be easy enough.

However, you should be able to work around this easily. Simply cancel the build.

Tried many times, no luck.

If this does not resolve the issue, then you can always simply bump the release.

OK, i'll bump version and rebuild then. Thank you.

Tried many times, no luck.

What did the cancel-build command say?

koji cancel returned nothing:

$ koji cancel gnome-flashback-3.36.3-4.fc32
$

And a resubmit after that cancel worked
https://koji.fedoraproject.org/koji/taskinfo?taskID=45318747

What did the cancel-build command say?

Sorry, for some reason i am missing email from Pagure. Nothing, just silently run and nothing said. I'll already bumped and rebuilt package, but @mohanboddu also helped me there and seems like canceled old one build.

Thank you, probably safe to close now.

Metadata Update from @tkopecek:
- Issue close_status updated to: Invalid
- Issue status updated to: Closed (was: Open)
3 years ago

Login to comment on this ticket.

Metadata
None
None
None
None
Normal
None
None
Powered by Pagure 5.13.3
Documentation • File an Issue • About • SSH Hostkey/Fingerprint
© Red Hat, Inc. and others.