Learn more about these different git repos.
Other Git URLs
I have noticed recently that some of the failed builds show errors 404 in their results directories. I decided to ignore it and hit rebuild.
However, I now see that even successful builds are gone.
Couple hours ago, the result of this was 1898 packages, now it is 415.
repoquery --refresh --repo=python39 --source --whatrequires 'libpython3.9.so.1.0()(64bit)'; repoquery --repo=python39 --source --whatrequires 'python(abi) = 3.9') | pkgname | sort | uniq > python39.pkgs
See for example https://copr.fedorainfracloud.org/coprs/g/python/python3.9/build/1141920/ - this is a successful build but setuptools are no longer in the repo:
https://copr-be.cloud.fedoraproject.org/results/@python/python3.9/fedora-rawhide-x86_64/01141920-python-setuptools/ 404 Not Found
E.g. https://copr.fedorainfracloud.org/coprs/g/python/python3.9/package/python-setuptools/
seems like multiple packages have the same NVR, might that be that prunerepo removed some of them by random?
seems like multiple packages have the same NVR
But all three are gone.
This sounds scary, I did chmod -x /usr/bin/prunerepo now till I realize what is going on here.
chmod -x /usr/bin/prunerepo
Note that if the builds are really gone and not just "unavailable" we should stop the automatic rebuilds in this repo, as literally nothing can build without setuptools and we cannot just rebuild them again, because there is a bootstrap sequence.
Yes, there's no python-setuptools build on backend. I'm trying to find why :-(
Last not from prunerepo is:
Removing: /var/lib/copr/public_html/results/@python/python3.9/fedora-rawhide-x86_64/01138091-python-setuptools/python-setuptools-41.6.0-2.fc32~bootstrap.src.rpm Removing: /var/lib/copr/public_html/results/@python/python3.9/fedora-rawhide-x86_64/01138091-python-setuptools/python3-setuptools-41.6.0-2.fc32~bootstrap.noarch.rpm
so prunerepo doesn't seem to be guilty.
Is there a backup of this, or shall we scope a rebootstrap?
Meh, sorry, this needs rebootstrap.
Did you remove any builds manually today from @python/python3.9?
.. I found this: https://copr.fedorainfracloud.org/backend/action/392860/ ... the action might be misgenerated (empty "subdir" in fedora-rawhide-x86_64)
There are two bugs which together caused this, one that the action is badly generated (frontend) and one in the new copr-repo tool which allowed os.path.join with empty string (backend), so the tool attempted to remove whole <user>/<project>/<chroot> content actually. I applied guard against this (raise exception if --delete '' is called) so this will not repeat again ...
--delete ''
+ if not subdir or '..' in subdir: + raise Exception("removal of \"%s\" subdir requested", subdir)
.. and proper fix hopefully tomorrow.
Unfortunately, there are multiple projects/chroots affected by this problem already:
From grep -- "--excludes '\*/\*" modifyrepo.log
grep -- "--excludes '\*/\*" modifyrepo.log
iucar/cran/fedora-rawhide-x86_64 kloxong/Testing/epel-8-x86_64 kloxong/Testing/epel-7-x86_64 kloxong/Testing/epel-6-x86_64 kloxong/Testing/epel-8-x86_64 kloxong/Testing/epel-7-x86_64 kloxong/Testing/epel-6-x86_64 nmstate/nm-build-deps/epel-8-ppc64le nmstate/nm-build-deps/epel-8-x86_64 networkmanager/NetworkManager-master/epel-8-x86_64 networkmanager/NetworkManager-master/fedora-31-x86_64 networkmanager/NetworkManager-master/fedora-30-x86_64 networkmanager/NetworkManager-master/fedora-29-x86_64 networkmanager/NetworkManager-master/fedora-rawhide-x86_64 networkmanager/NetworkManager-master/epel-7-x86_64 xxmitsu/fedora-gnome/fedora-31-x86_64 xxmitsu/fedora-gnome/fedora-rawhide-x86_64 @python/python3.9/fedora-rawhide-x86_64
Metadata Update from @praiskup: - Issue priority set to: High - Issue tagged with: bug
I've tried rebuilding some package from xxmitsu/fedora-gnome/ : https://copr.fedorainfracloud.org/coprs/xxmitsu/fedora-gnome/build/1152759/
The build fails instantly. When I try to access chroot build report/logs, I get a 403 Forbidden. And if I access https://copr-be.cloud.fedoraproject.org/results/xxmitsu/fedora-gnome/ there still seems to be empty directories with nothing created by the build that I've just triggered.
Can you also investigate if there isn't some other issue causing this? Or do I need to delete the project altogether and re-create it from scratch with all the packages ? That would be really cumbersome and I would appreciate if that would not be required.
Thank you in advance, Mike.
I don't recall, but I might have.
We are at 415 packages, so no more builds deleted during the night from @python/python3.9.
@xxmitsu, I don't know why (yet) but the chroot directories had wrong permissions (700 instead of 755). I fixed that, so you should be able to build now ...
@praiskup, I can confirm it works now. Thank you!
Cca 8 hours ago, there were 963 uniq source packages with built packages requiring Python 3.9 in our copr. Couple hundred builds finished since. Now there is 945. Something is getting deleted again.
Ignore this comment, I have failed to notice a temporary problem getting the repo metadata.
Metadata Update from @praiskup: - Issue assigned to praiskup
Commit 1ba7d22 relates to this ticket
Commit ed37f42 relates to this ticket
Commit e92bfbc fixes this issue
Login to comment on this ticket.