We are experiencing issues with OSBS pushing images to the candidate-registry. Looking at the log the docker client is trying to use the /v1 APIs which are on purpose not allowed.
The logs we are seeing on OSBS is b'atomic_reactor.core.RetryGeneratorException: Failed to push image (\'candidate-registry.fedoraproject.org/f28/origin-base\',): \'Error: Status 403 trying to push repository f28/origin-base: "<!DOCTYPE HTML PUBLIC \\\\"-//IETF//DTD HTML 2.0//EN\\\\">\\\\n<html><head>\\\\n<title>403 Forbidden</title>\\\\n</head><body>\\\\n<h1>Forbidden</h1>\\\\n<p>You don\\\'t have permission to access /v1/repositories/f28/origin-base/\\\\non this server.<br />\\\\n</p>\\\\n<hr>\\\\n<address>Apache/2.4.34 (Fedora) Server at candidate-registry.fedoraproject.org Port 443</address>\\\\n</body></html>\\\\n"\'' b''
b'atomic_reactor.core.RetryGeneratorException: Failed to push image (\'candidate-registry.fedoraproject.org/f28/origin-base\',): \'Error: Status 403 trying to push repository f28/origin-base: "<!DOCTYPE HTML PUBLIC \\\\"-//IETF//DTD HTML 2.0//EN\\\\">\\\\n<html><head>\\\\n<title>403 Forbidden</title>\\\\n</head><body>\\\\n<h1>Forbidden</h1>\\\\n<p>You don\\\'t have permission to access /v1/repositories/f28/origin-base/\\\\non this server.<br />\\\\n</p>\\\\n<hr>\\\\n<address>Apache/2.4.34 (Fedora) Server at candidate-registry.fedoraproject.org Port 443</address>\\\\n</body></html>\\\\n"\'' b''
It would be useful to see which requests were made to the registry to understand why docker is trying to use the /v1 APIs.
Following are a few examples of failling builds : https://koji.fedoraproject.org/koji/taskinfo?taskID=29795114 https://koji.fedoraproject.org/koji/taskinfo?taskID=29794338 https://koji.fedoraproject.org/koji/taskinfo?taskID=29792218 https://koji.fedoraproject.org/koji/taskinfo?taskID=29790819
When is this no longer needed or useful? (YYYY/MM/DD)
If we cannot complete your request, what is the impact?
The problem (or at least one) seems to be:
/dev/mapper/GuestVolGroup01-srv 89G 89G 24K 100% /srv
So, how can we prune space there?
This ansible commit should enable deletes: https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=398a6d8
So I have deleted the manifest for f25 and f26 images, I reckon that now we need to run the registry garbage collection (https://docs.docker.com/registry/garbage-collection/#run-garbage-collection).
We might want to have a playbook to perform all of these tasks.
yes, a playbook sounds good...
In any case I have run the GC and now:
/dev/mapper/GuestVolGroup01-srv 89G 60G 29G 68% /srv
So, hopefully this is solved?
Yes let's close it. Thanks for looking at this
Metadata Update from @cverna: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
:( it seems that something else is wrong while some build worked this one failed with the same issue
https://koji.fedoraproject.org/koji/taskinfo?taskID=29883396
Metadata Update from @cverna: - Issue status updated to: Open (was: Closed)
This was fixed by https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/thread/7IGK2QIBT3JU3GD77UDDM5VZ4TWKHNQ4/
Thanks @puiterwijk
Login to comment on this ticket.