#7258 OSBS fails to push to candidate registry.
Closed: Fixed 5 years ago Opened 5 years ago by cverna.

  • Describe what you need us to do:

We are experiencing issues with OSBS pushing images to the candidate-registry. Looking at the log the docker client is trying to use the /v1 APIs which are on purpose not allowed.

The logs we are seeing on OSBS is b'atomic_reactor.core.RetryGeneratorException: Failed to push image (\'candidate-registry.fedoraproject.org/f28/origin-base\',): \'Error: Status 403 trying to push repository f28/origin-base: "<!DOCTYPE HTML PUBLIC \\\\"-//IETF//DTD HTML 2.0//EN\\\\">\\\\n<html><head>\\\\n<title>403 Forbidden</title>\\\\n</head><body>\\\\n<h1>Forbidden</h1>\\\\n<p>You don\\\'t have permission to access /v1/repositories/f28/origin-base/\\\\non this server.<br />\\\\n</p>\\\\n<hr>\\\\n<address>Apache/2.4.34 (Fedora) Server at candidate-registry.fedoraproject.org Port 443</address>\\\\n</body></html>\\\\n"\'' b''

It would be useful to see which requests were made to the registry to understand why docker is trying to use the /v1 APIs.

Following are a few examples of failling builds :
https://koji.fedoraproject.org/koji/taskinfo?taskID=29795114
https://koji.fedoraproject.org/koji/taskinfo?taskID=29794338
https://koji.fedoraproject.org/koji/taskinfo?taskID=29792218
https://koji.fedoraproject.org/koji/taskinfo?taskID=29790819

  • When do you need this? (YYYY/MM/DD)
    The sooner the better
  • When is this no longer needed or useful? (YYYY/MM/DD)

  • If we cannot complete your request, what is the impact?


The problem (or at least one) seems to be:

/dev/mapper/GuestVolGroup01-srv 89G 89G 24K 100% /srv

So, how can we prune space there?

This ansible commit should enable deletes:
https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=398a6d8

So I have deleted the manifest for f25 and f26 images, I reckon that now we need to run the registry garbage collection (https://docs.docker.com/registry/garbage-collection/#run-garbage-collection).

We might want to have a playbook to perform all of these tasks.

yes, a playbook sounds good...

In any case I have run the GC and now:

/dev/mapper/GuestVolGroup01-srv 89G 60G 29G 68% /srv

So, hopefully this is solved?

Yes let's close it. Thanks for looking at this

Metadata Update from @cverna:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

5 years ago

:( it seems that something else is wrong while some build worked this one failed with the same issue

https://koji.fedoraproject.org/koji/taskinfo?taskID=29883396

Metadata Update from @cverna:
- Issue status updated to: Open (was: Closed)

5 years ago

Metadata Update from @cverna:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

5 years ago

Metadata Update from @cverna:
- Issue status updated to: Open (was: Closed)

5 years ago

Metadata Update from @cverna:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

5 years ago

Metadata Update from @cverna:
- Issue status updated to: Open (was: Closed)

5 years ago

Metadata Update from @cverna:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

5 years ago

Login to comment on this ticket.

Metadata