From https://koji.fedoraproject.org/koji/taskinfo?taskID=33797167
INFO - Calling: skopeo copy --dest-creds=<HIDDEN> oci:/tmp/tmph2n1zdpy/flatpak-oci-image:runtime/org.fedoraproject.Platform/x86_64/f30 docker://candidate-registry.fedoraproject.org/flatpak-runtime:f30-flatpak-candidate-90384-20190327204024-x86_64' ERROR - push failed with output:' b'b\'Getting image source signatures\\n\\r Copying blob 63dffad388ea: 0 B / 394.81 MiB 0s\\n\\x1b[1A\\r Copying blob 63dffad388ea: 0 B / 394.81 MiB 0s\\n time="2019-03-27T20:51:11Z" level=fatal msg="Error writing blob: Error initiating layer upload to /v2/flatpak-runtime/blobs/uploads/ in candidate-registry.fedoraproject.org: received unexpected HTTP status: 500 Internal Server Error" \\n\''
can we see if anything useful got logged with that 500?
I think we might have run out of disk space on the candidate registry.
So @puiterwijk had a look at the disk space it is was indeed full. I run a script to delete the manifest of the old F28 images but we now need to run a docker prune command to reclaim the space.
docker prune
Considering that we will have more builds with the flatpak now being built on OSBS it might be good to also increase the size of the disk on the registries.
Did the prune happen?
I'd certainly appreciate it if the registries had ample disk so we're not hitting this frequently - it's hard enough to keep the whole pipeline working!
ok. I ran a registry garbage-collect on it and it now has a lot of space.
I agree we should increase the space, or perhaps should we run garbage-collect more often? The docs are... scarce here, but we should figure out what is best for our case.
ok. I ran a registry garbage-collect on it and it now has a lot of space. I agree we should increase the space, or perhaps should we run garbage-collect more often? The docs are... scarce here, but we should figure out what is best for our case.
I ll try to get a script that deletes the manifests from images that are older than 1 month, I think we don't need to keep these on the candidate registry. Then we can have a weekly cron job running this script + garbage-collect.
Also I am not sure what is acceptable on the registry itself, is it ok to delete old images ? or should still keep f25 containers ?
Where should we track that work? A ticket in container sig?
Good question. Perhaps something to discuss on releng and/or container sig lists? I could see old containers being of use historically. Perhaps we should move eol ones to a registry-archive? but that might be too much work... or just keep the last image of each eol stream?
I ll try to get a script that deletes the manifests from images that are older than 1 month, I think we don't need to keep these on the candidate registry. Then we can have a weekly cron job running this script + garbage-collect. Where should we track that work? A ticket in container sig?
Created https://pagure.io/ContainerSIG/container-sig/issue/33
Also I am not sure what is acceptable on the registry itself, is it ok to delete old images ? or should still keep f25 containers ? Good question. Perhaps something to discuss on releng and/or container sig lists? I could see old containers being of use historically. Perhaps we should move eol ones to a registry-archive? but that might be too much work... or just keep the last image of each eol stream?
I can open a ticket in the container sig to discuss it and once we have an agreement check with releng if that sounds good.
Thanks and sounds perfect.
Closing this now...
:checkered_flag:
Metadata Update from @kevin: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Log in to comment on this ticket.