fedora-infrastructure

#12001 pulling from fedora container registry is really slow
Closed: Insufficient data by kevin. Opened by petersen.

Closed: Insufficient data
Reopen Issue

Recently, in the last few weeks at least, pulling images from the Fedora Container registry is really slow:

~$ podman pull fedora:40
Trying to pull registry.fedoraproject.org/fedora:40...
Getting image source signatures
Copying blob 4795e61369cf [======>---------------------------] 19.8MiB / 80.4MiB | 22.5 KiB/s

Downloading fedora-toolbox takes hours.

Any ideas? Can I provide more info to debug this?

Downloading from quay.io seems fine, but I think various tools, like fedora-review and toolbox, pull from registry.fp.o anyway.

I suppose a workaround is to pull first from quay.io:

$ podman pull quay.io/fedora/fedora:40
$ podman pull fedora:40

or a strong one might be to edit /etc/containers/registries.conf.d/000-shortnames.conf
Edited by petersen

We should introduce a redirect for toolbox containers as they are already on quay.io. Only containers that still are on registry.fedoraproject.org are flatpaks.

But there already a redirect for the main fedora container, presumably?
But it is still so slow? Maybe not working or ?

There isn't any redirect between the two (yet) but they have the same content on them.

I don't know why it would be any slower than normal recently.

Can you re-run with:

podman pull --log-level debug fedora:40 ?

 
~$ podman pull --log-level debug fedora:40
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called pull.PersistentPreRunE(podman pull --log-level debug fedora:40) 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
INFO[0000] Using boltdb as database backend             
DEBU[0000] Initializing boltdb state at /var/home/petersen/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/home/petersen/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /var/home/petersen/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /var/home/petersen/.local/share/containers/storage/volumes 
DEBU[0000] Using transient store: false                 
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that metacopy is not being used 
DEBU[0000] Cached value indicated that native-diff is usable 
DEBU[0000] backingFs=btrfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument 
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument 
DEBU[0000] Configured OCI runtime crun-vm initialization failed: no valid executable found for OCI runtime crun-vm: invalid argument 
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument 
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument 
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument 
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Setting parallel job count to 25             
DEBU[0000] Pulling image fedora:40 (policy: always)     
DEBU[0000] Looking up image "fedora:40" in local containers storage 
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/000-shortnames.conf" 
DEBU[0000] Trying "registry.fedoraproject.org/fedora:40" ... 
DEBU[0000] parsed reference into "[overlay@/var/home/petersen/.local/share/containers/storage+/run/user/1000/containers]@d28a0860ba79fde8db377237168e9df23e5793eb73e3ddb9e3dfea7c9e9ec456" 
DEBU[0000] Found image "fedora:40" as "registry.fedoraproject.org/fedora:40" in local containers storage 
DEBU[0000] Found image "fedora:40" as "registry.fedoraproject.org/fedora:40" in local containers storage ([overlay@/var/home/petersen/.local/share/containers/storage+/run/user/1000/containers]@d28a0860ba79fde8db377237168e9df23e5793eb73e3ddb9e3dfea7c9e9ec456) 
DEBU[0000] exporting opaque data as blob "sha256:d28a0860ba79fde8db377237168e9df23e5793eb73e3ddb9e3dfea7c9e9ec456" 
DEBU[0000] Image fedora:40 resolved to local image registry.fedoraproject.org/fedora:40 which will be used for pulling 
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Attempting to pull candidate registry.fedoraproject.org/fedora:40 for registry.fedoraproject.org/fedora:40 
DEBU[0000] parsed reference into "[overlay@/var/home/petersen/.local/share/containers/storage+/run/user/1000/containers]registry.fedoraproject.org/fedora:40" 
Trying to pull registry.fedoraproject.org/fedora:40...
DEBU[0000] Copying source image //registry.fedoraproject.org/fedora:40 to destination image [overlay@/var/home/petersen/.local/share/containers/storage+/run/user/1000/containers]registry.fedoraproject.org/fedora:40 
DEBU[0000] Using registries.d directory /etc/containers/registries.d 
DEBU[0000] Trying to access "registry.fedoraproject.org/fedora:40" 
DEBU[0000] No credentials matching registry.fedoraproject.org/fedora found in /run/user/1000/containers/auth.json 
DEBU[0000] No credentials matching registry.fedoraproject.org/fedora found in /var/home/petersen/.config/containers/auth.json 
DEBU[0000] No credentials matching registry.fedoraproject.org/fedora found in /var/home/petersen/.docker/config.json 
DEBU[0000] No credentials matching registry.fedoraproject.org/fedora found in /var/home/petersen/.dockercfg 
DEBU[0000] No credentials for registry.fedoraproject.org/fedora found 
DEBU[0000]  No signature storage configuration found for registry.fedoraproject.org/fedora:40, using built-in default file:///var/home/petersen/.local/share/containers/sigstore 
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.fedoraproject.org 
DEBU[0000] GET https://registry.fedoraproject.org/v2/   
DEBU[0001] Ping https://registry.fedoraproject.org/v2/ status 200 
DEBU[0001] GET https://registry.fedoraproject.org/v2/fedora/manifests/40 
DEBU[0001] Content-Type from manifest GET is "application/vnd.oci.image.index.v1+json" 
DEBU[0001] Using SQLite blob info cache at /var/home/petersen/.local/share/containers/cache/blob-info-cache-v1.sqlite 
DEBU[0001] Source is a manifest list; copying (only) instance sha256:039fd4ee7b1c61210df67982af9168c74d5c07c5275417d5ef878d1e245024aa for current system 
DEBU[0001] GET https://registry.fedoraproject.org/v2/fedora/manifests/sha256:039fd4ee7b1c61210df67982af9168c74d5c07c5275417d5ef878d1e245024aa 
DEBU[0002] Content-Type from manifest GET is "application/vnd.oci.image.manifest.v1+json" 
DEBU[0002] IsRunningImageAllowed for image docker:registry.fedoraproject.org/fedora:40 
DEBU[0002]  Using default policy section                
DEBU[0002]  Requirement 0: allowed                      
DEBU[0002] Overall: allowed                             
DEBU[0002] Downloading /v2/fedora/blobs/sha256:94a195eec409b6c342558376e8fc5970c03f4b27e6f9e6d61f0b9b37aa9dcdd4 
DEBU[0002] GET https://registry.fedoraproject.org/v2/fedora/blobs/sha256:94a195eec409b6c342558376e8fc5970c03f4b27e6f9e6d61f0b9b37aa9dcdd4 
Getting image source signatures
DEBU[0003] Reading /var/home/petersen/.local/share/containers/sigstore/fedora@sha256=039fd4ee7b1c61210df67982af9168c74d5c07c5275417d5ef878d1e245024aa/signature-1 
DEBU[0003] Not looking for sigstore attachments: disabled by configuration 
DEBU[0003] Manifest has MIME type application/vnd.oci.image.manifest.v1+json, ordered candidate list [application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.docker.distribution.manifest.v1+json] 
DEBU[0003] ... will first try using the original manifest unmodified 
DEBU[0003] Checking if we can reuse blob sha256:284f0c742e076c9b5a132c6c9f356faf00a387a4fc42d7ee4544df1920460e6c: general substitution = true, compression for MIME type "application/vnd.oci.image.layer.v1.tar+gzip" = true 
DEBU[0003] Failed to retrieve partial blob: convert_images not configured 
DEBU[0003] Downloading /v2/fedora/blobs/sha256:284f0c742e076c9b5a132c6c9f356faf00a387a4fc42d7ee4544df1920460e6c 
DEBU[0003] GET https://registry.fedoraproject.org/v2/fedora/blobs/sha256:284f0c742e076c9b5a132c6c9f356faf00a387a4fc42d7ee4544df1920460e6c 
Copying blob 284f0c742e07 [--------------------------------------] 0.0b / 80.4MiB | 0.0 b/s
DEBU[0004] Detected compression format gzip             
Copying blob 284f0c742e07 [--------------------------------------] 703.4KiB / 80.4MiB | 17.5 KiB/s

But is it really just me?

To put it in perspective, downloading the latest rawhide WS iso takes me like under 30s from a local SG mirror, but pulling fedora:41 from r.fp.o takes ~80min! In the past it took maybe 20-30s I think, maybe less: the base image is not very big.

Is the registry just getting hammered or ?
If it is a general problem, then I suggest it is kind of serious,
but if it is just me say, then I am not that worried.

Download speeds from dl.fedoraproject.org still seem about 20x faster than registry.fedoraproject.org.
Edited by petersen

Metadata Update from @zlopez:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: Needs investigation

I just checked the machine and the load on oci-registry01 is really low. So this is not what is causing the issue.

Can you run:

curl -o /dev/null --connect-to fedoraproject.org:443:proxy01.fedoraproject.org:443 https://fedoraproject.org/fmw/FedoraMediaWriter-win32-latest.ex

and see if it's 'fast' or 'slow'?

Any news here?

I guess lets close this and if you are still seeing it or see it again, please re-open and we can try and track it more...

Metadata Update from @kevin:
- Issue close_status updated to: Insufficient data
- Issue status updated to: Closed (was: Open)
Metadata
None

Needs investigation

None
None
Normal
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.