#9177 Pulling container images breaks randomly (registry problem?)
Opened 7 days ago by hhorak. Modified 2 days ago

Describe what you would like us to do:


I'd like to see the Fedora container repository to behave deterministically. What I see are random failures (using podman 1.6.x on CentOS7, not sure whether older version of a client can cause the troubles). What I see is that pulling images fails in some cases, and succeeds few seconds after the failure. It's very undeterministic:

[root@centos7]# podman pull registry.fedoraproject.org/f31/s2i-base:latest
Trying to pull registry.fedoraproject.org/f31/s2i-base:latest...
Getting image source signatures
Copying blob 1457171663a9 done  
Copying blob 64f48977383f done  
Copying blob 82422234ceca done  
Copying config 95c2bf26d4 done  
Writing manifest to image destination
Storing signatures
95c2bf26d49eba2049318841605a616e5580bd69aa7bab4fedd62fb25e908cdb

[root@centos7]# podman pull registry.fedoraproject.org/f32/s2i-base:latest
Trying to pull registry.fedoraproject.org/f32/s2i-base:latest...
Getting image source signatures
Copying blob 44e1bb1e1824 [--------------------------------------] 0.0b / 69.8MiB
Copying blob d6e069648152 [--------------------------------------] 8.2KiB / 16.4MiB
  unexpected EOF
Error: error pulling image "registry.fedoraproject.org/f32/s2i-base:latest": unable to pull registry.fedoraproject.org/f32/s2i-base:latest: unable to pull image: Error writing blob: error setting up to decompress blob: unexpected EOF

[root@centos7]# podman pull registry.fedoraproject.org/f32/s2i-base:latest
Trying to pull registry.fedoraproject.org/f32/s2i-base:latest...
Getting image source signatures
Copying blob d6e069648152 done  
Copying blob 192e0a29e681 [======================================] 177.4MiB / 177.4MiB
Copying blob 44e1bb1e1824 done  
Copying config 6be317b4e3 [======================================] 5.8KiB / 5.8KiB
Writing manifest to image destination
Storing signatures
6be317b4e3d03498c9805598bec11d06ee3b1577d5ff1fd3702a3c34c885f4be

[root@centos7]# podman pull registry.fedoraproject.org/f32/s2i-base:latest
Trying to pull registry.fedoraproject.org/f32/s2i-base:latest...
Getting image source signatures
Copying blob 192e0a29e681 skipped: already exists  
Copying blob 44e1bb1e1824 skipped: already exists  
Copying blob d6e069648152 [--------------------------------------] 0.0b / 0.0b
Copying config 6be317b4e3 done  
Writing manifest to image destination
Storing signatures
6be317b4e3d03498c9805598bec11d06ee3b1577d5ff1fd3702a3c34c885f4be

[root@centos7]# podman pull registry.fedoraproject.org/f33/s2i-base:latest
Trying to pull registry.fedoraproject.org/f33/s2i-base:latest...
Getting image source signatures
Copying blob df6804b6febf [>-------------------------------------] 4.0MiB / 164.4MiB
Copying blob f35b398c6aac [======================================] 70.1MiB / 70.1MiB
Copying blob 12542a84343f [======================>---------------] 4.4MiB / 7.2MiB
  unexpected EOF
Error: error pulling image "registry.fedoraproject.org/f33/s2i-base:latest": unable to pull registry.fedoraproject.org/f33/s2i-base:latest: unable to pull image: Error writing blob: error storing blob to file "/var/tmp/storage191954987/3": unexpected EOF

[root@centos7]# podman pull registry.fedoraproject.org/f33/s2i-base:latest
Trying to pull registry.fedoraproject.org/f33/s2i-base:latest...
Getting image source signatures
  unexpected EOF
Error: error pulling image "registry.fedoraproject.org/f33/s2i-base:latest": unable to pull registry.fedoraproject.org/f33/s2i-base:latest: unable to pull image: Error writing blob: error storing blob to file "/var/tmp/storage931605175/1": unexpected EOF

[root@centos7]# podman pull registry.fedoraproject.org/f33/s2i-base:latest
Trying to pull registry.fedoraproject.org/f33/s2i-base:latest...
Getting image source signatures
Copying blob f35b398c6aac done  
Copying blob 12542a84343f done  
Copying blob df6804b6febf done  
Copying config 390f2192d5 [======================================] 5.8KiB / 5.8KiB
Writing manifest to image destination
Storing signatures
390f2192d5c067c7c9e59a60e1b49d20054dcae7af2975d6f157c99240a1941e

When do you need this to be done by? (no specific deadline)



I have similar problem:

14:25:12 -> Pulling image registry.fedoraproject.org/f31/s2i-base:latest before building image from Dockerfile.fedora.
14:25:12 Trying to pull repository registry.fedoraproject.org/f31/s2i-base ... 
14:25:12 Pulling repository registry.fedoraproject.org/f31/s2i-base
14:25:12 Error: image f31/s2i-base:latest not found
14:25:12 Pulling image registry.fedoraproject.org/f31/s2i-base:latest failed. Let's wait 2 seconds and try one more time.
14:25:14 Trying to pull repository registry.fedoraproject.org/f31/s2i-base ... 
14:25:15 Pulling repository registry.fedoraproject.org/f31/s2i-base
14:25:15 Error: image f31/s2i-base:latest not found
14:25:15 make[1]: *** [3.7] Error 1
14:25:15 make[1]: Leaving directory `/root/sources'
14:25:15 make: *** [build-serial] Error 2

Metadata Update from @mohanboddu:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: groomed, high-gain, medium-trouble

7 days ago

Can you all re-run with '--log-level=debug' and attach the output? Might help us track this down. Thanks!

ok. I think I found at least part of the issue. One of our pair of registry hosts was throwing "too many open flies" errors on some operations.

Can you all look and see if you see any issues now?

Login to comment on this ticket.

Metadata