It seems that the BZ#1737471 is not really a Go stdlib bug, but it is triggered by bug in tls stack of the proxy/cdn/cache(possibly openssl) of registry.fp.o as pointed out in the upstream issue.
Opening this issue to track it so it gets fixed when fix lands in Fedora.
ok. Currently our proxies are Fedora 29 instances, running the current updated openssl in Fedora 29.
After Fedora 31 is released we plan to move them up to that.
Is it worth keeping this open here? Or can you just open/reopen it when there's a fix thats landed in Fedora?
Metadata Update from @kevin: - Issue priority set to: Waiting on Assignee (was: Needs Review)
There are a few bodhi updates that attempt to fix this problem, but they are against f30 and f31.
Can we use those by any chance?
No, but we could use the f29 one. ;) https://bodhi.fedoraproject.org/updates/FEDORA-2019-f5810ab475
Will need a freeze break.
Can we duplicate this problem in staging? can we do so and then apply the update to confirm it fixes it?
Yes we should be able to use staging to test that update.
now that the beta is a GO we are going to get heat from this if people can't use the registry. Yes they can use other registries OR set an env var to workaround, but would be nice if the user didn't hit a bug as soon as they started using f31 beta.
Hum. So, I checked this issue against our staging server(s):
podman pull registry.stg.fedoraproject.org/fedora:30 Trying to pull registry.stg.fedoraproject.org/fedora:30... Get https://registry.stg.fedoraproject.org/v2/: local error: tls: unexpected message Error: error pulling image "registry.stg.fedoraproject.org/fedora:30": unable to pull registry.stg.fedoraproject.org/fedora:30: unable to pull image: Error initializing source docker://registry.stg.fedoraproject.org/fedora:30: pinging docker registry returned: Get https://registry.stg.fedoraproject.org/v2/: local error: tls: unexpected message
then I upgraded openssl on the proxy server: [root@proxy01 ~][STG]# rpm -q openssl openssl-1.1.1c-6.fc29.x86_64
Rebooted the server.
But the problem persists: % podman pull registry.stg.fedoraproject.org/fedora:30 Trying to pull registry.stg.fedoraproject.org/fedora:30... Get https://registry.stg.fedoraproject.org/v2/: local error: tls: unexpected message Error: error pulling image "registry.stg.fedoraproject.org/fedora:30": unable to pull registry.stg.fedoraproject.org/fedora:30: unable to pull image: Error initializing source docker://registry.stg.fedoraproject.org/fedora:30: pinging docker registry returned: Get https://registry.stg.fedoraproject.org/v2/: local error: tls: unexpected message
What am I missing here? httpd doesn't need to be rebuilt does it?
Hum. So, I checked this issue against our staging server(s): podman pull registry.stg.fedoraproject.org/fedora:30 Trying to pull registry.stg.fedoraproject.org/fedora:30... Get https://registry.stg.fedoraproject.org/v2/: local error: tls: unexpected message Error: error pulling image "registry.stg.fedoraproject.org/fedora:30": unable to pull registry.stg.fedoraproject.org/fedora:30: unable to pull image: Error initializing source docker://registry.stg.fedoraproject.org/fedora:30: pinging docker registry returned: Get https://registry.stg.fedoraproject.org/v2/: local error: tls: unexpected message then I upgraded openssl on the proxy server: [root@proxy01 ~][STG]# rpm -q openssl openssl-1.1.1c-6.fc29.x86_64 Rebooted the server. But the problem persists: % podman pull registry.stg.fedoraproject.org/fedora:30 Trying to pull registry.stg.fedoraproject.org/fedora:30... Get https://registry.stg.fedoraproject.org/v2/: local error: tls: unexpected message Error: error pulling image "registry.stg.fedoraproject.org/fedora:30": unable to pull registry.stg.fedoraproject.org/fedora:30: unable to pull image: Error initializing source docker://registry.stg.fedoraproject.org/fedora:30: pinging docker registry returned: Get https://registry.stg.fedoraproject.org/v2/: local error: tls: unexpected message What am I missing here? httpd doesn't need to be rebuilt does it?
Yes, might need the mod_ssl package to be rebuilt using the latest version on openssl
Eagerly waiting for all kinds of updates needed on my production Fedora 31 Silverblue system to get toolbox working. Without that I can't do my development work.
tested the corrected update and it does indeed fix things in stg. Sent a FBR in.
Production now fixed.
➜ ~ podman pull registry.fedoraproject.org/fedora:30 WARN[0000] The configuration is using `runtime_path`, which is deprecated and will be removed in future. Please use `runtimes` and `runtime` WARN[0000] If you are using both `runtime_path` and `runtime`, the configuration from `runtime_path` is used Trying to pull registry.fedoraproject.org/fedora:30... Getting image source signatures Copying blob ed60cb1abc2e done Copying config 02781e9f50 done Writing manifest to image destination Storing signatures 02781e9f507f260dded796ec8784c36f9067a3d3739eb9e237afcd2735345b81
:panda_face:
Metadata Update from @kevin: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
thanks @kevin!
I confirm I can pull container images without any issues. Thank you Kevin. :thumbsup:
Login to comment on this ticket.