note skopeo gives us a bit more information for registry.fedoraproject.org/fedora:latest

[root@localhost ~]# skopeo copy docker://registry.fedoraproject.org/fedora:latest dir:/tmp/boo/
FATA[0001] Error choosing an image from manifest list docker://registry.fedoraproject.org/fedora:latest: no image found in manifest list for architecture arm64, OS linux

I have opened https://pagure.io/releng/pull-request/7773 to fix this.

you asked me for this today.. here is the output. I just ran the command:

[root@localhost ~]# skopeo copy docker://registry.fedoraproject.org/fedora:rawhide dir:/tmp/coo/
FATA[0001] Error choosing an image from manifest list docker://registry.fedoraproject.org/fedora:rawhide: no image found in manifest list for architecture arm64, OS linux

Well, doing a lookup on the registry, I am only seeing a raw manifest, and not a manifest list, for fedora:rawhide: "curl https://registry.fedoraproject.org/v2/fedora/manifests/rawhide", which only has the amd64 architecture?
i.e. it doesn't look like the multi-arch manifest list is being pushed?
I do see a bunch of tags like "28-armhfp" and "28-ppc64le" in the tag list, but still no manifest lists.

Well, doing a lookup on the registry, I am only seeing a raw manifest, and not a manifest list, for fedora:rawhide: "curl https://registry.fedoraproject.org/v2/fedora/manifests/rawhide", which only has the amd64 architecture?
i.e. it doesn't look like the multi-arch manifest list is being pushed?
I do see a bunch of tags like "28-armhfp" and "28-ppc64le" in the tag list, but still no manifest lists.

You need to pass the -H "Accept: application/vnd.docker.distribution.manifest.list.v2+json" to the curl command to get the manifest list

Ah. Thanks.
So, then I notice that your PR made things get pushed as "arm64v8", while Dusty's error says it can't find "arm64", which to me indeed don't seem like the same string at all?

Ah. Thanks.
So, then I notice that your PR made things get pushed as "arm64v8", while Dusty's error says it can't find "arm64", which to me indeed don't seem like the same string at all?

It's possible the client (podman) is to blame. Does anyone know what the value should be and then we can determine if its the server or the client that is at fault?

I used https://github.com/docker-library/official-images/blob/master/library/fedora#L13 as an example.
I could not find a proper documentation on what the registry expect. I fine with testing "arm64" instead of "arm64v8". It would be nice to know if there are more info on the registry logs.

I believe most of Docker uses the GOARCH terms, which are: https://golang.org/doc/install/source#environment .
So I guess that podman defaults to that too.
However, the Docker official-images program uses arm64v8: https://github.com/docker-library/official-images#architectures-other-than-amd64 .

@cverna I just see a 200 against the manifest list, and then a 200 against the actual manifest, and that's it.
All of the decisions for which arches to support or pull are on the client-side.

(@cverna, note that this is not meant as a "you can't see the logs", I'd be totally fine with that, just giving you the info you asked for until someone takes care of your ticket for that access)

Let give "arm64" a try then :smile:

(@cverna, note that this is not meant as a "you can't see the logs", I'd be totally fine with that, just giving you the info you asked for until someone takes care of your ticket for that access)

Sure thanks for helping :)

OK so that the manifest list from the docker hub

{
'schemaVersion': 2, 
'mediaType': 'application/vnd.docker.distribution.manifest.list.v2+json', 
'manifests': [
{'mediaType': 'application/vnd.docker.distribution.manifest.v2+json', 'size': 529, 'digest': 'sha256:6
fb84ba634fe68572a2ac99741062695db24b921d0aa72e61ee669902f88c187', 'platform': {'architecture': 'amd64', 'os': 'linux'}}, 
{'mediaType': 'application/vnd.docker.distribution.manifest.v2+json', 'size': 529, 'digest
': 'sha256:dadbe43c2d90b68cd9e9ba7bdd9907f32842f3b93bf66d8f22e1fd12ac5cefcf', 'platform': {'architecture': 'arm', 'os': 'linux', 'variant': 'v7'}}, 
{'mediaType': 'application/vnd.docker.distribution.manifest.v2+
json', 'size': 529, 'digest': 'sha256:a2adc1621935869bcd51a84b007b3ee6aa8aa9aef0bf30a947f7312fd43e5b0b', 'platform': {'architecture': 'arm64', 'os': 'linux', 'variant': 'v8'}}, 
{'mediaType': 'application/vnd.doc
ker.distribution.manifest.v2+json', 'size': 529, 'digest': 'sha256:74026e25854714e616f46182b4f41a1b25a90a76b484e30142d073bec6b861df', 'platform': {'architecture': 'ppc64le', 'os': 'linux'}}]}

So it looks like arm and arm64 is the way to go

So now it is picking the correct image but there is an issue with the certificates.

sudo docker -D pull registry.fedoraproject.org/fedora:rawhide
Trying to pull repository registry.fedoraproject.org/fedora ...
sha256:d1df13939e52b24fef32271b8f1543e90c38a1a47e84c859db45d437e463d0f3: Pulling from registry.fedoraproject.org/fedora                                                                                           
0fe73490aedd: Pulling fs layer
error pulling image configuration: Get https://cdn.registry.fedoraproject.org/v2/fedora/blobs/sha256:d00f2170d181b5605ef7b165a88be0a67200dd8f2878811df76572338d86f974: x509: certificate has expired or is not yet
valid

The same command works just fine on x86_64.

So now it is picking the correct image but there is an issue with the certificates.
sudo docker -D pull registry.fedoraproject.org/fedora:rawhide
Trying to pull repository registry.fedoraproject.org/fedora ...
sha256:d1df13939e52b24fef32271b8f1543e90c38a1a47e84c859db45d437e463d0f3: Pulling from registry.fedoraproject.org/fedora
0fe73490aedd: Pulling fs layer
error pulling image configuration: Get https://cdn.registry.fedoraproject.org/v2/fedora/blobs/sha256:d00f2170d181b5605ef7b165a88be0a67200dd8f2878811df76572338d86f974: x509: certificate has expired or is not yet
valid

The same command works just fine on x86_64.

It was just an issue with the date of the aarch64 box I was doing the testing from :D. So this should work now, it can be tested using the rawhide image.

I will coordinate with releng to do a base image release this week.