I've intermittently hit 503 before when updating flatpaks that are hosted on registry.fedoraproject.org, but this time I seem to be getting it all the time. A reproducer with just wget:
$ wget https://registry.fedoraproject.org/v2/glade/blobs/sha256:fbc9c0c54e70dbfa4570f321f02289ae3ebb483009944be07e7278c8fd84fc8a --2020-03-03 03:55:03-- https://registry.fedoraproject.org/v2/glade/blobs/sha256:fbc9c0c54e70dbfa4570f321f02289ae3ebb483009944be07e7278c8fd84fc8a Resolving registry.fedoraproject.org (registry.fedoraproject.org)... 140.211.169.206, 13.250.126.156, 8.43.85.73, ... Connecting to registry.fedoraproject.org (registry.fedoraproject.org)|140.211.169.206|:443... connected. HTTP request sent, awaiting response... 302 Found Location: https://cdn.registry.fedoraproject.org/v2/glade/blobs/sha256:fbc9c0c54e70dbfa4570f321f02289ae3ebb483009944be07e7278c8fd84fc8a [following] --2020-03-03 03:55:03-- https://cdn.registry.fedoraproject.org/v2/glade/blobs/sha256:fbc9c0c54e70dbfa4570f321f02289ae3ebb483009944be07e7278c8fd84fc8a Resolving cdn.registry.fedoraproject.org (cdn.registry.fedoraproject.org)... 195.181.172.3 Connecting to cdn.registry.fedoraproject.org (cdn.registry.fedoraproject.org)|195.181.172.3|:443... connected. HTTP request sent, awaiting response... 503 Service Temporarily Unavailable 2020-03-03 03:55:05 ERROR 503: Service Temporarily Unavailable.
Another attempt that resolved DNF differently (but still 503):
$ wget https://registry.fedoraproject.org/v2/glade/blobs/sha256:fbc9c0c54e70dbfa4570f321f02289ae3ebb483009944be07e7278c8fd84fc8a --2020-03-03 04:00:13-- https://registry.fedoraproject.org/v2/glade/blobs/sha256:fbc9c0c54e70dbfa4570f321f02289ae3ebb483009944be07e7278c8fd84fc8a Resolving registry.fedoraproject.org (registry.fedoraproject.org)... 13.250.126.156, 152.19.134.142, 209.132.181.15, ... Connecting to registry.fedoraproject.org (registry.fedoraproject.org)|13.250.126.156|:443... connected. HTTP request sent, awaiting response... 302 Found Location: https://cdn.registry.fedoraproject.org/v2/glade/blobs/sha256:fbc9c0c54e70dbfa4570f321f02289ae3ebb483009944be07e7278c8fd84fc8a [following] --2020-03-03 04:00:14-- https://cdn.registry.fedoraproject.org/v2/glade/blobs/sha256:fbc9c0c54e70dbfa4570f321f02289ae3ebb483009944be07e7278c8fd84fc8a Resolving cdn.registry.fedoraproject.org (cdn.registry.fedoraproject.org)... 185.59.222.18 Connecting to cdn.registry.fedoraproject.org (cdn.registry.fedoraproject.org)|185.59.222.18|:443... connected. HTTP request sent, awaiting response... 503 Service Temporarily Unavailable 2020-03-03 04:00:16 ERROR 503: Service Temporarily Unavailable.
Might be related: https://pagure.io/releng/issue/9282
Yep, seems to be the same issue. (I also got the above wget test to pass a few times, so it's not completely broken for me.)
I have mailed our cdn provider asking if they might see anything. It always seems to be the 185.59.222.18 backend (at least from our testing)
Thanks! For what it's worth, I got 503 from the 195.181.172.3 backend as well in the morning.
Metadata Update from @kevin: - Issue assigned to kevin - Issue priority set to: Waiting on Assignee (was: Needs Review)
Can you please try now and see if you get any 503s?
I can't seem to reproduce it any more. Thanks! What was the issue in the end?
It was a firewall on the registry boxes, blocking a new proxy, causing the cdn to show a 503 when it hit that proxy for caching in the cdn.
The firewall is updated now and all should be back to normal.
Metadata Update from @kevin: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.