fedora-infrastructure

Clone
Source Code
GIT

#7662 regindexer service not indexing
Closed: Fixed 4 years ago by kevin. Opened 5 years ago by otaylor.

Closed: Fixed
Reopen Issue

When I look at https://registry.fedoraproject.org/index/flatpak-testing.json - I see:

https://bodhi.fedoraproject.org/updates/FEDORA-FLATPAK-2019-7464011a27 (pushed to testing 6 days ago)

but I don't see

https://bodhi.fedoraproject.org/updates/FEDORA-FLATPAK-2019-768a57ee57 (pushed to testing 3 days ago)

Can we examine the logs and see if there are any errors occurring?

I do see: 

WARN[0000] generating tags template for repo "0ad" failed: getting v1 manifest for 0ad:master-2920190208081909.1 failed: missing signature key 
WARN[0000] generating tags template for repo "wesnoth" failed: getting v1 manifest for wesnoth:master-2920190214144818.1 failed: missing signature key 
WARN[0000] generating tags template for repo "eog" failed: getting v1 manifest for eog:master-20181128204005.1 failed: missing signature key 
WARN[0000] generating tags template for repo "transmission" failed: getting v1 manifest for transmission:master-2920190227112914.1 failed: missing signature key 
WARN[0000] generating tags template for repo "gnome-clocks" failed: getting v1 manifest for gnome-clocks:master-2920190201161932.1 failed: missing signature key 
WARN[0000] generating tags template for repo "feedreader" failed: getting v1 manifest for feedreader:master-20181203200949.1 failed: missing signature key 
WARN[0000] generating tags template for repo "quadrapassel" failed: getting v1 manifest for quadrapassel:master-20181203181243.2 failed: missing signature key 
WARN[0000] generating tags template for repo "mediawriter" failed: getting v1 manifest for mediawriter:master-2920190124072439.3 failed: missing signature key 
WARN[0000] generating tags template for repo "flatpak-runtime" failed: getting v1 manifest for flatpak-runtime:f29-20181128194032.1 failed: missing signature key 
WARN[0000] generating tags template for repo "gnome-sound-recorder" failed: getting v1 manifest for gnome-sound-recorder:master-2920190304163154.1 failed: missing signature key 
WARN[0000] generating tags template for repo "thunderbird" failed: getting v1 manifest for thunderbird:master-2920190221203332.1 failed: missing signature key 
WARN[0000] generating tags template for repo "gnome-characters" failed: getting v1 manifest for gnome-characters:master-2920190321113835.1 failed: missing signature key

Hmm, those are errors from 'reg' which we use to build the HTML front page, and not from 'regindexer'. So not related.

Since regindexer isn't run off a cron job, but is instead fedmsg triggered, any errors would be back a few days. Easiest thing would be to restart the service, which will trigger a reindex, and see what gets logged.

oops. thats what I get for looking right before heading to sleep. :)

ok, here's logs after a restart: 

Mar 27 18:49:28 sundries01.phx2.fedoraproject.org systemd[1]: Stopping Daemon for creating an index of a container registry...
Mar 27 18:49:28 sundries01.phx2.fedoraproject.org systemd[1]: Stopped Daemon for creating an index of a container registry.
Mar 27 18:49:28 sundries01.phx2.fedoraproject.org systemd[1]: Started Daemon for creating an index of a container registry.
Mar 27 18:49:32 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer.daemon:Listening for messages with topic org.fedoraproject.prod.bodhi.mashtask.complete
Mar 27 18:49:32 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer.daemon:Rebuilding index for: Initialization
Mar 27 18:49:32 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): registry.fedoraproject.org
Mar 27 18:49:32 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: /usr/lib/python2.7/site-packages/fedmsg/config.py:817: UserWarning: [Errno 13] Permission denied: '/etc/fedmsg.d/fas_credentials.py'
Mar 27 18:49:32 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: warnings.warn(str(e))
Mar 27 18:49:32 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: /usr/lib/python2.7/site-packages/fedmsg/core.py:159: UserWarning: fedmsg is not configured to send any zmq messages for name 'regindexer.sundries01'
Mar 27 18:49:32 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: "for name %r" % config.get("name", None))
Mar 27 18:49:32 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying 0ad:testing
Mar 27 18:49:32 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying 0ad:latest
Mar 27 18:49:32 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying 0ad:sha256:c3538248344016f332e2a1e33701ddd8cc70d024d3b71c3391e5d4862e4d0a1a
Mar 27 18:49:32 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): cdn.registry.fedoraproject.org
Mar 27 18:49:33 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying 0ad:sha256:c3538248344016f332e2a1e33701ddd8cc70d024d3b71c3391e5d4862e4d0a1a
Mar 27 18:49:33 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying 0ad:sha256:c3538248344016f332e2a1e33701ddd8cc70d024d3b71c3391e5d4862e4d0a1a
Mar 27 18:49:33 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying 0ad:sha256:c3538248344016f332e2a1e33701ddd8cc70d024d3b71c3391e5d4862e4d0a1a
Mar 27 18:49:33 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying eog:testing
Mar 27 18:49:33 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying eog:latest
Mar 27 18:49:33 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying eog:sha256:30811750e83233727f8b8006b8686b838baafe56f7793ec9f8ce16ea539349cf
Mar 27 18:49:33 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying eog:sha256:30811750e83233727f8b8006b8686b838baafe56f7793ec9f8ce16ea539349cf
Mar 27 18:49:33 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying eog:sha256:30811750e83233727f8b8006b8686b838baafe56f7793ec9f8ce16ea539349cf
Mar 27 18:49:33 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying eog:sha256:30811750e83233727f8b8006b8686b838baafe56f7793ec9f8ce16ea539349cf
Mar 27 18:49:33 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f25/cockpit:latest
Mar 27 18:49:33 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f25/container-engine:latest
Mar 27 18:49:33 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f25/docker:latest
Mar 27 18:49:33 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f25/etcd:latest
Mar 27 18:49:33 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f25/flannel:latest
Mar 27 18:49:33 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f25/httpd:latest
Mar 27 18:49:33 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f25/kubernetes-apiserver:latest
Mar 27 18:49:34 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f25/kubernetes-controller-manager:latest
Mar 27 18:49:34 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: WARNING:fedmsg.core:Couldn't resolve 'bodhi-backend03.phx2.fedoraproject.org'
Mar 27 18:49:34 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f25/kubernetes-kubelet:latest
Mar 27 18:49:34 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f25/kubernetes-master:latest
Mar 27 18:49:34 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f25/kubernetes-node:latest
Mar 27 18:49:34 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f25/kubernetes-proxy:latest
Mar 27 18:49:34 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f25/kubernetes-scheduler:latest
Mar 27 18:49:34 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f25/mariadb:latest
Mar 27 18:49:34 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f25/mirrormanager2-mirrorlist:latest
Mar 27 18:49:34 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f25/mongodb:latest
Mar 27 18:49:34 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f25/nodejs:latest
Mar 27 18:49:34 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f25/perl:latest
Mar 27 18:49:35 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f25/php:latest
Mar 27 18:49:35 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f25/redis:latest
Mar 27 18:49:35 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f25/ruby:latest
Mar 27 18:49:35 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f25/s2i-base:latest
Mar 27 18:49:35 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f25/toolchain:latest
Mar 27 18:49:35 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f25/waiverdb:latest
Mar 27 18:49:35 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f26/cockpit:latest
Mar 27 18:49:35 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f26/docker:latest
Mar 27 18:49:35 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f26/docker-distribution:latest
Mar 27 18:49:36 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f26/etcd:latest
Mar 27 18:49:36 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f26/flannel:latest
Mar 27 18:49:36 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f26/greenwave:latest
Mar 27 18:49:36 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f26/httpd:latest
Mar 27 18:49:36 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f26/kubernetes-apiserver:latest
Mar 27 18:49:36 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f26/kubernetes-controller-manager:latest
Mar 27 18:49:36 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f26/kubernetes-kubelet:latest
Mar 27 18:49:36 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f26/kubernetes-master:latest
Mar 27 18:49:36 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f26/kubernetes-node:latest
Mar 27 18:49:36 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f26/kubernetes-proxy:latest
Mar 27 18:49:36 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f26/kubernetes-scheduler:latest
Mar 27 18:49:36 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f26/mariadb:latest
Mar 27 18:49:37 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f26/mirrormanager2-mirrorlist:latest
Mar 27 18:49:37 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f26/mongodb:latest
Mar 27 18:49:37 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f26/nodejs:latest
Mar 27 18:49:37 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f26/php:latest
Mar 27 18:49:37 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f26/python-classroom:latest
Mar 27 18:49:37 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f26/python3:latest
Mar 27 18:49:38 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f26/redis:latest
Mar 27 18:49:38 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f26/s2i-base:latest
Mar 27 18:49:38 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:requests.packages.urllib3.connectionpool:Resetting dropped connection: cdn.registry.fedoraproject.org
Mar 27 18:49:38 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f26/s2i-core:latest
Mar 27 18:49:38 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f26/tools:latest
Mar 27 18:49:38 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f26/waiverdb:latest
Mar 27 18:49:38 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f26-modular/boltron:latest
Mar 27 18:49:38 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f27/cassandra:latest
Mar 27 18:49:39 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f27/cockpit:latest
Mar 27 18:49:39 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f27/cri-o:latest
Mar 27 18:49:39 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f27/docker:latest
Mar 27 18:49:40 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f27/docker-distribution:latest
Mar 27 18:49:40 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f27/etcd:latest
Mar 27 18:49:40 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f27/flannel:latest
Mar 27 18:49:40 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f27/glusterfs:latest
Mar 27 18:49:40 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f27/httpd:latest
Mar 27 18:49:40 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f27/kubernetes-apiserver:latest
Mar 27 18:49:41 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f27/kubernetes-controller-manager:latest
Mar 27 18:49:41 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f27/kubernetes-kubelet:latest
Mar 27 18:49:41 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f27/kubernetes-master:latest
Mar 27 18:49:41 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f27/kubernetes-node:latest
Mar 27 18:49:41 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f27/kubernetes-proxy:latest
Mar 27 18:49:41 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f27/kubernetes-scheduler:latest
Mar 27 18:49:41 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f27/mariadb:latest
Mar 27 18:49:41 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f27/memcached:latest
Mar 27 18:49:42 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f27/nginx:latest
Mar 27 18:49:42 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f27/python-classroom:latest
Mar 27 18:49:42 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f27/s2i-base:latest
Mar 27 18:49:43 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f27/s2i-core:latest
Mar 27 18:49:43 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f27/tools:latest
Mar 27 18:49:43 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f27/varnish:latest
Mar 27 18:49:43 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f27/waiverdb:latest
Mar 27 18:49:44 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/cassandra:testing
Mar 27 18:49:44 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/cassandra:latest
Mar 27 18:49:44 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/cassandra:sha256:6b9f29ccd90853b41d96fca0cecabe7e8ad997ba3acaad7256a5f33db3ff7c29
Mar 27 18:49:44 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/cassandra:sha256:6b9f29ccd90853b41d96fca0cecabe7e8ad997ba3acaad7256a5f33db3ff7c29
Mar 27 18:49:44 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/cassandra:sha256:6b9f29ccd90853b41d96fca0cecabe7e8ad997ba3acaad7256a5f33db3ff7c29
Mar 27 18:49:44 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/cassandra:sha256:6b9f29ccd90853b41d96fca0cecabe7e8ad997ba3acaad7256a5f33db3ff7c29
Mar 27 18:49:44 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/cockpit:latest
Mar 27 18:49:44 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/cri-o:latest
Mar 27 18:49:45 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/docker:latest
Mar 27 18:49:45 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/docker-distribution:latest
Mar 27 18:49:45 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/fedora-toolbox:testing
Mar 27 18:49:45 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/fedora-toolbox:latest
Mar 27 18:49:45 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/fedora-toolbox:sha256:ec7d8cde1ad18ac517576b61ccc184894ffbce609c38b1c30bd655c97c282bd9
Mar 27 18:49:45 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/fedora-toolbox:sha256:ec7d8cde1ad18ac517576b61ccc184894ffbce609c38b1c30bd655c97c282bd9
Mar 27 18:49:45 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/fedora-toolbox:sha256:f802ac3f977ea86471ed3b42abd607c81919b77eb8b6bfcb4d0f08f43d17321a
Mar 27 18:49:45 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/fedora-toolbox:sha256:f802ac3f977ea86471ed3b42abd607c81919b77eb8b6bfcb4d0f08f43d17321a
Mar 27 18:49:45 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/golang:testing
Mar 27 18:49:45 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/golang:latest
Mar 27 18:49:45 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/golang:sha256:2e4a9c787c35079a24abdbad45d560eaa5aeecc90275aa20d5e55496c4f2f7d2
Mar 27 18:49:46 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/golang:sha256:2e4a9c787c35079a24abdbad45d560eaa5aeecc90275aa20d5e55496c4f2f7d2
Mar 27 18:49:46 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/golang:sha256:2e4a9c787c35079a24abdbad45d560eaa5aeecc90275aa20d5e55496c4f2f7d2
Mar 27 18:49:46 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/golang:sha256:2e4a9c787c35079a24abdbad45d560eaa5aeecc90275aa20d5e55496c4f2f7d2
Mar 27 18:49:46 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/haproxy:latest
Mar 27 18:49:46 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/httpd:testing
Mar 27 18:49:46 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/httpd:latest
Mar 27 18:49:46 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/httpd:sha256:3e0d266f56467b6952176cf6db9de51608369033a3eff58d548661c4bfa60acb
Mar 27 18:49:46 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/httpd:sha256:3e0d266f56467b6952176cf6db9de51608369033a3eff58d548661c4bfa60acb
Mar 27 18:49:46 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/httpd:sha256:3e0d266f56467b6952176cf6db9de51608369033a3eff58d548661c4bfa60acb
Mar 27 18:49:46 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/httpd:sha256:3e0d266f56467b6952176cf6db9de51608369033a3eff58d548661c4bfa60acb
Mar 27 18:49:46 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/kubernetes-apiserver:latest
Mar 27 18:49:46 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/kubernetes-controller-manager:latest
Mar 27 18:49:46 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/kubernetes-kubelet:latest
Mar 27 18:49:46 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/kubernetes-master:latest
Mar 27 18:49:46 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/kubernetes-node:latest
Mar 27 18:49:46 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/kubernetes-scheduler:latest
Mar 27 18:49:46 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/mariadb:testing
Mar 27 18:49:46 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/mariadb:latest
Mar 27 18:49:46 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/mariadb:sha256:a812d2cffbaf514cd1a380ebfecd38587e027c33eb4485f05d8af77b1ec2a82e
Mar 27 18:49:47 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/mariadb:sha256:a812d2cffbaf514cd1a380ebfecd38587e027c33eb4485f05d8af77b1ec2a82e
Mar 27 18:49:47 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/mariadb:sha256:a812d2cffbaf514cd1a380ebfecd38587e027c33eb4485f05d8af77b1ec2a82e
Mar 27 18:49:47 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:requests.packages.urllib3.connectionpool:Resetting dropped connection: cdn.registry.fedoraproject.org
Mar 27 18:49:47 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/mariadb:sha256:a812d2cffbaf514cd1a380ebfecd38587e027c33eb4485f05d8af77b1ec2a82e
Mar 27 18:49:47 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/memcached:latest
Mar 27 18:49:47 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/mongodb:testing
Mar 27 18:49:47 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/mongodb:latest
Mar 27 18:49:47 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/mongodb:sha256:091e42b3d1ad7bfefaaa0a6d392cfab61506af2c6296751357df379716c3a337
Mar 27 18:49:47 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/mongodb:sha256:091e42b3d1ad7bfefaaa0a6d392cfab61506af2c6296751357df379716c3a337
Mar 27 18:49:47 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/mongodb:sha256:091e42b3d1ad7bfefaaa0a6d392cfab61506af2c6296751357df379716c3a337
Mar 27 18:49:47 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/mongodb:sha256:091e42b3d1ad7bfefaaa0a6d392cfab61506af2c6296751357df379716c3a337
Mar 27 18:49:47 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/mysql:testing
Mar 27 18:49:47 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/mysql:latest
Mar 27 18:49:47 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/mysql:sha256:5bf2ca2ac05cc7fc554d081c5bab77693332bc14ae62857f3a3edba9a71ca606
Mar 27 18:49:47 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/mysql:sha256:5bf2ca2ac05cc7fc554d081c5bab77693332bc14ae62857f3a3edba9a71ca606
Mar 27 18:49:47 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/mysql:sha256:5bf2ca2ac05cc7fc554d081c5bab77693332bc14ae62857f3a3edba9a71ca606
Mar 27 18:49:47 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/mysql:sha256:5bf2ca2ac05cc7fc554d081c5bab77693332bc14ae62857f3a3edba9a71ca606
Mar 27 18:49:47 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/nginx:testing
Mar 27 18:49:47 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/nginx:latest
Mar 27 18:49:47 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/nginx:sha256:e82d3f6cdeb907d6ca9415add50eb6869aa8a6aead56c7cc3e020c3e03766c38
Mar 27 18:49:48 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/nginx:sha256:e82d3f6cdeb907d6ca9415add50eb6869aa8a6aead56c7cc3e020c3e03766c38
Mar 27 18:49:48 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/nginx:sha256:e82d3f6cdeb907d6ca9415add50eb6869aa8a6aead56c7cc3e020c3e03766c38
Mar 27 18:49:48 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/nginx:sha256:e82d3f6cdeb907d6ca9415add50eb6869aa8a6aead56c7cc3e020c3e03766c38
Mar 27 18:49:48 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/origin:latest
Mar 27 18:49:49 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/origin-base:latest
Mar 27 18:49:49 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/perl:testing
Mar 27 18:49:49 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/perl:latest
Mar 27 18:49:49 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/perl:sha256:70e3cca92778a586c783c483501a2ea3f5849996229c1f40d71c511a25bf9381
Mar 27 18:49:49 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/perl:sha256:70e3cca92778a586c783c483501a2ea3f5849996229c1f40d71c511a25bf9381
Mar 27 18:49:49 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/perl:sha256:70e3cca92778a586c783c483501a2ea3f5849996229c1f40d71c511a25bf9381
Mar 27 18:49:49 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/perl:sha256:70e3cca92778a586c783c483501a2ea3f5849996229c1f40d71c511a25bf9381
Mar 27 18:49:49 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/postgresql:testing
Mar 27 18:49:49 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/postgresql:latest
Mar 27 18:49:49 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/postgresql:sha256:39f19b8b068ade44bd7908b93721e95c38cb99c31b963608e0c1c34c248e948a
Mar 27 18:49:49 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/postgresql:sha256:39f19b8b068ade44bd7908b93721e95c38cb99c31b963608e0c1c34c248e948a
Mar 27 18:49:49 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/postgresql:sha256:39f19b8b068ade44bd7908b93721e95c38cb99c31b963608e0c1c34c248e948a
Mar 27 18:49:49 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/postgresql:sha256:39f19b8b068ade44bd7908b93721e95c38cb99c31b963608e0c1c34c248e948a
Mar 27 18:49:49 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/python-classroom:latest
Mar 27 18:49:50 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/python3:testing
Mar 27 18:49:50 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/python3:latest
Mar 27 18:49:50 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/python3:sha256:08a2dbeb4bed9520ce9ebde9e591f64010a3f1e8b5266ec7da87bc06695ef225
Mar 27 18:49:50 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:requests.packages.urllib3.connectionpool:Resetting dropped connection: registry.fedoraproject.org
Mar 27 18:49:50 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/python3:sha256:08a2dbeb4bed9520ce9ebde9e591f64010a3f1e8b5266ec7da87bc06695ef225
Mar 27 18:49:50 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/python3:sha256:08a2dbeb4bed9520ce9ebde9e591f64010a3f1e8b5266ec7da87bc06695ef225
Mar 27 18:49:50 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/python3:sha256:08a2dbeb4bed9520ce9ebde9e591f64010a3f1e8b5266ec7da87bc06695ef225
Mar 27 18:49:50 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/redis:latest
Mar 27 18:49:50 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/ruby:testing
Mar 27 18:49:50 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/ruby:latest
Mar 27 18:49:50 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/ruby:sha256:ffeb67338b506695a1f9d841d10adbb84f634914ea270d7553ee4fecd764bac4
Mar 27 18:49:50 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/ruby:sha256:ffeb67338b506695a1f9d841d10adbb84f634914ea270d7553ee4fecd764bac4
Mar 27 18:49:50 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/ruby:sha256:ffeb67338b506695a1f9d841d10adbb84f634914ea270d7553ee4fecd764bac4
Mar 27 18:49:50 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/ruby:sha256:ffeb67338b506695a1f9d841d10adbb84f634914ea270d7553ee4fecd764bac4
Mar 27 18:49:50 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/s2i-base:testing
Mar 27 18:49:50 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/s2i-base:latest
Mar 27 18:49:50 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/s2i-base:sha256:8763afb9f0053f2152c7ac4b7535d97c8eb7b9cf5fc65d51384031d25b5f2eb1
Mar 27 18:49:50 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/s2i-base:sha256:8763afb9f0053f2152c7ac4b7535d97c8eb7b9cf5fc65d51384031d25b5f2eb1
Mar 27 18:49:50 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/s2i-base:sha256:8763afb9f0053f2152c7ac4b7535d97c8eb7b9cf5fc65d51384031d25b5f2eb1
Mar 27 18:49:50 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/s2i-base:sha256:8763afb9f0053f2152c7ac4b7535d97c8eb7b9cf5fc65d51384031d25b5f2eb1
Mar 27 18:49:50 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/s2i-core:testing
Mar 27 18:49:50 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/s2i-core:latest
Mar 27 18:49:50 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/s2i-core:sha256:4a2868969dceeef84237a588ceaea9c73bc1d10be756b2b733d0867630a5bd70
Mar 27 18:49:50 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/s2i-core:sha256:4a2868969dceeef84237a588ceaea9c73bc1d10be756b2b733d0867630a5bd70
Mar 27 18:49:50 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/s2i-core:sha256:4a2868969dceeef84237a588ceaea9c73bc1d10be756b2b733d0867630a5bd70
Mar 27 18:49:50 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/s2i-core:sha256:4a2868969dceeef84237a588ceaea9c73bc1d10be756b2b733d0867630a5bd70
Mar 27 18:49:50 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/varnish:testing
Mar 27 18:49:50 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/varnish:latest
Mar 27 18:49:50 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/varnish:sha256:c863e5340a06fc90b1763f9c403dafbe864df2c5746141cd61993c14429deb6d
Mar 27 18:49:50 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/varnish:sha256:c863e5340a06fc90b1763f9c403dafbe864df2c5746141cd61993c14429deb6d
Mar 27 18:49:50 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/varnish:sha256:c863e5340a06fc90b1763f9c403dafbe864df2c5746141cd61993c14429deb6d
Mar 27 18:49:50 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f28/varnish:sha256:c863e5340a06fc90b1763f9c403dafbe864df2c5746141cd61993c14429deb6d
Mar 27 18:49:50 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/cassandra:testing
Mar 27 18:49:50 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/cassandra:latest
Mar 27 18:49:51 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/cockpit:testing
Mar 27 18:49:51 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/cockpit:latest
Mar 27 18:49:51 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/fedora-toolbox:testing
Mar 27 18:49:51 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/fedora-toolbox:latest
Mar 27 18:49:51 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/fedora-toolbox:sha256:8e7630563a7889869886e685bb1048e6cf9da81f695729d7bcb5df6d06f663c5
Mar 27 18:49:51 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/fedora-toolbox:sha256:8e7630563a7889869886e685bb1048e6cf9da81f695729d7bcb5df6d06f663c5
Mar 27 18:49:51 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/fedora-toolbox:sha256:aa2d95c483e83e78bd0f9113b2488847801a13db138acbf372c4e4cc3583b4e2
Mar 27 18:49:51 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/fedora-toolbox:sha256:aa2d95c483e83e78bd0f9113b2488847801a13db138acbf372c4e4cc3583b4e2
Mar 27 18:49:51 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/golang:testing
Mar 27 18:49:51 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/golang:latest
Mar 27 18:49:51 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/golang:sha256:4eb167db96dc7678bd578bad531d9cdbf28911a598bdc48cf8b209c59bd6dfa6
Mar 27 18:49:51 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/golang:sha256:4eb167db96dc7678bd578bad531d9cdbf28911a598bdc48cf8b209c59bd6dfa6
Mar 27 18:49:51 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/golang:sha256:4eb167db96dc7678bd578bad531d9cdbf28911a598bdc48cf8b209c59bd6dfa6
Mar 27 18:49:51 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/golang:sha256:4eb167db96dc7678bd578bad531d9cdbf28911a598bdc48cf8b209c59bd6dfa6
Mar 27 18:49:51 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/httpd:testing
Mar 27 18:49:51 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/httpd:latest
Mar 27 18:49:52 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/mirrormanager2-mirrorlist:testing
Mar 27 18:49:52 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/mirrormanager2-mirrorlist:sha256:fab70f7c423e6165267e8c2849bd6e5a923158dcce599a23dbc7f890f97a8310
Mar 27 18:49:52 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/mirrormanager2-mirrorlist:sha256:fab70f7c423e6165267e8c2849bd6e5a923158dcce599a23dbc7f890f97a8310
Mar 27 18:49:52 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/nginx:testing
Mar 27 18:49:52 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/nginx:latest
Mar 27 18:49:52 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-base:testing
Mar 27 18:49:52 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-base:latest
Mar 27 18:49:52 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-base:sha256:a7c8eedb7f46202f168fbb09d675a2b6d7cac918a858ac29d56d80b353ab4525
Mar 27 18:49:52 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-base:sha256:a7c8eedb7f46202f168fbb09d675a2b6d7cac918a858ac29d56d80b353ab4525
Mar 27 18:49:52 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-base:sha256:a7c8eedb7f46202f168fbb09d675a2b6d7cac918a858ac29d56d80b353ab4525
Mar 27 18:49:52 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-base:sha256:a7c8eedb7f46202f168fbb09d675a2b6d7cac918a858ac29d56d80b353ab4525
Mar 27 18:49:52 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-cli:testing
Mar 27 18:49:52 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-cli:latest
Mar 27 18:49:52 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-cli:sha256:2fea59aa79f9b783905f1e7048fd0df128ba6e0e3bc6228e8f5e7394bef7b694
Mar 27 18:49:52 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-cli:sha256:2fea59aa79f9b783905f1e7048fd0df128ba6e0e3bc6228e8f5e7394bef7b694
Mar 27 18:49:52 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-cli:sha256:2fea59aa79f9b783905f1e7048fd0df128ba6e0e3bc6228e8f5e7394bef7b694
Mar 27 18:49:52 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-cli:sha256:2fea59aa79f9b783905f1e7048fd0df128ba6e0e3bc6228e8f5e7394bef7b694
Mar 27 18:49:52 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-control-plane:testing
Mar 27 18:49:52 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-control-plane:latest
Mar 27 18:49:52 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-control-plane:sha256:b7f8a50f2da7771598a2edb00522403f47302beb5f33fb9bfaa7450449a47c8c
Mar 27 18:49:52 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-control-plane:sha256:b7f8a50f2da7771598a2edb00522403f47302beb5f33fb9bfaa7450449a47c8c
Mar 27 18:49:52 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-control-plane:sha256:b7f8a50f2da7771598a2edb00522403f47302beb5f33fb9bfaa7450449a47c8c
Mar 27 18:49:52 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-control-plane:sha256:b7f8a50f2da7771598a2edb00522403f47302beb5f33fb9bfaa7450449a47c8c
Mar 27 18:49:52 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-docker-builder:testing
Mar 27 18:49:52 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-docker-builder:latest
Mar 27 18:49:52 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-docker-builder:sha256:063f9ecec1e6d62a269a44348f6bf7ae84166b76b20f2b6fcc8de0444c4a306e
Mar 27 18:49:52 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-docker-builder:sha256:063f9ecec1e6d62a269a44348f6bf7ae84166b76b20f2b6fcc8de0444c4a306e
Mar 27 18:49:52 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-docker-builder:sha256:063f9ecec1e6d62a269a44348f6bf7ae84166b76b20f2b6fcc8de0444c4a306e
Mar 27 18:49:52 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-docker-builder:sha256:063f9ecec1e6d62a269a44348f6bf7ae84166b76b20f2b6fcc8de0444c4a306e
Mar 27 18:49:53 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-egress-dns-proxy:testing
Mar 27 18:49:53 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-egress-dns-proxy:latest
Mar 27 18:49:53 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-egress-dns-proxy:sha256:5ace61acc7da72ca065ca1948e52fa5853320ca329dfbf439f5eb1056717c8e4
Mar 27 18:49:53 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-egress-dns-proxy:sha256:5ace61acc7da72ca065ca1948e52fa5853320ca329dfbf439f5eb1056717c8e4
Mar 27 18:49:53 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-egress-dns-proxy:sha256:5ace61acc7da72ca065ca1948e52fa5853320ca329dfbf439f5eb1056717c8e4
Mar 27 18:49:53 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-egress-dns-proxy:sha256:5ace61acc7da72ca065ca1948e52fa5853320ca329dfbf439f5eb1056717c8e4
Mar 27 18:49:53 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-egress-router:testing
Mar 27 18:49:53 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-egress-router:latest
Mar 27 18:49:53 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-egress-router:sha256:3ddf2e1d02648b47a3085af83f5724d785b145c600e022d3890c27cfaadc86f7
Mar 27 18:49:53 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-egress-router:sha256:3ddf2e1d02648b47a3085af83f5724d785b145c600e022d3890c27cfaadc86f7
Mar 27 18:49:53 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-egress-router:sha256:3ddf2e1d02648b47a3085af83f5724d785b145c600e022d3890c27cfaadc86f7
Mar 27 18:49:53 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:requests.packages.urllib3.connectionpool:Resetting dropped connection: cdn.registry.fedoraproject.org
Mar 27 18:49:53 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-egress-router:sha256:3ddf2e1d02648b47a3085af83f5724d785b145c600e022d3890c27cfaadc86f7
Mar 27 18:49:53 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-f5-router:testing
Mar 27 18:49:53 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-f5-router:latest
Mar 27 18:49:53 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-f5-router:sha256:abdaab2b3e50f71e981bf47aa51a0ff65878974389bd90946fb34e11c4153dbf
Mar 27 18:49:53 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-f5-router:sha256:abdaab2b3e50f71e981bf47aa51a0ff65878974389bd90946fb34e11c4153dbf
Mar 27 18:49:53 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-f5-router:sha256:abdaab2b3e50f71e981bf47aa51a0ff65878974389bd90946fb34e11c4153dbf
Mar 27 18:49:53 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-f5-router:sha256:abdaab2b3e50f71e981bf47aa51a0ff65878974389bd90946fb34e11c4153dbf
Mar 27 18:49:53 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-hyperkube:testing
Mar 27 18:49:53 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-hyperkube:latest
Mar 27 18:49:53 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-hyperkube:sha256:ec10775e973377e196ed29fa9cab1f509a42052c2230a05750a9f20c2dbf81e5
Mar 27 18:49:54 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-hyperkube:sha256:ec10775e973377e196ed29fa9cab1f509a42052c2230a05750a9f20c2dbf81e5
Mar 27 18:49:54 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-hyperkube:sha256:ec10775e973377e196ed29fa9cab1f509a42052c2230a05750a9f20c2dbf81e5
Mar 27 18:49:54 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-hyperkube:sha256:ec10775e973377e196ed29fa9cab1f509a42052c2230a05750a9f20c2dbf81e5
Mar 27 18:49:54 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-hypershift:testing
Mar 27 18:49:54 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-hypershift:latest
Mar 27 18:49:54 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-hypershift:sha256:a1b9c6f97e13465555ad99c4cd5f85a83f1ad98c1142a6f023b33c0764c721fb
Mar 27 18:49:54 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-hypershift:sha256:a1b9c6f97e13465555ad99c4cd5f85a83f1ad98c1142a6f023b33c0764c721fb
Mar 27 18:49:54 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-hypershift:sha256:a1b9c6f97e13465555ad99c4cd5f85a83f1ad98c1142a6f023b33c0764c721fb
Mar 27 18:49:54 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-hypershift:sha256:a1b9c6f97e13465555ad99c4cd5f85a83f1ad98c1142a6f023b33c0764c721fb
Mar 27 18:49:54 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-keepalived-ipfailover:testing
Mar 27 18:49:54 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-keepalived-ipfailover:latest
Mar 27 18:49:54 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-keepalived-ipfailover:sha256:52f09c67ce3d57d088466cfd460ed8870e42637e4439684fb51847a5dc843bea
Mar 27 18:49:54 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-keepalived-ipfailover:sha256:52f09c67ce3d57d088466cfd460ed8870e42637e4439684fb51847a5dc843bea
Mar 27 18:49:54 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-keepalived-ipfailover:sha256:52f09c67ce3d57d088466cfd460ed8870e42637e4439684fb51847a5dc843bea
Mar 27 18:49:54 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-keepalived-ipfailover:sha256:52f09c67ce3d57d088466cfd460ed8870e42637e4439684fb51847a5dc843bea
Mar 27 18:49:54 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-nginx-router:testing
Mar 27 18:49:54 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-nginx-router:latest
Mar 27 18:49:54 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-nginx-router:sha256:4e8840fc969551509b4c4ae9b97979a5c091a1b38366fdbc11c0092276f66439
Mar 27 18:49:54 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-nginx-router:sha256:4e8840fc969551509b4c4ae9b97979a5c091a1b38366fdbc11c0092276f66439
Mar 27 18:49:54 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-nginx-router:sha256:4e8840fc969551509b4c4ae9b97979a5c091a1b38366fdbc11c0092276f66439
Mar 27 18:49:54 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-nginx-router:sha256:4e8840fc969551509b4c4ae9b97979a5c091a1b38366fdbc11c0092276f66439
Mar 27 18:49:54 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-recycler:testing
Mar 27 18:49:54 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-recycler:latest
Mar 27 18:49:54 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-recycler:sha256:a82e35c1d4569adcc699bb722a2cebaeea617b14f9cb6cbb943780bdf3fdf606
Mar 27 18:49:55 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-recycler:sha256:a82e35c1d4569adcc699bb722a2cebaeea617b14f9cb6cbb943780bdf3fdf606
Mar 27 18:49:55 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-recycler:sha256:a82e35c1d4569adcc699bb722a2cebaeea617b14f9cb6cbb943780bdf3fdf606
Mar 27 18:49:55 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-recycler:sha256:a82e35c1d4569adcc699bb722a2cebaeea617b14f9cb6cbb943780bdf3fdf606
Mar 27 18:49:55 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-template-service-broker:testing
Mar 27 18:49:55 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-template-service-broker:latest
Mar 27 18:49:55 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-template-service-broker:sha256:d17a538700961b28a8eec8ff23f0dc7a1ec32da5c74ce99c899b775cfb1e0035
Mar 27 18:49:55 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-template-service-broker:sha256:d17a538700961b28a8eec8ff23f0dc7a1ec32da5c74ce99c899b775cfb1e0035
Mar 27 18:49:55 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-template-service-broker:sha256:d17a538700961b28a8eec8ff23f0dc7a1ec32da5c74ce99c899b775cfb1e0035
Mar 27 18:49:55 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-template-service-broker:sha256:d17a538700961b28a8eec8ff23f0dc7a1ec32da5c74ce99c899b775cfb1e0035
Mar 27 18:49:55 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-tests:testing
Mar 27 18:49:55 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-tests:latest
Mar 27 18:49:55 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-tests:sha256:035fd07e3dfbde8c3c1b223a294e306c931946f8f7c5f5476b64554af1e99f35
Mar 27 18:49:55 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-tests:sha256:035fd07e3dfbde8c3c1b223a294e306c931946f8f7c5f5476b64554af1e99f35
Mar 27 18:49:55 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-tests:sha256:035fd07e3dfbde8c3c1b223a294e306c931946f8f7c5f5476b64554af1e99f35
Mar 27 18:49:55 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/origin-tests:sha256:035fd07e3dfbde8c3c1b223a294e306c931946f8f7c5f5476b64554af1e99f35
Mar 27 18:49:55 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/postgresql:testing
Mar 27 18:49:55 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/postgresql:latest
Mar 27 18:49:55 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/ruby:testing
Mar 27 18:49:55 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/ruby:latest
Mar 27 18:49:55 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/s2i-base:testing
Mar 27 18:49:56 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/s2i-base:latest
Mar 27 18:49:56 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/s2i-base:sha256:cd695eb883a66ee39ef0a78843fe8c22f113fa5b43766e0bb3ee8d9a5ccc9c59
Mar 27 18:49:56 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/s2i-base:sha256:cd695eb883a66ee39ef0a78843fe8c22f113fa5b43766e0bb3ee8d9a5ccc9c59
Mar 27 18:49:56 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/s2i-base:sha256:cd695eb883a66ee39ef0a78843fe8c22f113fa5b43766e0bb3ee8d9a5ccc9c59
Mar 27 18:49:56 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/s2i-base:sha256:cd695eb883a66ee39ef0a78843fe8c22f113fa5b43766e0bb3ee8d9a5ccc9c59
Mar 27 18:49:56 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/s2i-core:testing
Mar 27 18:49:56 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/s2i-core:latest
Mar 27 18:49:56 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/s2i-core:sha256:c18e64bee792cbdf331890bfd4bdb27c6a17db9f9daba91cc88982eafb69cdaf
Mar 27 18:49:56 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/s2i-core:sha256:c18e64bee792cbdf331890bfd4bdb27c6a17db9f9daba91cc88982eafb69cdaf
Mar 27 18:49:56 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/s2i-core:sha256:c18e64bee792cbdf331890bfd4bdb27c6a17db9f9daba91cc88982eafb69cdaf
Mar 27 18:49:56 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f29/s2i-core:sha256:c18e64bee792cbdf331890bfd4bdb27c6a17db9f9daba91cc88982eafb69cdaf
Mar 27 18:49:56 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f30/fedora-toolbox:testing
Mar 27 18:49:56 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f30/fedora-toolbox:latest
Mar 27 18:49:56 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f30/fedora-toolbox:sha256:53c62c79a24675cc2da3da51da6d1fbeeeefe5a7f532135ed277d118aa73bb3c
Mar 27 18:49:56 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f30/fedora-toolbox:sha256:53c62c79a24675cc2da3da51da6d1fbeeeefe5a7f532135ed277d118aa73bb3c
Mar 27 18:49:56 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying f30/s2i-core:latest
Mar 27 18:49:56 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying fedora:latest
Mar 27 18:49:56 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying fedora:sha256:faa4581556515da9f3621cde484db2439563cda9cdc8fbd8db2837999598428a
Mar 27 18:49:56 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying fedora:sha256:b888973905a3aad7392532dce6a5aef7eb79b6748ecd604a26c6f6f120492361
Mar 27 18:49:56 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying fedora:sha256:725c45a19d4d916deb91f5cf1125aeb6adbb17f2cc3b0ac4c700e1825c5425b1
Mar 27 18:49:56 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying fedora:sha256:82ed24d5e06ba7139553cf2b538659f89a5cabc2fa1da676b7b326b9a9488fcd
Mar 27 18:49:57 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying fedora:sha256:313b0bf4188b6b0b7504f3d0f70cce7b4023060ef1ef81aadf12df8b108194bf
Mar 27 18:49:57 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying fedora:sha256:faa4581556515da9f3621cde484db2439563cda9cdc8fbd8db2837999598428a
Mar 27 18:49:57 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying fedora:sha256:b888973905a3aad7392532dce6a5aef7eb79b6748ecd604a26c6f6f120492361
Mar 27 18:49:57 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying fedora:sha256:725c45a19d4d916deb91f5cf1125aeb6adbb17f2cc3b0ac4c700e1825c5425b1
Mar 27 18:49:57 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying fedora:sha256:82ed24d5e06ba7139553cf2b538659f89a5cabc2fa1da676b7b326b9a9488fcd
Mar 27 18:49:57 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying fedora:sha256:313b0bf4188b6b0b7504f3d0f70cce7b4023060ef1ef81aadf12df8b108194bf
Mar 27 18:49:57 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying fedora-minimal:latest
Mar 27 18:49:57 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying fedora-minimal:sha256:1123bb2879415bd8694ceca7af2eeb149fcd22a44adb7d4bd371e919594db74e
Mar 27 18:49:57 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying fedora-minimal:sha256:db78d56d619031631bfb90b666e395fdf5c0cd2d2958bc725e853df2ae270644
Mar 27 18:49:57 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying fedora-minimal:sha256:7f2b3f8e2e61e93f2587c8908fbd4886abfc2baa221562a115239c2e33735acc
Mar 27 18:49:57 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying fedora-minimal:sha256:f4fb43230893a52ba6e4842bc6a540a2aaddbc09a9b7457037cde7e608a2aaa5
Mar 27 18:49:57 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying fedora-minimal:sha256:fe6ca3db26cd4e1bc77fe1d8a1f8134d324a69d04c79c61d709bd60a2cace243
Mar 27 18:49:57 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying fedora-minimal:sha256:1123bb2879415bd8694ceca7af2eeb149fcd22a44adb7d4bd371e919594db74e
Mar 27 18:49:57 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying fedora-minimal:sha256:db78d56d619031631bfb90b666e395fdf5c0cd2d2958bc725e853df2ae270644
Mar 27 18:49:57 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying fedora-minimal:sha256:7f2b3f8e2e61e93f2587c8908fbd4886abfc2baa221562a115239c2e33735acc
Mar 27 18:49:57 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying fedora-minimal:sha256:f4fb43230893a52ba6e4842bc6a540a2aaddbc09a9b7457037cde7e608a2aaa5
Mar 27 18:49:57 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying fedora-minimal:sha256:fe6ca3db26cd4e1bc77fe1d8a1f8134d324a69d04c79c61d709bd60a2cace243
Mar 27 18:49:57 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying feedreader:testing
Mar 27 18:49:57 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying feedreader:latest
Mar 27 18:49:57 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying feedreader:sha256:b88dc894ad71094b6856725b610bd1668d60eb0ab01ca6f5651c1f1b2f1808d5
Mar 27 18:49:58 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying feedreader:sha256:b88dc894ad71094b6856725b610bd1668d60eb0ab01ca6f5651c1f1b2f1808d5
Mar 27 18:49:58 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying feedreader:sha256:b88dc894ad71094b6856725b610bd1668d60eb0ab01ca6f5651c1f1b2f1808d5
Mar 27 18:49:58 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying feedreader:sha256:b88dc894ad71094b6856725b610bd1668d60eb0ab01ca6f5651c1f1b2f1808d5
Mar 27 18:49:58 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying flatpak-runtime:testing
Mar 27 18:49:58 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying flatpak-runtime:latest
Mar 27 18:49:58 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying flatpak-runtime:sha256:72e740300f03abeaa43cf10c795d1f9990ec0a426c7b4408ab6315d86fc53ed6
Mar 27 18:49:58 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying flatpak-runtime:sha256:72e740300f03abeaa43cf10c795d1f9990ec0a426c7b4408ab6315d86fc53ed6
Mar 27 18:49:58 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying flatpak-runtime:sha256:72e740300f03abeaa43cf10c795d1f9990ec0a426c7b4408ab6315d86fc53ed6
Mar 27 18:49:58 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying flatpak-runtime:sha256:72e740300f03abeaa43cf10c795d1f9990ec0a426c7b4408ab6315d86fc53ed6
Mar 27 18:49:58 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying gnome-characters:testing
Mar 27 18:49:58 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying gnome-characters:sha256:18564f138f1de36c1284fc6a6f07e9789ed7dbc1a6e23097224a8aa6d637551a
Mar 27 18:49:59 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Storing icon: /var/lib/regindexer/icons/23/79ab28064acf368c6c9ea84662c3e504ceca9e32d18a2c9ae0b66701e2ae33.png
Mar 27 18:49:59 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Storing icon: /var/lib/regindexer/icons/af/9f6b50bcdb3895c7926dcb08fee5940fe733e532dbe563dd59852e14a918b4.png
Mar 27 18:49:59 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying gnome-characters:sha256:18564f138f1de36c1284fc6a6f07e9789ed7dbc1a6e23097224a8aa6d637551a
Mar 27 18:49:59 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying gnome-clocks:testing
Mar 27 18:49:59 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying gnome-clocks:latest
Mar 27 18:49:59 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying gnome-clocks:sha256:929c0ac2ba2cc79fe5dd0346a6424ab72dd1b2684eb3cd3403a6762f12c251f7
Mar 27 18:49:59 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying gnome-clocks:sha256:929c0ac2ba2cc79fe5dd0346a6424ab72dd1b2684eb3cd3403a6762f12c251f7
Mar 27 18:49:59 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying gnome-clocks:sha256:929c0ac2ba2cc79fe5dd0346a6424ab72dd1b2684eb3cd3403a6762f12c251f7
Mar 27 18:49:59 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying gnome-clocks:sha256:929c0ac2ba2cc79fe5dd0346a6424ab72dd1b2684eb3cd3403a6762f12c251f7
Mar 27 18:49:59 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying gnome-sound-recorder:testing
Mar 27 18:49:59 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying gnome-sound-recorder:sha256:1a95c92ee66e7c5ec95cda85b76d2daf39505617205a57ae5bb029f46fcc8e7b
Mar 27 18:49:59 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying gnome-sound-recorder:sha256:1a95c92ee66e7c5ec95cda85b76d2daf39505617205a57ae5bb029f46fcc8e7b
Mar 27 18:49:59 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying mediawriter:testing
Mar 27 18:49:59 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying mediawriter:latest
Mar 27 18:49:59 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying mediawriter:sha256:d4e8c9981d7a678daa35a04e92eb48af59e16a4e6ee8612539a4d11c02e8cd46
Mar 27 18:49:59 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying mediawriter:sha256:d4e8c9981d7a678daa35a04e92eb48af59e16a4e6ee8612539a4d11c02e8cd46
Mar 27 18:49:59 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying mediawriter:sha256:d4e8c9981d7a678daa35a04e92eb48af59e16a4e6ee8612539a4d11c02e8cd46
Mar 27 18:49:59 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying mediawriter:sha256:d4e8c9981d7a678daa35a04e92eb48af59e16a4e6ee8612539a4d11c02e8cd46
Mar 27 18:49:59 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying quadrapassel:testing
Mar 27 18:49:59 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying quadrapassel:latest
Mar 27 18:49:59 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying quadrapassel:sha256:da885d587491cc004e1056a3709ba8d9b0083bb4dd5d78c9250b4970e46c19c9
Mar 27 18:49:59 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying quadrapassel:sha256:da885d587491cc004e1056a3709ba8d9b0083bb4dd5d78c9250b4970e46c19c9
Mar 27 18:49:59 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying quadrapassel:sha256:da885d587491cc004e1056a3709ba8d9b0083bb4dd5d78c9250b4970e46c19c9
Mar 27 18:49:59 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying quadrapassel:sha256:da885d587491cc004e1056a3709ba8d9b0083bb4dd5d78c9250b4970e46c19c9
Mar 27 18:49:59 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying thunderbird:testing
Mar 27 18:50:00 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying thunderbird:latest
Mar 27 18:50:00 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying thunderbird:sha256:49fde415c096f5716b0e2f060bdf0fedc32b588369248e7580b62535d900289b
Mar 27 18:50:00 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:requests.packages.urllib3.connectionpool:Resetting dropped connection: cdn.registry.fedoraproject.org
Mar 27 18:50:00 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying thunderbird:sha256:49fde415c096f5716b0e2f060bdf0fedc32b588369248e7580b62535d900289b
Mar 27 18:50:00 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying thunderbird:sha256:45c1e9dc9ca6da061a376ee0e7606e3baef6748923cff9a27a0c65426e75a393
Mar 27 18:50:00 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying thunderbird:sha256:45c1e9dc9ca6da061a376ee0e7606e3baef6748923cff9a27a0c65426e75a393
Mar 27 18:50:00 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying transmission:testing
Mar 27 18:50:00 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying transmission:latest
Mar 27 18:50:00 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying transmission:sha256:ff52abbaba41da4325154e89f2eea2b09ca932db85df7570d8b2f5c9b84e7d7d
Mar 27 18:50:01 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying transmission:sha256:ff52abbaba41da4325154e89f2eea2b09ca932db85df7570d8b2f5c9b84e7d7d
Mar 27 18:50:01 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying transmission:sha256:ff52abbaba41da4325154e89f2eea2b09ca932db85df7570d8b2f5c9b84e7d7d
Mar 27 18:50:01 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying transmission:sha256:ff52abbaba41da4325154e89f2eea2b09ca932db85df7570d8b2f5c9b84e7d7d
Mar 27 18:50:01 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying wesnoth:testing
Mar 27 18:50:01 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying wesnoth:latest
Mar 27 18:50:01 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying wesnoth:sha256:2cd388e8dbc7cb51b15247602db71cad55d89e6cb9ab93415c3d3b9faf396a17
Mar 27 18:50:01 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying wesnoth:sha256:2cd388e8dbc7cb51b15247602db71cad55d89e6cb9ab93415c3d3b9faf396a17
Mar 27 18:50:01 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying wesnoth:sha256:2cd388e8dbc7cb51b15247602db71cad55d89e6cb9ab93415c3d3b9faf396a17
Mar 27 18:50:01 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying wesnoth:sha256:2cd388e8dbc7cb51b15247602db71cad55d89e6cb9ab93415c3d3b9faf396a17
Mar 27 18:50:01 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Wrote /var/lib/regindexer/index/flatpak.json
Mar 27 18:50:01 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Wrote /var/lib/regindexer/index/flatpak-testing.json
Mar 27 18:50:01 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Wrote /var/lib/regindexer/index/flatpak-testing-amd64.json
Mar 27 18:50:01 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Wrote /var/lib/regindexer/index/flatpak-amd64.json

Are those 'resetting dropped connections' anything to worry about?

And now it has the content...

Looking back in logs:

Mar 23 03:50:39 sundries01.phx2.fedoraproject.org regindexer-daemon[3044]: INFO:regindexer.daemon:Got org.fedoraproject.prod.bodhi.mashtask.complete message for ctype=container
..
Mar 23 03:51:03 sundries01.phx2.fedoraproject.org regindexer-daemon[3044]: INFO:regindexer.daemon:Got org.fedoraproject.prod.bodhi.mashtask.complete message for ctype=flatpak

Is it bad for both of those to run at the same time?
I don't see any errors aside from that...

The rerun seems to have picked up gnome-characters.

The code is supposed to be able to handle multiple messages like that fine. Are there lines like:

Mar 27 18:49:58 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying gnome-characters:testing
Mar 27 18:49:58 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying gnome-characters:sha256:18564f138f1de36c1284fc6a6f07e9789ed7dbc1a6e23097224a8aa6d637551a
Mar 27 18:49:59 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Storing icon: /var/lib/regindexer/icons/23/79ab28064acf368c6c9ea84662c3e504ceca9e32d18a2c9ae0b66701e2ae33.png
Mar 27 18:49:59 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Storing icon: /var/lib/regindexer/icons/af/9f6b50bcdb3895c7926dcb08fee5940fe733e532dbe563dd59852e14a918b4.png
Mar 27 18:49:59 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying gnome-characters:sha256:18564f138f1de36c1284fc6a6f07e9789ed7dbc1a6e23097224a8aa6d637551a
Mar 27 18:49:59 sundries01.phx2.fedoraproject.org regindexer-daemon[9154]: INFO:regindexer:Querying gnome-clocks:testing

In the Mar 23 lines after Mar 23 03:51:03 sundries01.phx2.fedoraproject.org regindexer-daemon[3044]: INFO:regindexer.daemon:Got org.fedoraproject.prod.bodhi.mashtask.complete message for ctype=flatpak ?

Nope. No lines with gnome-characters except the ones from today.

Maybe a caching problem - when there is a container and flatpak push in sequence - then perhaps the results of https://egistry.fedoraproject.org/v2/_catalog get cached and the second scan gets a stale list of repositories?

Though i don't see caching in our setup other than the CDN, which should only cache the blobs. Is there anything that would cache the _catalog response?

A less likely source of problems might be FS level if the bodhi push happens to one node and the _catalog is read from the other node.

Somewhat less likely than HTTP level caching would be FS
level caching if the push happens on one node and we read the _catalog from the other node - but I wouldn't think that things would be fast enough to cause problems there.

Maybe a caching problem - when there is a container and flatpak push in sequence - then perhaps the results of https://egistry.fedoraproject.org/v2/_catalog get cached and the second scan gets a stale list of repositories?

Could be...

Though i don't see caching in our setup other than the CDN, which should only cache the blobs. Is there anything that would cache the _catalog response?

Yeah, varnish might?

RewriteCond %{REQUEST_METHOD} ^(PATCH|POST|PUT|DELETE)$
RewriteRule ^/v2/(.*)$ http://oci-registry02:5000/v2/$1 [P,L]
RewriteRule ^/v2/(.*)$ http://localhost:6081/v2/$1 [P,L]

A less likely source of problems might be FS level if the bodhi push happens to one node and the _catalog is read from the other node.

The two nodes are using an nfs volume, so I think they should be pretty much in sync there.

Somewhat less likely than HTTP level caching would be FS
level caching if the push happens on one node and we read the _catalog from the other node - but I wouldn't think that things would be fast enough to cause problems there.

Yeah, me either.

Not sure what else to look at here...

If the logs for the registry nodes are still available, they should be
instructive.

  • what node did gnome-characters get pushed to in the bodhi flatpak push
    (think this might be fixed by the httpd config, but can double check in the
    logs)
  • what node handled the subsequent _catalog request from regindexer? If
    neither, then it presumably got cached at the proxy

On 01:

Mar 23 03:50:58 oci-registry01 registry[725]: time="2019-03-23T03:50:58Z" level=error msg="response completed with error" err.code="blob unknown" err.detail=sha256:f286b24eab81cec9689a75a3f1f72d0fe9feaa9b12dccf86958419f23556e3ba err.message="blob unknown to registry" go.version=go1.10rc2 http.request.host=registry.fedoraproject.org http.request.id=8afe0194-f320-4593-82d7-56c812121f16 http.request.method=HEAD http.request.remoteaddr=10.5.125.135 http.request.uri="/v2/gnome-characters/blobs/sha256:f286b24eab81cec9689a75a3f1f72d0fe9feaa9b12dccf86958419f23556e3ba" http.request.useragent="python-requests/2.20.0" http.response.contenttype="application/json; charset=utf-8" http.response.duration=7.258629ms http.response.status=404 http.response.written=157 instance.id=c22aeae6-9c22-4791-b3ae-f34ae787a90d vars.digest="sha256:f286b24eab81cec9689a75a3f1f72d0fe9feaa9b12dccf86958419f23556e3ba" vars.name=gnome-characters version="v2.6.2+unknown"
Mar 23 03:50:58 oci-registry01 registry[725]: 10.5.126.9 - - [23/Mar/2019:03:50:58 +0000] "HEAD /v2/gnome-characters/blobs/sha256:f286b24eab81cec9689a75a3f1f72d0fe9feaa9b12dccf86958419f23556e3ba HTTP/1.1" 404 157 "" "python-requests/2.20.0"
Mar 23 03:50:58 oci-registry01 registry[725]: time="2019-03-23T03:50:58Z" level=info msg="response completed" go.version=go1.10rc2 http.request.host=registry.fedoraproject.org http.request.id=2f20db56-ac82-4d33-812e-6dae6e5bd0b9 http.request.method=GET http.request.remoteaddr=10.5.126.38 http.request.uri="/v2/f26/docker/manifests/latest" http.request.useragent="python-requests/2.6.0 CPython/2.7.5 Linux/3.10.0-957.5.1.el7.x86_64" http.response.contenttype="application/vnd.docker.distribution.manifest.v2+json" http.response.duration=5.753789ms http.response.status=200 http.response.written=719 instance.id=c22aeae6-9c22-4791-b3ae-f34ae787a90d version="v2.6.2+unknown"

on 02:

Mar 23 03:50:58 oci-registry02 registry[776]: time="2019-03-23T03:50:58Z" level=error msg="response completed with error" err.code="blob unknown" err.detail=sha256:1c6c80856abf27262179163366e36059d5e47bbce19c256feb786a4170696abf err.message="blob unknown to registry" go.version=go1.10rc2 http.request.host=registry.fedoraproject.org http.request.id=cf8b786e-e4b0-4e42-a25f-5e47612a4a76 http.request.method=HEAD http.request.remoteaddr=10.5.125.135 http.request.uri="/v2/gnome-characters/blobs/sha256:1c6c80856abf27262179163366e36059d5e47bbce19c256feb786a4170696abf" http.request.useragent="python-requests/2.20.0" http.response.contenttype="application/json; charset=utf-8" http.response.duration=1.824799ms http.response.status=404 http.response.written=157 instance.id=86dbe242-1e24-4664-963c-16733b06c50c vars.digest="sha256:1c6c80856abf27262179163366e36059d5e47bbce19c256feb786a4170696abf" vars.name=gnome-characters version="v2.6.2+unknown"
Mar 23 03:50:58 oci-registry02 registry[776]: 10.5.126.9 - - [23/Mar/2019:03:50:58 +0000] "HEAD /v2/gnome-characters/blobs/sha256:1c6c80856abf27262179163366e36059d5e47bbce19c256feb786a4170696abf HTTP/1.1" 404 157 "" "python-requests/2.20.0"
Mar 23 03:50:58 oci-registry02 registry[776]: time="2019-03-23T03:50:58Z" level=info msg="response completed" go.version=go1.10rc2 http.request.host=registry.fedoraproject.org http.request.id=abd867bf-fa24-414f-b74d-9d2f49d1841a http.request.method=POST http.request.remoteaddr=10.5.125.135 http.request.uri="/v2/gnome-characters/blobs/uploads/" http.request.useragent="python-requests/2.20.0" http.response.duration=17.169734ms http.response.status=202 http.response.written=0 instance.id=86dbe242-1e24-4664-963c-16733b06c50c version="v2.6.2+unknown"
Mar 23 03:50:58 oci-registry02 registry[776]: 10.5.126.9 - - [23/Mar/2019:03:50:58 +0000] "POST /v2/gnome-characters/blobs/uploads/ HTTP/1.1" 202 0 "" "python-requests/2.20.0"
Mar 23 03:50:58 oci-registry02 registry[776]: time="2019-03-23T03:50:58Z" level=info msg="response completed" go.version=go1.10rc2 http.request.contenttype="application/octet-stream" http.request.host=registry.fedoraproject.org http.request.id=23515e6c-a499-459c-be29-ad81726b2fef http.request.method=PUT http.request.remoteaddr=10.5.125.135 http.request.uri="/v2/gnome-characters/blobs/uploads/29f0e87b-174a-4429-b47e-ce5b6db306ff?_state=N1oq9531UyD6JldV5Izh4XLgfAdTG0nNvKiC9X6QyyJ7Ik5hbWUiOiJnbm9tZS1jaGFyYWN0ZXJzIiwiVVVJRCI6IjI5ZjBlODdiLTE3NGEtNDQyOS1iNDdlLWNlNWI2ZGIzMDZmZiIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAxOS0wMy0yM1QwMzo1MDo1OC41NjkwNDA3NzJaIn0%3D&digest=sha256:1c6c80856abf27262179163366e36059d5e47bbce19c256feb786a4170696abf" http.request.useragent="python-requests/2.20.0" http.response.duration=31.833467ms http.response.status=201 http.response.written=0 instance.id=86dbe242-1e24-4664-963c-16733b06c50c version="v2.6.2+unknown"
Mar 23 03:50:58 oci-registry02 registry[776]: 10.5.126.9 - - [23/Mar/2019:03:50:58 +0000] "PUT /v2/gnome-characters/blobs/uploads/29f0e87b-174a-4429-b47e-ce5b6db306ff?_state=N1oq9531UyD6JldV5Izh4XLgfAdTG0nNvKiC9X6QyyJ7Ik5hbWUiOiJnbm9tZS1jaGFyYWN0ZXJzIiwiVVVJRCI6IjI5ZjBlODdiLTE3NGEtNDQyOS1iNDdlLWNlNWI2ZGIzMDZmZiIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAxOS0wMy0yM1QwMzo1MDo1OC41NjkwNDA3NzJaIn0%3D&digest=sha256:1c6c80856abf27262179163366e36059d5e47bbce19c256feb786a4170696abf HTTP/1.1" 201 0 "" "python-requests/2.20.0"
Mar 23 03:50:58 oci-registry02 registry[776]: time="2019-03-23T03:50:58Z" level=info msg="response completed" go.version=go1.10rc2 http.request.host=registry.fedoraproject.org http.request.id=ab1257de-0a6c-4ff8-a1d6-b1c997872356 http.request.method=POST http.request.remoteaddr=10.5.125.135 http.request.uri="/v2/gnome-characters/blobs/uploads/" http.request.useragent="python-requests/2.20.0" http.response.duration=12.818875ms http.response.status=202 http.response.written=0 instance.id=86dbe242-1e24-4664-963c-16733b06c50c version="v2.6.2+unknown"
Mar 23 03:50:58 oci-registry02 registry[776]: 10.5.126.8 - - [23/Mar/2019:03:50:58 +0000] "POST /v2/gnome-characters/blobs/uploads/ HTTP/1.1" 202 0 "" "python-requests/2.20.0"

I guess that initial 404 is it checking that it's not already uploaded it? then it uploads (to 02 always per the config).

Looks like 01 got a _catalog request with gnome-characters in it on 01 about 5min later... 

Mar 23 03:55:02 oci-registry01 registry[725]: time="2019-03-23T03:55:02Z" level=info msg="response completed" go.version=go1.10rc2 http.request.host=registry.fedoraproject.org http.request.id=4f8d8cc5-e5cf-4b5f-a22b-6a95463f631a http.request.method=GET http.request.remoteaddr=10.5.126.38 http.request.uri="/v2/gnome-characters/tags/list" http.request.useragent="Go-http-client/2.0" http.response.contenttype="application/json; charset=utf-8" http.response.duration=5.712763ms http.response.status=200 http.response.written=84 instance.id=c22aeae6-9c22-4791-b3ae-f34ae787a90d version="v2.6.2+unknown"

I admit that there's a lot of log info, but the denseness makes them kinda hard to read. If you like I can put the logs from the 23rd up somewhere for you to poke through?
Edited 5 years ago by kevin

@otaylor Should we investigate this more? Has it happened again?

Metadata Update from @kevin:
- Issue priority set to: Waiting on Reporter (was: Needs Review)
4 years ago

Closing this for now, please re-open if you see it happen again or wish further debugging.

:beer:

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)
4 years ago

Login to comment on this ticket.

Metadata
None
None
None
None
Waiting on Reporter
Powered by Pagure 5.13.3
Documentation • File an Issue • About • SSH Hostkey/Fingerprint
© Red Hat, Inc. and others.