I can see this is because it cannot auth to koji, but I cannot for the life of me figure out why.

I had it make a new keytab, but that didn't change anything.

The error(s) are:

Jul 23 21:15:36 autosign01.stg.phx2.fedoraproject.org fedmsg-hub[8632]: [2019-07-23 21:15:36][m
oksha.hub   ERROR] Failed to init <class 'robosignatory.tagconsumer.TagSignerConsumer'> consume
r.
Jul 23 21:15:36 autosign01.stg.phx2.fedoraproject.org fedmsg-hub[8632]: Traceback (most recent 
call last):
Jul 23 21:15:36 autosign01.stg.phx2.fedoraproject.org fedmsg-hub[8632]: File "/usr/lib/python2.
7/site-packages/moksha/hub/hub.py", line 410, in __init_consumers
Jul 23 21:15:36 autosign01.stg.phx2.fedoraproject.org fedmsg-hub[8632]: c = c_class(self)
Jul 23 21:15:36 autosign01.stg.phx2.fedoraproject.org fedmsg-hub[8632]: File "/usr/lib/python2.
7/site-packages/robosignatory/tagconsumer.py", line 51, in __init__
Jul 23 21:15:36 autosign01.stg.phx2.fedoraproject.org fedmsg-hub[8632]: client.krb_login(**kwar
gs)
Jul 23 21:15:36 autosign01.stg.phx2.fedoraproject.org fedmsg-hub[8632]: File "/usr/lib/python2.
7/site-packages/koji/__init__.py", line 2189, in krb_login
Jul 23 21:15:36 autosign01.stg.phx2.fedoraproject.org fedmsg-hub[8632]: ccache.init_creds_keyta
b(principal=cprinc, keytab=keytab)
Jul 23 21:15:36 autosign01.stg.phx2.fedoraproject.org fedmsg-hub[8632]: Krb5Error: (-1765328174
, 'Generic preauthentication failure')

@puiterwijk can you look?

@puiterwijk I know you are in Robosignatory headspace at the moment, could you divert a few minutes to check on this and see what could be the cause of it. This is blocking our Rawhide Gating push for tomorrow as we can no longer test in staging

This was caused by the keytab not being readable by the fedmsg user, and has now been fixed.
A test sign and tag move of fedora-gather-easyfix-0.1.1-21.fc31 just finished.

Great, thanks! :)

I'm going to run my script and close this ticket if it passes :)

There is still an issue with sigul after an extended period of runtime without activity in staging, but I restarted the bridge and server for now to avoid that.
This isn't a sigul bug, but instead just the firewall stopping to remember the connection after a long idle time.
I think we worked around this in production with some static firewall rules that allow the connection to be idle longer/indefinitely.
That should be a different ticket, since it should remain working for a bit now.

hm, I've done two builds and both are still in f31-updates-candidate: fedora-gather-easyfix-0.1.1-24.fc31 and fedora-gather-easyfix-0.1.1-25.fc31

could you check the logs?

INFO] Build fedora-gather-easyfix-0.1.1-25.fc31 (1229897) tagged into f31-updates-candidate on primary
 INFO] Koji instance not known, skipping

When it had been started, it didn't finish reading its configuration before it crashed, so it wasn't aware of what to sign.
I just restarted it, and it's now reporting TagSignerConsumer ready for service.

So please retry now.

fedora-gather-easyfix-0.1.1-26.fc31 building :)

Tags: f31-updates-testing-pending

Thanks! :)

@pingou Great! So we can close this ticket?

Yup confirmed it's all good now.

Thanks again for your help! :)