It looks like all builds in mbs.stg.fedoraproject.org are failing with - 'No credentials cache found'. See, e.g: https://mbs.stg.fedoraproject.org/module-build-service/1/module-builds/234
This is apparently authentication of MBS to koji. My guess is that the credentials cache got cleaned out of /tmp, and there is manual or once-on-boot step used to kinit. (I don't see a keytab in the configuration of MBS.)
@jkaluza @mprahl do you have any insight into what is going on here?
@ralph I think you were the one to set this up. Do you have access to the keytab to generate the Kerberos cache file?
I'll bounce fedmsg-hub to see if that fixes it as @oytalor suggested.
Got the same error:
May 30 19:38:01 mbs-backend01.stg.phx2.fedoraproject.org fedmsg-hub[5528]: [2018-05-30 19:38:01][module_build_service INFO] Connecting to koji 'https://koji.stg.fedoraproject.org/kojihub' with 'kerberos'. May 30 19:38:01 mbs-backend01.stg.phx2.fedoraproject.org fedmsg-hub[5528]: [2018-05-30 19:38:01][module_build_service ERROR] Could not process message handler. See the traceback. May 30 19:38:01 mbs-backend01.stg.phx2.fedoraproject.org fedmsg-hub[5528]: Traceback (most recent call last): May 30 19:38:01 mbs-backend01.stg.phx2.fedoraproject.org fedmsg-hub[5528]: File "/usr/lib/python2.7/site-packages/module_build_service/scheduler/consumer.py", line 240, in process_message May 30 19:38:01 mbs-backend01.stg.phx2.fedoraproject.org fedmsg-hub[5528]: further_work = handler(conf, session, msg) or [] May 30 19:38:01 mbs-backend01.stg.phx2.fedoraproject.org fedmsg-hub[5528]: File "/usr/lib/python2.7/site-packages/module_build_service/scheduler/handlers/modules.py", line 274, in wait May 30 19:38:01 mbs-backend01.stg.phx2.fedoraproject.org fedmsg-hub[5528]: session, build, config) May 30 19:38:01 mbs-backend01.stg.phx2.fedoraproject.org fedmsg-hub[5528]: File "/usr/lib/python2.7/site-packages/module_build_service/builder/base.py", line 149, in create_from_module May 30 19:38:01 mbs-backend01.stg.phx2.fedoraproject.org fedmsg-hub[5528]: components=components) May 30 19:38:01 mbs-backend01.stg.phx2.fedoraproject.org fedmsg-hub[5528]: File "/usr/lib/python2.7/site-packages/module_build_service/builder/base.py", line 130, in create May 30 19:38:01 mbs-backend01.stg.phx2.fedoraproject.org fedmsg-hub[5528]: config=config, **extra) May 30 19:38:01 mbs-backend01.stg.phx2.fedoraproject.org fedmsg-hub[5528]: File "/usr/lib/python2.7/site-packages/module_build_service/utils/general.py", line 183, in wrapper May 30 19:38:01 mbs-backend01.stg.phx2.fedoraproject.org fedmsg-hub[5528]: return function(*args, **kwargs) May 30 19:38:01 mbs-backend01.stg.phx2.fedoraproject.org fedmsg-hub[5528]: File "/usr/lib/python2.7/site-packages/module_build_service/builder/KojiModuleBuilder.py", line 170, in __init__ May 30 19:38:01 mbs-backend01.stg.phx2.fedoraproject.org fedmsg-hub[5528]: self.koji_session = self.get_session(config, owner) May 30 19:38:01 mbs-backend01.stg.phx2.fedoraproject.org fedmsg-hub[5528]: File "/usr/lib/python2.7/site-packages/module_build_service/utils/general.py", line 76, in inner May 30 19:38:01 mbs-backend01.stg.phx2.fedoraproject.org fedmsg-hub[5528]: return function(*args, **kwargs) May 30 19:38:01 mbs-backend01.stg.phx2.fedoraproject.org fedmsg-hub[5528]: File "/usr/lib/python2.7/site-packages/module_build_service/builder/KojiModuleBuilder.py", line 374, in get_session May 30 19:38:01 mbs-backend01.stg.phx2.fedoraproject.org fedmsg-hub[5528]: ccache=ccache May 30 19:38:01 mbs-backend01.stg.phx2.fedoraproject.org fedmsg-hub[5528]: File "/usr/lib/python2.7/site-packages/koji/__init__.py", line 2157, in krb_login May 30 19:38:01 mbs-backend01.stg.phx2.fedoraproject.org fedmsg-hub[5528]: ccache.init(cprinc) May 30 19:38:01 mbs-backend01.stg.phx2.fedoraproject.org fedmsg-hub[5528]: Krb5Error: (-1765328189, 'No credentials cache found')
Looks like the issue was that /var/cache/fedmsg/ was not present. Will fix in ansible.
/var/cache/fedmsg/
https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=66eecf8b14bbe24d44f9e90e329b0e9cd37f4a7a
Metadata Update from @ralph: - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.