#7870 RFR: Deploying coreos-koji-tagger in Fedora Openshift
Opened a month ago by dustymabe. Modified 17 days ago

  • Software: coreos-koji-tagger
  • Advantage for Fedora: Aid in development of Fedora CoreOS
  • Sponsor: needed

I would like to request to deploy coreos-koji-tagger in Fedora Openshift.

The point of this service is to monitor input files in git repositories and tag packages into the coreos-pool tag when necessary. Currently the source code lives in https://github.com/coreos/fedora-coreos-koji-tagger. It will be moved to a more permanent location before deploying to production openshift.

More Info:
- https://github.com/coreos/fedora-coreos-tracker/issues/179
- https://github.com/coreos/fedora-coreos-tracker/issues/188

for future reference for me: https://fedora-infra-docs.readthedocs.io/en/latest/sysadmin-guide/sops/requestforresources.html#ticket-comment-template

@mizdebsk - are you willing to be a sponsor for me for this initiative ?

I'll sponsor this RFR. We can move to planning phase.
@dustymabe Please start discussion on infra list, if not already done.

Metadata Update from @mizdebsk:
- Issue assigned to mizdebsk
- Issue priority set to: Waiting on Reporter (was: Needs Review)

a month ago

I'll sponsor this RFR. We can move to planning phase.


@dustymabe Please start discussion on infra list, if not already done.

done: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/thread/MO2RF632PEXHLWCVQFYZGKFRKBVS7Q7B/

@mizdebsk - looks like there was no negative discussion on mailing list. What would you like me to do next?

RFR SOP has been updated in the meantime and it looks like there are extra steps required before staging instance can be created. I will familiarize myself with updated SOP and let you know.

@dustymabe The only remaining step before creating staging instance is determining who is going to be involved in maintenance of the app.

Metadata Update from @mizdebsk:
- Issue tagged with: request-for-resources

a month ago

@dustymabe The only remaining step before creating staging instance is determining who is going to be involved in maintenance of the app.

@mizdebsk - can we name me as well as anyone in the sysadmin-coreos group.

ACK. I hereby confirm that the resource is ready to move to the next phase - creating staging instance. @dustymabe Please commit Ansible roles and playbooks to ansible.git and let me know once done. I will grant you permission to run the playbook.

Thanks @mizdebsk. Sorry it took me a few days to get these set up. The files I'm proposing for ansible are in https://github.com/dustymabe/fedora-infra-ansible/pull/1. I have a few questions/comments:

  • Previously @kevin created a keytab for us that is associated with the coreos-bot FAS user for manipulating these tags using bots.
  • From other apps (like the bodhi one) I see that those actually call out to the openshift-keytab which calls out to the keytab/service role.
  • Which way should I go? Should we embed our keytab for coreos-bot into the ansible private vars and populate it that way or should we call openshift-keytab?
  • If we should call openshift-keytab can you help me set that up?

I was talking with patrick in IRC yesterday about compose-tracker and security audit and coreos-koji-tagger came up as well. Here are a few generic questions he asked and my answers:

1. what way is it deployed (i.e. does it just "pip install" on all its deps, or does it install deps from Fedora and then just install the code itself from git)
2. what access does it have
3. what is the worst that could happen, etc

My answers:

  1. all deps are from fedora + github source code for python script
  2. access to koji coreos-pool tag via keytab
  3. if someone got the keytab they could much with our coreos-pool tag - not ideal, but can be recovered from easily

@mizdebsk - can we make some progress on this this week?

We should use a different keytab, created by one of existing Ansible roles. The keytab can be linked to the existing coreos-bot user if desired.

ok - the code in that PR was pushed to ansible repo. Can we add permissions to run them and set it up in staging?

Permission granted for sysadmin-coreos members to run the playbook.

hey @mizdebsk - i'm seeing an issue using the keytab you provided:

2019-06-27 20:29:44,364 ERROR coreos_koji_tagger - COMMAND: ['/usr/bin/kinit', '-k', '-t', '/etc/coreos-koji-tagger-keytab/koji-keytab', 'coreos-koji-tagger/coreos-koji-tagger.stg.phx2.fedoraproject.org@STG.FEDORAPROJECT.ORG']
2019-06-27 20:29:44,364 ERROR coreos_koji_tagger -  STDOUT: None
2019-06-27 20:29:44,364 ERROR coreos_koji_tagger -  STDERR: None
kinit: Keytab contains no suitable keys for coreos-koji-tagger/coreos-koji-tagger.stg.phx2.fedoraproject.org@STG.FEDORAPROJECT.ORG while getting initial credentials

looks like it should have been: coreos-koji-tagger/coreos-koji-tagger.stg.fedoraproject.org@STG.FEDORAPROJECT.ORG

ok - now the username is right but I don't can't talk to koji:

2019-06-27 21:36:16,343 INFO coreos_koji_tagger - Authenticating with keytab: /etc/coreos-koji-tagger-keytab/koji-keytab
2019-06-27 21:36:16,852 ERROR coreos_koji_tagger - Running command returned bad exitcode
2019-06-27 21:36:16,853 ERROR coreos_koji_tagger - COMMAND: ['/usr/bin/koji', 'moshimoshi']
2019-06-27 21:36:16,853 ERROR coreos_koji_tagger -  STDOUT: b''
2019-06-27 21:36:16,853 ERROR coreos_koji_tagger - STDERR: b'2019-06-27 21:36:16,824 [ERROR] koji: AuthError: unable to obtain a session\n'

I know @mizdebsk mentioned that the user would need to be granted permissions. Is that the problem here? Can you add the same permissions to the user as the coreos-bot user has?

@mizdebsk can you create the coreos_koji_tagger_webhook_secret ansible var too and pass that to me in a private message?

he created the coreos_koji_tagger_webhook_secret_prod and coreos_koji_tagger_webhook_secret_stg variables for me today and I updated the roles to use them.

current status: still having this issue: https://pagure.io/fedora-infrastructure/issue/7870#comment-579894

Are you sure you are connecting to staging koji with tthe stg keytab? ie, stg-koji or koji -p staging ?

there was a bug where I wasn't using stg-koji.. but i just updated it and added debug output. Looks like I still have the same problem:

2019-06-28 22:15:54,714 INFO coreos_koji_tagger - Authenticating with keytab: /etc/coreos-koji-tagger-keytab/koji-keytab
2019-06-28 22:15:54,714 DEBUG coreos_koji_tagger - Running command: ['/usr/bin/klist', '-k', '/etc/coreos-koji-tagger-keytab/koji-keytab']
2019-06-28 22:15:54,720 DEBUG coreos_koji_tagger - Found principal coreos-koji-tagger/coreos-koji-tagger.stg.fedoraproject.org@STG.FEDORAPROJECT.ORG in keytab
2019-06-28 22:15:54,720 INFO coreos_koji_tagger - Using principal coreos-koji-tagger/coreos-koji-tagger.stg.fedoraproject.org@STG.FEDORAPROJECT.ORG
2019-06-28 22:15:54,720 DEBUG coreos_koji_tagger - Running command: ['/usr/bin/kinit', '-k', '-t', '/etc/coreos-koji-tagger-keytab/koji-keytab', 'coreos-koji-tagger/coreos-koji-tagger.stg.fedoraproject.org@STG.FEDORAPROJECT.ORG']
2019-06-28 22:15:54,860 DEBUG coreos_koji_tagger - Running command: ['/usr/bin/stg-koji', 'moshimoshi']
2019-06-28 22:15:55,235 ERROR coreos_koji_tagger - Running command returned bad exitcode
2019-06-28 22:15:55,236 ERROR coreos_koji_tagger - COMMAND: ['/usr/bin/stg-koji', 'moshimoshi']
2019-06-28 22:15:55,236 ERROR coreos_koji_tagger -  STDOUT: b''
2019-06-28 22:15:55,236 ERROR coreos_koji_tagger -  STDERR: b'2019-06-28 22:15:55,208 [ERROR] koji: AuthError: unable to obtain a session\n'
Traceback (most recent call last):
  File "/usr/bin/fedora-messaging", line 11, in <module>
    load_entry_point('fedora-messaging==1.7.0', 'console_scripts', 'fedora-messaging')()
  File "/usr/lib/python3.7/site-packages/click/core.py", line 763, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3.7/site-packages/click/core.py", line 716, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3.7/site-packages/click/core.py", line 1136, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3.7/site-packages/click/core.py", line 955, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3.7/site-packages/click/core.py", line 554, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3.7/site-packages/fedora_messaging/cli.py", line 144, in consume
    callback, bindings=bindings, queues=queues
  File "/usr/lib/python3.7/site-packages/fedora_messaging/api.py", line 108, in twisted_consume
    callback = _check_callback(callback)
  File "/usr/lib/python3.7/site-packages/fedora_messaging/api.py", line 51, in _check_callback
    callback_object = callback()
  File "/usr/lib/python3.7/site-packages/coreos_koji_tagger.py", line 117, in __init__
  File "/usr/lib/python3.7/site-packages/coreos_koji_tagger.py", line 274, in kinit
    check_koji_connection(check=True) # Make sure it works
  File "/usr/lib/python3.7/site-packages/coreos_koji_tagger.py", line 463, in check_koji_connection
    cp = runcmd(cmd, check=check, capture_output=True)
  File "/usr/lib/python3.7/site-packages/coreos_koji_tagger.py", line 279, in runcmd
    cp = subprocess.run(cmd, **kwargs)
  File "/usr/lib64/python3.7/subprocess.py", line 487, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/usr/bin/stg-koji', 'moshimoshi']' returned non-zero exit status 1.

thought as as sanity check can someone confirm that stg-koji moshimoshi works? I'm using moshimoshi as a check that we can talk to the remote koji instance but it looks like that is failing for stage, where it works for prod.

though as as sanity check can someone confirm that stg-koji moshimoshi works?

just checked, it does work for my user (dustymabe) if I kinit to stg first. From the debug output above you can see that I kinit first and then run moshimoshi. This should work.

According to kevin it turns out that user doesn't exist in stage koji.

Login to comment on this ticket.