#10789 monitor-gating app fails kinit
Closed: Fixed with Explanation 2 years ago by kevin. Opened 2 years ago by kevin.

Since moving to ocp4, the monitor-gating app is unable to authenticate. It gets a kerberos error.

It might be that we need new keytabs, or something else in the ocp4 env is breaking it.

Running in production
# pkgs.fedoraproject.org:22 SSH-2.0-OpenSSH_8.0
# pkgs.fedoraproject.org:22 SSH-2.0-OpenSSH_8.0
# pkgs.fedoraproject.org:22 SSH-2.0-OpenSSH_8.0
# pkgs.fedoraproject.org:22 SSH-2.0-OpenSSH_8.0
# pkgs.fedoraproject.org:22 SSH-2.0-OpenSSH_8.0
Keytab name: FILE:/etc/keytabs/monitor-gating-keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   1 06/27/22 15:10:32 packagerbot/os-control01.iad2.fedoraproject.org@FEDORAPROJECT.ORG
   1 06/27/22 15:10:32 packagerbot/os-control01.iad2.fedoraproject.org@FEDORAPROJECT.ORG
   1 06/27/22 15:10:32 packagerbot/os-control01.iad2.fedoraproject.org@FEDORAPROJECT.ORG
Command `kinit packagerbot/os-master02.iad2.fedoraproject.org@FEDORAPROJECT.ORG -kt /etc/keytabs/monitor-gating-keytab` return code: `1`
stdout:
-------
b''
stderr:
-------
b'kinit: Keytab contains no suitable keys for packagerbot/os-master02.iad2.fedoraproject.org@FEDORAPROJECT.ORG while getting initial credentials\n'
Namespace(conf='/opt/config/runner.cfg')
{'delay': 3600, 'delay_when_failing': 43200, 'single_run': False, 'kb_principal': 'packagerbot/os-master02.iad2.fedoraproject.org@FEDORAPROJECT.ORG', 'kb_keytab_file': '/etc/keytabs/monitor-gating-keytab', 'workflow_single_gating_args': '--conf /opt/config/monitor_gating.cfg --auto-update --no-pr', 'workflow_multi_gating_args': '', 'pagure_blocking_project': 'packager-workflow', 'pagure_report_project': 'packager-workflow', 'env': 'prod', 'fedpkg': 'fedpkg', 'blocker_tags': ['packager_workflow_blocker'], 'pagure_api_token': '3RSTXYTPBQ6ANIZ3MBLF6WL508BG71C3RE8FPKJ9IDBHZQN9DLHM89TLLC6C5FEW'}
Logging into kerberos using: /etc/keytabs/monitor-gating-keytab
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/monitor_gating-0.0.1-py3.9.egg/monitor_gating/utils.py", line 803, in run_command
    output = subprocess.check_output(command, cwd=cwd, stderr=subprocess.PIPE)
  File "/usr/lib64/python3.9/subprocess.py", line 424, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib64/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['kinit', 'packagerbot/os-master02.iad2.fedoraproject.org@FEDORAPROJECT.ORG', '-kt', '/etc/keytabs/monitor-gating-keytab']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/monitor-gating", line 33, in <module>
    sys.exit(load_entry_point('monitor-gating==0.0.1', 'console_scripts', 'monitor-gating')())
  File "/usr/local/lib/python3.9/site-packages/monitor_gating-0.0.1-py3.9.egg/monitor_gating/runner.py", line 210, in main
    scheduler.run()
  File "/usr/lib64/python3.9/sched.py", line 151, in run
    action(*argument, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/monitor_gating-0.0.1-py3.9.egg/monitor_gating/runner.py", line 74, in schedule
    run_command(cmd)
  File "/usr/local/lib/python3.9/site-packages/monitor_gating-0.0.1-py3.9.egg/monitor_gating/utils.py", line 811, in run_command
    raise MonitoringException("Command failed to run")
monitor_gating.utils.MonitoringException: Command failed to run

production is now working...

staging isn't.

Metadata Update from @phsmoura:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: low-gain, low-trouble, ops

2 years ago

This part is now working in staging. The entire app in staging is not working due to a ssh issue, but this thing is solved.

I repointed the app to the production branch.

Metadata Update from @kevin:
- Issue close_status updated to: Fixed with Explanation
- Issue status updated to: Closed (was: Open)

2 years ago

Login to comment on this ticket.

Metadata
Boards 1
ops Status: Backlog