fedora-infra / toddlers

Clone
Source Code
GIT

#245 koji_block_retired: Add kerberos authentication for koji
Closed 2 months ago by lenkaseg. Opened 2 months ago by lenkaseg.
fedora-infra/ lenkaseg/toddlers koji_retired_packages  into  production

file added
+3 
@@ -0,0 +1,3 @@ 
 

+ UPGRADE_PIP_TO_LATEST=true
 

+ ENABLE_MICROPIPENV=true
 

+ APP_SCRIPT=.s2i/run.sh

file added
+3 
@@ -0,0 +1,3 @@ 
 

+ #!/bin/sh
 

+ 

+ exec fedora-messaging consume

file modified
+21 
@@ -11,12 +11,29 @@ 
 

                  - libmodulemd
 

                  - python3.12
 

                  - python3.12-devel
 

+                 - poetry
 

          - fi-tox-format:
 

              vars:
 

                tox_envlist: black
 

+               dependencies:
 

+                 - cairo-devel
 

+                 - cairo-gobject-devel
 

+                 - gobject-introspection-devel
 

+                 - libmodulemd
 

+                 - python3.12
 

+                 - python3.12-devel
 

+                 - poetry
 

          - fi-tox-lint:
 

              vars:
 

                tox_envlist: flake8
 

+               dependencies:
 

+                 - cairo-devel
 

+                 - cairo-gobject-devel
 

+                 - gobject-introspection-devel
 

+                 - libmodulemd
 

+                 - python3.12
 

+                 - python3.12-devel
 

+                 - poetry
 

          - fi-tox-python39:
 

              vars:
 

                dependencies:
 
@@ -24,6 +41,7 @@ 
 

                  - cairo-gobject-devel
 

                  - gobject-introspection-devel
 

                  - libmodulemd
 

+                 - poetry
 

          - fi-tox-python310:
 

              vars:
 

                dependencies:
 
@@ -37,6 +55,7 @@ 
 

                  - python3-pluggy
 

                  - python3-py
 

                  - python3-toml
 

+                 - poetry
 

          - fi-tox-python311:
 

              vars:
 

                dependencies:
 
@@ -49,6 +68,7 @@ 
 

                  - python3-pluggy
 

                  - python3-py
 

                  - python3-toml
 

+                 - poetry
 

          - fi-tox-python312:
 

              vars:
 

                dependencies:
 
@@ -62,4 +82,5 @@ 
 

                  - python3-pluggy
 

                  - python3-py
 

                  - python3-toml
 

+                 - poetry
 

  ...

file modified
+1 -1 
@@ -27,7 +27,7 @@ 
 

  

  

  COPY . /code
 

- RUN cd /code && pip install -r requirements.txt
 

+ RUN cd /code && pip install .
 

  #ENTRYPOINT ["/code/entrypoint.sh"]
 

  ENV PYTHONPATH=/code
 

  CMD ["fedora-messaging", "consume"]

file added
+2566
The added file is too large to be shown here, see it at: poetry.lock
file added
+46 
@@ -0,0 +1,46 @@ 
 

+ [tool.poetry]
 

+ name = "toddlers"
 

+ version = "0.1.0"
 

+ description = "Small fedora-messaging toddlers that keep running around"
 

+ authors = ["Pierre-Yves Chibon <pingou@pingoured.fr>"]
 

+ license = "GPL-2.0-or-later"
 

+ readme = "README.rst"
 

+ 

+ [tool.poetry.dependencies]
 

+ python = "^3.9"
 

+ arrow = "^1.3.0"
 

+ beanbag = "^1.9.2"
 

+ bs4 = "^0.0.2"
 

+ defusedxml = "^0.7.1"
 

+ fasjson-client = "^1.0.8"
 

+ fedora-messaging = "^3.6.0"
 

+ fedora-messaging-git-hook-messages = "^1.0.1"
 

+ gitpython = "^3.1.43"
 

+ koji = "^1.34.2"
 

+ requests = "^2.32.3"
 

+ noggin-messages = "^1.1.0"
 

+ pagure-messages = "^1.2.0"
 

+ pygobject = "^3.48.2"
 

+ python-fedora = "^1.1.1"
 

+ python-bugzilla = ">=3.2.0"
 

+ pdc-client = "^1.8.0"
 

+ zstandard = "^0.23.0"
 

+ 

+ 

+ [tool.poetry.group.dev.dependencies]
 

+ pytest = "^8.2.2"
 

+ pytest-cov = "^5.0.0"
 

+ types-requests = "^2.32.0.20240622"
 

+ types-toml = "^0.10.8.20240310"
 

+ mypy = "^1.10.1"
 

+ flake8 = "^7.1.0"
 

+ flake8-import-order = "^0.18.2"
 

+ black = ">=24.4.2"
 

+ diff-cover = "^9.1.1"
 

+ 

+ [tool.poetry.scripts]
 

+ toddlers-playtime = "toddlers.playtime:main"
 

+ 

+ [build-system]
 

+ requires = ["poetry-core"]
 

+ build-backend = "poetry.core.masonry.api"

file removed
-17 
@@ -1,17 +0,0 @@ 
 

- arrow
 

- beanbag
 

- bs4
 

- defusedxml
 

- fasjson-client
 

- fedora-messaging
 

- fedora-messaging-git-hook-messages
 

- GitPython
 

- koji
 

- requests
 

- noggin-messages
 

- pagure-messages
 

- pyGObject
 

- python-fedora
 

- python-bugzilla>=3.2.0
 

- pdc-client
 

- zstandard

file removed
-6 
@@ -1,6 +0,0 @@ 
 

- pytest
 

- pytest-cov
 

- 

- # Typing libraries
 

- types-requests
 

- types-toml

file modified
+39 -12 
@@ -3,7 +3,7 @@ 
 

  """
 

  

  import logging
 

- from unittest.mock import MagicMock
 

+ from unittest.mock import MagicMock, patch
 

  

  import koji
 

  import pytest
 
@@ -71,6 +71,8 @@ 
 

          """Initialize toddler."""
 

          self.toddler_cls = koji_block_retired.KojiBlockRetired()
 

          self.toddler_cls.koji_session = MagicMock()
 

+         self.toddler_cls.get_rawhide_tag = MagicMock()
 

+         self.toddler_cls.get_rawhide_tag.return_value = "f41"
 

  

      def test_no_dead_package_file(self, caplog):
 

          """
 
@@ -80,7 +82,11 @@ 
 

          message = MagicMock()
 

          message.body = {"commit": {"stats": {"files": {}}}}
 

          config = MagicMock()
 

-         config = {"koji_url": "https://example.koji.org"}
 

+         config = {
 

+             "koji_url": "https://example.koji.org",
 

+             "principal": "principal",
 

+             "keytab": "keytab",
 

+         }
 

          self.toddler_cls.process_block_retired(config, message)
 

          assert caplog.records[-1].message == "No dead.package in the commit, bailing"
 

  
@@ -96,7 +102,11 @@ 
 

              }
 

          }
 

          config = MagicMock()
 

-         config = {"koji_url": "https://example.koji.org"}
 

+         config = {
 

+             "koji_url": "https://example.koji.org",
 

+             "principal": "principal",
 

+             "keytab": "keytab",
 

+         }
 

          self.toddler_cls.process_block_retired(config, message)
 

          assert caplog.records[-1].message == "dead.package file was not added, bailing"
 

  
@@ -116,8 +126,15 @@ 
 

              }
 

          }
 

          config = MagicMock()
 

-         config = {"koji_url": "https://example.koji.org"}
 

-         self.toddler_cls.process_block_retired(config, message)
 

+         config = {
 

+             "koji_url": "https://example.koji.org",
 

+             "principal": "principal",
 

+             "keytab": "keytab",
 

+         }
 

+         with patch.object(
 

+             self.toddler_cls, "get_rawhide_tag", MagicMock(return_value="f41")
 

+         ):
 

+             self.toddler_cls.process_block_retired(config, message)
 

          self.toddler_cls.koji_session.packageListBlock.assert_called_once_with(
 

              taginfo="f41",
 

              pkginfo="example-repo",
 
@@ -138,7 +155,11 @@ 
 

              }
 

          }
 

          config = MagicMock()
 

-         config = {"koji_url": "https://example.koji.org"}
 

+         config = {
 

+             "koji_url": "https://example.koji.org",
 

+             "principal": "principal",
 

+             "keytab": "keytab",
 

+         }
 

          self.toddler_cls.process_block_retired(config, message)
 

          self.toddler_cls.koji_session.packageListBlock.assert_called_once_with(
 

              taginfo="f38",
 
@@ -159,12 +180,15 @@ 
 

              }
 

          }
 

          config = MagicMock()
 

-         config = {"koji_url": "https://example.koji.org"}
 

+         config = {
 

+             "koji_url": "https://example.koji.org",
 

+             "principal": "principal",
 

+             "keytab": "keytab",
 

+         }
 

          self.toddler_cls.koji_session.packageListBlock.side_effect = koji.GenericError(
 

              "Failed"
 

          )
 

-         with pytest.raises(koji.GenericError):
 

-             self.toddler_cls.process_block_retired(config, message)
 

+         self.toddler_cls.process_block_retired(config, message)
 

          assert (
 

              caplog.records[-1].message
 

              == "Failed to block retired package example-repo on branch f38: Failed"
 
@@ -184,12 +208,15 @@ 
 

              }
 

          }
 

          config = MagicMock()
 

-         config = {"koji_url": "https://example.stg.koji.org"}
 

+         config = {
 

+             "koji_url": "https://example.koji.org",
 

+             "principal": "principal",
 

+             "keytab": "keytab",
 

+         }
 

          self.toddler_cls.koji_session.packageListBlock.side_effect = koji.GenericError(
 

              "Failed"
 

          )
 

-         with pytest.raises(koji.GenericError):
 

-             self.toddler_cls.process_block_retired(config, message)
 

+         self.toddler_cls.process_block_retired(config, message)
 

          assert (
 

              caplog.records[-1].message
 

              == "Failed to block retired package example-epel-repo on branch f38: Failed"

file modified
+44 -1 
@@ -1,8 +1,31 @@ 
 

+ import logging
 

+ from unittest import mock
 

+ 

+ import fedora_messaging.api
 

+ import pytest
 

+ 

  from toddlers.base import ToddlerBase
 

  

  

+ class DummyToddlerBase(ToddlerBase):
 

+ 

+     @property
 

+     def name(self):
 

+         return super().name
 

+ 

+     def accepts_topic(self, topic):
 

+         return super().accepts_topic(topic)
 

+ 

+     @property
 

+     def amqp_topics(self):
 

+         return super().amqp_topics
 

+ 

+     def process(self, *args, **kwargs):
 

+         return super().process(*args, **kwargs)
 

+ 

+ 

  class TestToddlerBase:
 

-     toddler_cls = ToddlerBase
 

+     toddler_cls = DummyToddlerBase
 

  

      def test_name(self, toddler):
 

          assert toddler.name == "base"
 
@@ -15,3 +38,23 @@ 
 

  

      def test_process(self, toddler):
 

          assert toddler.process(config={}, message={}) is None
 

+ 

+     @mock.patch.object(
 

+         DummyToddlerBase,
 

+         "process",
 

+         new=mock.Mock(side_effect=Exception("haha")),
 

+     )
 

+     def test___call__exception(self, caplog):
 

+         caplog.set_level(logging.INFO)
 

+         msg = fedora_messaging.api.Message()
 

+         msg.id = 123
 

+         msg.topic = "foo.bar"
 

+         runner = DummyToddlerBase()
 

+         with pytest.raises(
 

+             fedora_messaging.exceptions.Nack, match=r"A toddler tripped"
 

+         ):
 

+             runner(msg)
 

+         assert (
 

+             caplog.records[-1].message
 

+             == "Toddler 'base' failed to process message id: 123 -- putting it back in the queue"
 

+         )

file modified
+4 
@@ -85,6 +85,10 @@ 
 

  # Base URL for the Koji build system
 

  koji_url = "https://koji.fedoraproject.org"
 

  

+ # Credentials for koji session
 

+ principal = "insert_principal"
 

+ keytab = "path_to_file"
 

+ 

  # Base URL for the Koji package db
 

  kojipkgs_url = "https://kojipkgs.fedoraproject.org"

file modified
+45 -2 
@@ -1,8 +1,40 @@ 
 

  import abc
 

+ import logging
 

  

+ import fedora_messaging.config
 

+ import fedora_messaging.exceptions
 

  

- class ToddlerBase(object):
 

-     __metaclass__ = abc.ABCMeta
 

+ from .utils.misc import merge_dicts
 

+ 

+ 

+ _log = logging.getLogger(__name__)
 

+ 

+ 

+ def _get_toddler_config(toddler_name: str) -> dict:
 

+     """Merge default and private configuration for a toddler."""
 

+     base_config = fedora_messaging.config.conf["consumer_config"]
 

+ 

+     default_config = base_config.get("default", {})
 

+     private_config = base_config.get(toddler_name, {})
 

+ 

+     return merge_dicts(default_config, private_config)
 

+ 

+ 

+ class ToddlerBase(metaclass=abc.ABCMeta):
 

+ 

+     def __init__(self):
 

+         self.reset_routing_keys()
 

+         self._config = _get_toddler_config(self.name)
 

+ 

+     def reset_routing_keys(self):
 

+         # Only for dev/debug: in Fedora infra all bindings are set in ansible
 

+         for binding in fedora_messaging.config.conf["bindings"]:
 

+             if binding["routing_keys"] == ["#"]:
 

+                 _log.info(
 

+                     "Updating the routing_keys of the queue %s to the topics of interest",
 

+                     binding["queue"],
 

+                 )
 

+                 binding["routing_keys"] = self.amqp_topics
 

  

      @property
 

      @abc.abstractmethod
 
@@ -34,3 +66,14 @@ 
 

      def process(self, config, message):
 

          """Process a given message."""
 

          return
 

+ 

+     def __call__(self, message):
 

+         try:
 

+             return self.process(self._config, message)
 

+         except Exception:
 

+             _log.exception(
 

+                 "Toddler '%s' failed to process message id: %s -- putting it back in the queue",
 

+                 self.name,
 

+                 message.id,
 

+             )
 

+             raise fedora_messaging.exceptions.Nack("A toddler tripped")

file modified
+2 -2 
@@ -68,7 +68,7 @@ 
 

      return parser.parse_args(args)
 

  

  

- def main(args=[]):
 

+ def main(args=None):
 

      """Main function of the playtime script"""
 

      args = get_arguments(args)
 

      setup_logging(log_level=args.log_level)
 
@@ -100,4 +100,4 @@ 
 

  

  

  if __name__ == "__main__":  # pragma: no cover
 

-     sys.exit(main(sys.argv[1:]))
 

+     sys.exit(main())

file modified
+9 -6 
@@ -35,9 +35,10 @@ 
 

      def __init__(self):
 

          self.koji_session = None
 

  

-     def _create_session(self, koji_url):
 

+     def _create_session(self, koji_url, principal, keytab):
 

          """Makes a koji session, that handles logging in"""
 

          self.koji_session = koji.ClientSession(koji_url)
 

+         self.koji_session.gssapi_login(principal=principal, keytab=keytab)
 

  

      def get_rawhide_tag(self):
 

          releases = []
 
@@ -70,7 +71,9 @@ 
 

          """
 

          msg = message.body
 

  

-         koji_url = config["koji_url"]
 

+         koji_url = f"{config['koji_url']}/kojihub"
 

+         principal = config["principal"]
 

+         keytab = config["keytab"]
 

  

          # If there is no dead.package file in commit, then it can be ignored
 

          if "dead.package" not in msg["commit"]["stats"]["files"]:
 
@@ -97,7 +100,7 @@ 
 

          _log.info("Processing Koji block retired for %s", repo)
 

  

          if self.koji_session is None:
 

-             self._create_session(koji_url)
 

+             self._create_session(koji_url, principal, keytab)
 

  

          # Untag builds first due to koji/mash bug:
 

          # https://fedorahosted.org/koji/ticket/299
 
@@ -108,7 +111,6 @@ 
 

              self.koji_session.untagBuild(tag=branch_name, build=repo)
 

          except koji.GenericError as e:
 

              _log.exception(f"Failed to untag build in koji: {e}")
 

-             raise
 

  

          try:
 

              _log.info(f"Blocking package {repo}, tag: {branch_name}")
 
@@ -117,7 +119,9 @@ 
 

              _log.exception(
 

                  f"Failed to block retired package {repo} on branch {branch_name}: {e}"
 

              )
 

-             raise
 

+ 

+         # TODO: In case of exception while blocking a package in koji,
 

+         # send email with the list of packages that failed to block
 

  

          # If a package moves from EPEL to RHEL it can only be built if it is unblocked
 

          # in the epel build tag. Therefore unblock all retired EPEL packages in the
 
@@ -134,7 +138,6 @@ 
 

                      )
 

                  except koji.GenericError as e:
 

                      _log.exception(f"Failed unblocking epel build tag: {e}")
 

-                     raise
 

  

  

  def get_arguments(args):

file modified
+13 -25 
@@ -3,39 +3,27 @@ 
 

  # If the user is missing an interpreter, don't fail
 

  skip_missing_interpreters = True
 

  skipsdist = True
 

+ isolated_build = true
 

  

  [testenv]
 

- deps =
 

-     -r requirements.txt
 

-     -r test-requirements.txt
 

- sitepackages = True
 

- setenv =
 

-     PYTHONPATH={toxinidir}
 

+ passenv = HOME
 

+ sitepackages = false
 

+ skip_install = true
 

+ allowlist_externals =
 

+     poetry
 

+ commands_pre =
 

+     poetry install --all-extras
 

  commands =
 

-     pytest {posargs}
 

+     poetry run pytest -vv {posargs}
 

+     poetry run diff-cover --compare-branch origin/main --fail-under 100 coverage.xml
 

  

  [testenv:black]
 

- deps =
 

-     black
 

- sitepackages = False
 

  commands =
 

-     black --check --diff .
 

+     poetry run black --check --diff .
 

  

  [testenv:mypy]
 

  basepython = python3.12
 

- allowlist_externals =
 

-     mypy
 

- deps =
 

-     {[testenv]deps}
 

-     mypy
 

- setenv =
 

-     {[testenv]setenv}
 

- commands = mypy --config-file {toxinidir}/mypy.cfg toddlers tests
 

+ commands = poetry run mypy --config-file {toxinidir}/mypy.cfg toddlers tests
 

  

  [testenv:flake8]
 

- deps =
 

-     flake8
 

-     flake8-import-order
 

- sitepackages = False
 

- commands =
 

-     flake8 --ignore=W503 toddlers/ tests/ {posargs}
 

+ commands = poetry run flake8 --ignore=W503 toddlers/ tests/ {posargs}

no initial comment

Let's make koji_block_retired run in production (the last 4 commits related to koji_block_retired)
Changes confirmed to work in staging.

Build failed. More information on how to proceed and troubleshoot errors available at https://fedoraproject.org/wiki/Zuul-based-ci
https://fedora.softwarefactory-project.io/zuul/buildset/c68f57b53f96402a8643de3f0abfadb3

This is probably missing the fix I introduced in #243

one badly written test failing after the mass branching :) fixing

1 new commit added

  • koji_block_retired: Mock get_rawhide_tag
2 months ago

9 new commits added

  • koji_block_retired: Mock get_rawhide_tag
  • Add diff-cover
  • Use Poetry for requirements and project management
  • Make the toddlers more independant (teenagers)
  • Make the `test_dead_package_added_to_main_branch` test more stable
  • koji_block_retired: Add kerberos authentication for koji
  • koji_block_retired: Fix koji_url
  • koji_block_retired: fix tests
  • koji_block_retired: do not raise an error when tag not present in koji
2 months ago

Pull-Request has been closed by lenkaseg
2 months ago

Build succeeded.
https://fedora.softwarefactory-project.io/zuul/buildset/f75f1d76c00f43c6bce80b96c14b5134
Metadata
None
No Tags
Changes Summary 15
+3
file added
.s2i/environment
+3
file added
.s2i/run.sh
+21 -0
file changed
.zuul.yaml
+1 -1
file changed
Dockerfile
+2566
file added
poetry.lock
+46
file added
pyproject.toml
-17
file removed
requirements.txt
-6
file removed
test-requirements.txt
+39 -12
file changed
tests/plugins/test_koji_block_retired.py
+44 -1
file changed
tests/test_base.py
+4 -0
file changed
toddlers.toml.example
+45 -2
file changed
toddlers/base.py
+2 -2
file changed
toddlers/playtime.py
+9 -6
file changed
toddlers/plugins/koji_block_retired.py
+13 -25
file changed
tox.ini
Powered by Pagure 5.14.1
Documentation • File an Issue • About • SSH Hostkey/Fingerprint
© Red Hat, Inc. and others.