fedora-infra / toddlers

Clone
Source Code
GIT

#155 [WIP] scm_request_processor: Check the availability of external services before doing anything
Closed 2 years ago by jnsamyak. Opened 2 years ago by jnsamyak.
fedora-infra/ jnsamyak/toddlers ft/checkForExternalServicesSCMRequests  into  main

file added
+23 
@@ -0,0 +1,23 @@ 
 

+ from toddlers.utils.check_services import check_services
 

+ 

+ 

+ class TestCheckServices:
 

+     def test_check_services_available(self):
 

+         services = {
 

+             "dist-git": "https://src.fedoraproject.org/api/0/",
 

+             "bugzilla": "https://bugzilla.redhat.com/",
 

+             "PDC": "https://pdc.fedoraproject.org/rest_api/v1/",
 

+             "pagure": "https://src.fedoraproject.org/api/0/",
 

+             "fas": "https://id.fedoraproject.org/fas/",
 

+         }
 

+         assert check_services(services)
 

+ 

+     def test_check_services_not_available(self):
 

+         services = {
 

+             "dist-git": "https://src.fedoraproject.org/api/0/",
 

+             "bugzilla": "https://bugzilla.redhat.com/",
 

+             "PDC": "https://pdc.fedoraproject.org/rest_api/v1/",
 

+             "pagure": "https://src.fedoraproject.org/api/0/",
 

+             "fas": "https://id.fedoraproject.org/fas/",
 

+         }
 

+         assert not check_services(services)

file modified
+34 -1 
@@ -23,7 +23,15 @@ 
 

  

  from toddlers.base import ToddlerBase
 

  from toddlers.exceptions import ValidationError
 

- from toddlers.utils import bugzilla_system, fedora_account, git, pagure, pdc, requests
 

+ from toddlers.utils import (
 

+     bugzilla_system,
 

+     check_services,
 

+     fedora_account,
 

+     git,
 

+     pagure,
 

+     pdc,
 

+     requests,
 

+ )
 

  

  # Regex for branch name validation
 

  STREAM_NAME_REGEX = r"^[a-zA-Z0-9.\-_+]+$"
 
@@ -43,6 +51,15 @@ 
 

      "https://pagure.io/releng/issues"
 

  )
 

  

+ # External services: dist-git, pdc, pagure, fas
 

+ EXTERNAL_SERVICES = {
 

+     "dist-git": "https://src.fedoraproject.org/api/0/",
 

+     "bugzilla": "https://bugzilla.redhat.com/",
 

+     "PDC": "https://pdc.fedoraproject.org/rest_api/v1/",
 

+     "pagure": "https://src.fedoraproject.org/api/0/",
 

+     "fas": "https://id.fedoraproject.org/fas/",
 

+ }
 

+ 

  _log = logging.getLogger(__name__)
 

  

  # Currently there is no way to generate PDC API token on either
 
@@ -379,6 +396,14 @@ 
 

              issue body
 

            initial_commit: indicate whether to create an initial commit.
 

          """
 

+ 

+         _log.info(
 

+             "Before creating new repo, checking if all the services are available"
 

+         )
 

+         is_services_available = check_services(EXTERNAL_SERVICES)
 

+         if not is_services_available:
 

+             return
 

+ 

          _log.info("New repo was requested. Validating request.")
 

          required_keys = [
 

              "repo",
 
@@ -667,6 +692,14 @@ 
 

            issue_body_json: a partially validated dictionary of the JSON in the
 

              issue body
 

          """
 

+ 

+         _log.info(
 

+             "Before creating new branch, checking if all the services are available"
 

+         )
 

+         is_services_available = check_services(EXTERNAL_SERVICES)
 

+         if not is_services_available:
 

+             return
 

+ 

          required_keys = ["action", "namespace", "branch", "sls", "repo"]
 

          _log.debug("JSON keys in message body {}".format(issue_body_json.keys()))
 

          for key in required_keys:

file added
+21 
@@ -0,0 +1,21 @@ 
 

+ import requests
 

+ 

+ 

+ def check_services(services):
 

+     """
 

+     Check the availability of external services.
 

+ 

+     Arguments:
 

+     - services (dict): A dictionary of service names and their URLs.
 

+ 

+     Returns:
 

+     - True if all services are available, False otherwise.
 

+     """
 

+     for service_name, service_url in services.items():
 

+         try:
 

+             response = requests.get(service_url)
 

+             response.raise_for_status()
 

+         except requests.exceptions.RequestException as e:
 

+             print(f"Service {service_name} is not available: {e}")
 

+             return False
 

+     return True

Some time back there was a request[1] raised where it was noted that due to some services' unavailability, some tickets were processed at that time to end in an inconsistent state. This can happen due to various reasons, but the script should be smart enough to not process the tickets if there is an outage of the services we need (PDC, pagure, fas, dist-git). This PR adds the same check-in create_new_repo and create_new_branch methods.

This is my first time contributing to toddlers, so all types of suggestions are welcome \0/

[1] https://pagure.io/fedora-infra/toddlers/issue/150

Signed-off-by: Samyak Jain samyak.jn11@gmail.com

Build failed. More information on how to proceed and troubleshoot errors available at https://fedoraproject.org/wiki/Zuul-based-ci
https://fedora.softwarefactory-project.io/zuul/buildset/cba1be3a9acd4766bbc029f79083c285

Simple boolean returned from self.check_services would be enough and faster to process.

Simple boolean returned from self.check_services would be enough and faster to process.

Don't hardcode the URLs, these are loaded in each corresponding module in utils from config file.

Leave this as a wrapper for method in each corresponding module inutils and just call them here.

This should be moved to each corresponding module and be adjusted for the service itself, so it could be possibly used by other toddlers as well.

Also, you should fix all the tox tests.

1 new commit added

  • Move function to utils, add tests
2 years ago

Build failed. More information on how to proceed and troubleshoot errors available at https://fedoraproject.org/wiki/Zuul-based-ci
https://fedora.softwarefactory-project.io/zuul/buildset/d9e461a63f284151aba88409e2c9610f

2 new commits added

  • Move function to utils, add tests
  • Init: Adds a method to check if external services are available while creating new branches and repos
2 years ago

2 new commits added

  • Move function to utils, add tests
  • Init: Adds a method to check if external services are available while creating new branches and repos
2 years ago

Build failed. More information on how to proceed and troubleshoot errors available at https://fedoraproject.org/wiki/Zuul-based-ci
https://fedora.softwarefactory-project.io/zuul/buildset/793ca55486424512be28c416ff483b9b

Pull-Request has been closed by jnsamyak
2 years ago
Metadata
None
No Tags
Changes Summary 3
+23
file added
tests/utils/test_check_services.py
+34 -1
file changed
toddlers/plugins/scm_request_processor.py
+21
file added
toddlers/utils/check_services.py
Powered by Pagure 5.14.1
Documentation • File an Issue • About • SSH Hostkey/Fingerprint
© Red Hat, Inc. and others.