We use python-jenkins to trigger the build which I had hope would do the right thing for us: https://pagure.io/pagure/blob/master/f/pagure/lib/lib_ci.py#_137

We use python-jenkins to trigger the build which I had hope would do the right thing for us: https://pagure.io/pagure/blob/master/f/pagure/lib/lib_ci.py#_137

Thanks, I noticed that while looking through the code. I copied the trigger_jenkins_build function into a test script which still receives a 403 error when I try to submit a job. A copy of the script is available at the following URL.

https://watters.ws/jenkins_test.txt

I will check with the python-jenkins project to see if there is a way to resolve this. Most likely the issue is being caused by recent security changes in Jenkins. Previously I was able to get builds to run by disabling CSRF protection however Jenkins no longer allows that.

We're using python3-jenkins 1.6.0 in production on RHEL 8, which might make a difference for you. RHEL 7 offered python-jenkins 0.4.16.

We're using python3-jenkins 1.6.0 in production on RHEL 8, which might make a difference for you. RHEL 7 offered python-jenkins 0.4.16.

That certainly could make a difference. I'll see if there is a way to upgrade the python module.

Upgrading the python-jenkins module seems to have helped but now I'm getting an SSL certificate error from the pagure_ci process.

Oct 13 10:48:40 git.example.com celery[5805]: 2020-10-13 10:48:40,352 [INFO] pagure.lib.lib_ci: Pagure-CI:An error occured: HTTPSConnectionPool(host='jenkins.example.com', port=443): Max retries exceeded with url: /crumbIssuer/api/json (Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)'),))

Is there a way to make pagure_ci accept the certificate and connect? We are using certs signed by our own internal CA which is listed as a trusted authority in the system's trust anchors.

I have found a way to resolve this by disabling SSL verification in the python-jenkins module. The file located at /usr/lib/python2.7/site-packages/jenkins/init.py must be modified as follows.

--- __init__.py.orig    2020-10-14 13:00:03.171935772 -0400
+++ __init__.py 2020-10-14 13:00:17.348045268 -0400
@@ -340,12 +340,8 @@
                 header, value = token.split(":", 1)
                 self._session.headers[header] = value.strip()
-        if os.getenv('PYTHONHTTPSVERIFY', '1') == '0':
-            logging.debug('PYTHONHTTPSVERIFY=0 detected so we will '
-                          'disable requests library SSL verification to keep '
-                          'compatibility with older versions.')
-            requests.packages.urllib3.disable_warnings(InsecureRequestWarning)
-            self._session.verify = False
+        requests.packages.urllib3.disable_warnings(InsecureRequestWarning)
+        self._session.verify = '/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'
     def _get_encoded_params(self, params):
         for k, v in params.items():rabbit: 

I'm not sure if pagure has a way to pass the PYTHONHTTPSVERIFY environment variable in however that would be a better solution.

If you're using system packaged requests and urllib3, then it should use the system trust store automatically already...

If you're using system packaged requests and urllib3, then it should use the system trust store automatically already...

The server is using the python-requests and python-urllib3 packages from the default repos. Here is the package info shown by rpm.

[root@git ~]# rpm -qi python-requests
Name        : python-requests
Version     : 2.6.0
Release     : 9.el7_8
[root@git ~]# rpm -qi python-urllib3
Name        : python-urllib3
Version     : 1.10.2
Release     : 7.el7

I was also using the default python-jenkins package up until yesterday.

I've been able to resolve the issue by modifying httplib2 to disable SSL verification entirely. I'm going to close this issue since the problem is not with pagure itself. For anybody else having this issue the modified version of httplib2 is available at this URL.

https://gist.github.com/aa46f80d6509c5de33a873f09e91f163

Place this file at /usr/lib/python2.7/site-packages/httplib2/__init__.py and restart the pagure_ci service, that should resolve any issues with SSL validation failures.