Well, that means that what was discussed before wasn't implemented, as clearly in git I see this :

* 919c149 - Switched back apps.ci to Digicert, during migration, per bstinson's request (5 months ago) <Fabian Arrotin>
* 571f835 - Renewed (through centos process) apps.ci.centos.org tls letsencrypt cert (5 months ago) <Fabian Arrotin>

And from that git repo where all certs are stored :

openssl x509 -in apps.ci.centos.org.crt -noout -text |egrep -A 2 'Issuer|Validity'
        Issuer: C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert SHA2 High Assurance Server CA
        Validity
            Not Before: Jul  6 00:00:00 2020 GMT
            Not After : May 27 12:00:00 2022 GMT

So clearly a valid cert is where it's supposed to be, but just waiting for someone with access (I don't) to have pushed it some time ago ?

@siddharthvipul1 is that for you then ?

any updates?

any updates?

I discussed with @bstinson yesterday and he will get this done asap
sorry for the delay

Adding myself here. Without this, the Pagure cannot trigger Jenkins jobs.

Hi,

As said, nothing I can do to help with this issue right now .. But also, does it now make sense to fix something that will disappear in one month ?

See https://lists.centos.org/pipermail/ci-users/2021-January/002156.html about the last notification, and the one that was sent more than 6months ago now :
https://lists.centos.org/pipermail/ci-users/2020-June/002095.html

So my underlying question would be more : why is that still running on the old and legacy cluster ? (not denying the TLS cert issue but rather understand why it's still on something that we can't manage )

I will work on migrating it
makes sense

Adding myself here. Without this, the Pagure cannot trigger Jenkins jobs.

we created a separate jenkins instance [0] for Pagure. I will sync with @pingou this week to migrate it completely. (last time we tried, there was a small hiccup)

[0] https://jenkins-pagure.apps.ocp.ci.centos.org/

When the new jenkins instance will be ready? we are looking forward to it.
I tried to login https://jenkins-pagure.apps.ocp.ci.centos.org/ with my ACO but it doesn't work.

When the new jenkins instance will be ready? we are looking forward to it.
I tried to login https://jenkins-pagure.apps.ocp.ci.centos.org/ with my ACO but it doesn't work.

can you please tell me your ACO email address?
I will add you to pagure namespace
They are ready but I haven't migrated the jobs yet
(which I will try to do today evening or tomorrow)

When the new jenkins instance will be ready? we are looking forward to it.
I tried to login https://jenkins-pagure.apps.ocp.ci.centos.org/ with my ACO but it doesn't work.

can you please tell me your ACO email address?
I will add you to pagure namespace

My ACO account name is hlin, email hlin@redhat.com

@siddharthvipul1 , can you also give me access to this new jenkins? My ACO is jkaluza, my email is jkaluza@redhat.com.

@hlin @jkaluza
Both of you now have access to fedora-infra and pagure namespaces
Please login to [0] to see them
If you have any doubts, please let me know

[0] https://console-openshift-console.apps.ocp.ci.centos.org/dashboards

@siddharthvipul1 I can login now. When the jobs could be migrated?

@siddharthvipul1 I can login now. When the jobs could be migrated?

oh no,
sorry I missed this
Projects have been migrated and I added F32, F33, el7, el8 as jenkins agents that you can select as labels to run your projects against

@siddharthvipul1 I can login now. When the jobs could be migrated?

oh no,
sorry I missed this
Projects have been migrated and I added F32, F33, el7, el8 as jenkins agents that you can select as labels to run your projects against

https://jenkins-fedora-infra.apps.ocp.ci.centos.org/ this is the new jenkins instance right? I see pungi, odcs, cts jobs in it rather than in https://jenkins-pagure.apps.ocp.ci.centos.org/

@siddharthvipul1 I can login now. When the jobs could be migrated?

oh no,
sorry I missed this
Projects have been migrated and I added F32, F33, el7, el8 as jenkins agents that you can select as labels to run your projects against

https://jenkins-fedora-infra.apps.ocp.ci.centos.org/ this is the new jenkins instance right? I see pungi, odcs, cts jobs in it rather than in https://jenkins-pagure.apps.ocp.ci.centos.org/

I just copy-pasted all job configs to this new pv
do you want these specific jobs to be in Pagure namespace (do they make more sense there)?
I can do it now

@siddharthvipul1 I can login now. When the jobs could be migrated?

oh no,
sorry I missed this
Projects have been migrated and I added F32, F33, el7, el8 as jenkins agents that you can select as labels to run your projects against

https://jenkins-fedora-infra.apps.ocp.ci.centos.org/ this is the new jenkins instance right? I see pungi, odcs, cts jobs in it rather than in https://jenkins-pagure.apps.ocp.ci.centos.org/

I just copy-pasted all job configs to this new pv
do you want these specific jobs to be in Pagure namespace (do they make more sense there)?
I can do it now

The namespace doesn't matter to me. I just want to confirm which jenkins master (namespace) we can use. https://jenkins-fedora-infra.apps.ocp.ci.centos.org/ is ok for me as the jobs are already there.

@siddharthvipul1 I can login now. When the jobs could be migrated?

oh no,
sorry I missed this
Projects have been migrated and I added F32, F33, el7, el8 as jenkins agents that you can select as labels to run your projects against

All these agents are offline.

Can you tell me the job you are trying to run?
I think the issue might be something else
Agents are connected and running fine

@siddharthvipul1 Can I get access permissions? My login is lsedlar, e-mail lsedlar@redhat.com

Additionally, is there a way to provide read only access without authentication? The jobs should be triggered for pull requests, which can come from any contributor, and ideally they should have access to test results of their change without having to create another account and request permissions.

@siddharthvipul1 Can I get access permissions? My login is lsedlar, e-mail lsedlar@redhat.com

Done

Additionally, is there a way to provide read only access without authentication? The jobs should be triggered for pull requests, which can come from any contributor, and ideally they should have access to test results of their change without having to create another account and request permissions.

Done :)

Thank you, I can now see the job without authentication. I'm however having issues logging in. I click log in, then "Log in with OpenShift", then select accounts-centos-org, put in my username and password and then get redirected back to the Jenkins page, but it doesn't seem like I'm logged in. When I try with wrong password, I do get an error as expected.

Thank you, I can now see the job without authentication. I'm however having issues logging in. I click log in, then "Log in with OpenShift", then select accounts-centos-org, put in my username and password and then get redirected back to the Jenkins page, but it doesn't seem like I'm logged in. When I try with wrong password, I do get an error as expected.

weird,
can you try to login here first and see if this helps: https://console-openshift-console.apps.ocp.ci.centos.org/dashboards

A job was triggered automatically but it failed https://jenkins-fedora-infra.apps.ocp.ci.centos.org/job/pungi/1555/console

ERROR: Error cloning remote repo 'origin'
Caused by: hudson.plugins.git.GitException: Error performing git command: git init /home/jenkins/workspace/workspace/pungi

Caused by: java.io.IOException: Cannot run program "git" (in directory "/home/jenkins/workspace/workspace/pungi"): error=2, No such file or directory

A job was triggered automatically but it failed https://jenkins-fedora-infra.apps.ocp.ci.centos.org/job/pungi/1555/console

ERROR: Error cloning remote repo 'origin'
Caused by: hudson.plugins.git.GitException: Error performing git command: git init /home/jenkins/workspace/workspace/pungi

Caused by: java.io.IOException: Cannot run program "git" (in directory "/home/jenkins/workspace/workspace/pungi"): error=2, No such file or directory

So git was not installed on the agent (fixed) (also, oops)
and after a manual trigger, I also noticed requirement of tox (installed)
are there any other packages that you would like me to install across agents?

These packages are needed

dnf install make findutils python3-tox python3-createrepo_c python3-urlgrabber koji

and another issue is email notification not work

ERROR: Couldn't connect to host, port: localhost, 25; timeout 60000

These packages are needed

dnf install make findutils python3-tox python3-createrepo_c python3-urlgrabber koji

done in fedora nodes (packages not found in epel, I will try to see alternate names and install in a while, added it on my list)

and another issue is email notification not work

Here is the sop to use email notification: https://github.com/centosci/ocp4-docs/tree/master/sops/migration#configuring-smtp-server-for-email-notifications

ERROR: Couldn't connect to host, port: localhost, 25; timeout 60000

These packages are needed

dnf install make findutils python3-tox python3-createrepo_c python3-urlgrabber koji

done in fedora nodes (packages not found in epel, I will try to see alternate names and install in a while, added it on my list)

That's ok just for fedora nodes.

Here is the sop to use email notification: https://github.com/centosci/ocp4-docs/tree/master/sops/migration#configuring-smtp-server-for-email-notifications

email notification works now.

I retriggered a job https://jenkins-fedora-infra.apps.ocp.ci.centos.org/job/pungi/1564/console, it failed with error

PermissionError: [Errno 13] Permission denied: '/home/jenkins/workspace/workspace/pungi/.tox/log/.lock'

PermissionError: [Errno 13] Permission denied: '/home/jenkins/workspace/workspace/pungi/.tox/log/.lock'

I see that these files are owned by root
user not being having sudo access could be the reason? (it's definitely not recommended)
can the test be adjusted to avoid this?
or can you recommend a solution I can implement for you?

PermissionError: [Errno 13] Permission denied: '/home/jenkins/workspace/workspace/pungi/.tox/log/.lock'

I see that these files are owned by root
user not being having sudo access could be the reason? (it's definitely not recommended)
can the test be adjusted to avoid this?
or can you recommend a solution I can implement for you?

Actually it's fine to run with non-root user, I'm not sure why it's owned by root. Could you delete .tox dir?

Actually it's fine to run with non-root user, I'm not sure why it's owned by root. Could you delete .tox dir?

Deleted

@siddharthvipul1 could you make docker available in fedora nodes? I'm trying https://www.jenkins.io/doc/book/pipeline/docker/ for a job but it failed with

/var/lib/jenkins/jobs/pungi-test/workspace@tmp/durable-b24d7ba9/script.sh: line 1: docker: command not found

@siddharthvipul1 could you make docker available in fedora nodes? I'm trying https://www.jenkins.io/doc/book/pipeline/docker/ for a job but it failed with

/var/lib/jenkins/jobs/pungi-test/workspace@tmp/durable-b24d7ba9/script.sh: line 1: docker: command not found

I can install docker in nodes
Just confirming, with Docker now supporting cgroups v2, I won't have to change anything else, right?
so just install and start/enable daemon?

@siddharthvipul1 could you make docker available in fedora nodes? I'm trying https://www.jenkins.io/doc/book/pipeline/docker/ for a job but it failed with

/var/lib/jenkins/jobs/pungi-test/workspace@tmp/durable-b24d7ba9/script.sh: line 1: docker: command not found

I can install docker in nodes
Just confirming, with Docker now supporting cgroups v2, I won't have to change anything else, right?
so just install and start/enable daemon?

could podman be used here instead of docker?
I don't have an issue with docker but podman doesn't need a daemon
excuse me if it's necessary, I am here talking out loud

@siddharthvipul1 could you make docker available in fedora nodes? I'm trying https://www.jenkins.io/doc/book/pipeline/docker/ for a job but it failed with

/var/lib/jenkins/jobs/pungi-test/workspace@tmp/durable-b24d7ba9/script.sh: line 1: docker: command not found

I can install docker in nodes
Just confirming, with Docker now supporting cgroups v2, I won't have to change anything else, right?
so just install and start/enable daemon?

could podman be used here instead of docker?
I don't have an issue with docker but podman doesn't need a daemon
excuse me if it's necessary, I am here talking out loud

I'm sure podman works fine in my local env, we can try it and make sure podman-docker installed.

I'm sure podman works fine in my local env, we can try it and make sure podman-docker installed.

podman-docker is now installed in F32 and F33

@hlin
Do you have all the things needed? :)

@hlin
Do you have all the things needed? :)

Yes thanks for your help :smile:

Closing this as original ticket issue has been resolved (along with internal in-ticket discusison)