As discussed in #fedora-ci:

<paragan> Can some one please check https://pagure.io/fedora-ci/general/issue/99 ?
<jbair> paragan, sure, give me a few minutes and I'll see what I can find :) 
<paragan> thanks
<jbair> error is here, but I need to look at the code in the test and the repo to see where the conflict
<jbair> is
<jbair> 01:15:37  ++ rpm -q --define 'dist .fc33' --queryformat '%{name}-%{version}-%{release}\n' --specfile oflb-asana-math-fonts.spec
<jbair> 01:15:37  ++ awk NR==1
<jbair> 01:15:37  error: line 27: Unknown tag: %fontpkg
<jbair> 01:15:37  error: query of specfile oflb-asana-math-fonts.spec failed, can't parse
<paragan> does that mean that system does not have fonts-rpm-macros package installed?
<jbair> unsure, I'll check =) In a meeting so only able to lightly look at it, but I didn't want to make you wait half an hour =) 
<paragan> sure no hurry, take your time :)
<msrb> paragan, there is not buildrequire on fonts-rpm-macros in the spec file
<msrb> is that correct?
<jbair> msrb, I don't see one
<jbair> https://src.fedoraproject.org/fork/pnemade/rpms/oflb-asana-math-fonts/blob/master/f/oflb-asana-math-fonts.spec
<msrb> yeah :)
<jbair> silly question: wouldn't something like %fontbuild define that?
<paragan> its part of %fontpkg macro
<jbair> msrb, do we need to add that into the buildroots, or should spec files contain the buildrequire? Is there a doc that defines best practice (assuming those are our options)
<paragan> spec file already contains BR: fonts-rpm-macros
<msrb> paragan, https://src.fedoraproject.org/fork/pnemade/rpms/oflb-asana-math-fonts/blob/329823575166ed43e63638c14267d81e45093ee4/f/oflb-asana-math-fonts.spec
<msrb> paragan, https://src.fedoraproject.org/fork/pnemade/rpms/madan-fonts/blob/3a3a5b3280052ffbd3d539a38c5eaef466bcaca4/f/madan-fonts.spec
<msrb> paragan, how does that work exactly?
<msrb> what macro/line expands to br?
<paragan> %fontpkg
<msrb> and that macro is provided by which package? :)
<paragan> just "rpm -E %fontpkg" on your system provided you have fonts-rpm-macros installed on your system
<paragan> actual macro package is fonts-srpm-macros
<paragan> but it should get pulled by fonts-rpm-macros
<msrb> paragan, I tried modifying the madan spec file
<msrb> https://paste.centos.org/view/d14486f9
<msrb> it magically works when I add those few lines (copied from the spec that worked)
<msrb> CI calls: "rpm -q --define "dist .f32" --queryformat '%{name}-%{version}-%{release}\n' --specfile madan-fonts.spec | awk 'NR==1'"
<paragan> strange
<paragan> that modification is not required
<msrb> I agree - but I know nothing about packaging fonts :)
<jbair> that makes two of us :) 
<paragan> let me check scratch build if it works without that modification
<msrb> paragan, if the package builds in Koji, then it's probably a bug in CI
<paragan> let's check
<paragan> these are new packaging guidelines, so I am too learning to use them :)
<jbair> msrb, ty for your help 
<msrb> paragan, I will be offline/afk for 1-2 hours
<msrb> jbair, np! :)
<paragan> looks like there is fedora infra issue also going on
<paragan> I will also leave for the day and will join tomorrow
<paragan> thanks

We'll see if we have better luck tomorrow.

I still do not know how to trigger re-run of CI in any PR. Now that koji is working I did scratch builds.

1) madan-fonts package -> https://koji.fedoraproject.org/koji/taskinfo?taskID=42439697
Found mistake that I forgot to re-add BR: fontforge
After this fix, scratch build worked -> https://koji.fedoraproject.org/koji/taskinfo?taskID=42440017
Updated PR https://src.fedoraproject.org/rpms/madan-fonts/pull-request/2 now, awaiting Fedora CI to pick this PR for CI build

2) oflb-asana-math-fonts package -> https://koji.fedoraproject.org/koji/taskinfo?taskID=42439703
This scratch build is successful but why https://src.fedoraproject.org/rpms/oflb-asana-math-fonts/pull-request/2 shows CI error?
The error is same %fontpkg, See https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/jenkins/fedora-rawhide-pr-pipeline/detail/fedora-rawhide-pr-pipeline/2838/pipeline/

3) its been 17 hrs as of now but no CI run for https://src.fedoraproject.org/rpms/google-roboto-slab-fonts/pull-request/2
Is this normal time period for Fedora CI to run? If yes I will ignore the Fedora CI then and merge the PR to continue my work.

It appears fixed madan-fonts PR, Fedora CI build failed again for same reason, %fontpkg
https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/jenkins/fedora-rawhide-pr-pipeline/detail/fedora-rawhide-pr-pipeline/2841/pipeline/

Anyway, I don't want to wait more, hence I merged the madan-fonts PR.
The official successful working build link is https://koji.fedoraproject.org/koji/taskinfo?taskID=42440284

So it looks some issues in Fedora CI only.

Its now 1 hour still no Fedora CI job build for https://src.fedoraproject.org/rpms/sil-abyssinica-fonts/pull-request/2

Has Fedora CI remained with less available resources?

pnemade,

I am looking into the stalled PR trigger; in the meantime, you can comment in a PR with [citest] like msrb did in https://src.fedoraproject.org/rpms/sil-abyssinica-fonts/pull-request/2 as a temporary work-around.

Again, my apologies I missed you in IRC - I replied to your ci@ email and I'll be in IRC for ~5 more hours today. I'm in the CDT timezone in the US, though other members are online much earlier than I am. =)

I'll continue to update the ticket here as I have more updates.

-Jim

Okay, so the error we were seeing with PRs not running was related to the Jenkins restart that occurred yesterday, when openshift relocated our Jenkins instance. Looking at the logs, I saw:

Caused by: hudson.plugins.git.GitException: Command "git config remote.origin.url https://github.com/CentOS-PaaS-SIG/upstream-fedora-pipeline.git" returned status code 255:
stdout: 
stderr: error: could not lock config file .git/config: File exists

I cleared out the working directory, re-started the most recent PR trigger, and it ran as expected. Sorry the PR triggers weren't working, but they should now be working again.

Let me know if we can help with the %fontpkg macro issues beyond what we discussed yesterday. I'm not terribly familiar, but I would be happy to help in any way that I can.

Thanks for fixing PR trigger. I cannot test it quickly but hope it will work fine for any future PR's.
I did tried [citest] on some PR in last few days but it too did not trigger CI hence confused which is way to re-trigger CI.

koji scratch builds are working fine. Fedora CI is failing to build packages which uses %fontpkg. this macro is provided by fonts-srpm-macros package. I am not sure how further I can help you.

Let's meet on Monday on IRC. I am in IST timezone.

Talked to pnemade a bit in IRC; they gave me some additional context in this PR:

https://src.fedoraproject.org/rpms/khmeros-fonts/pull-request/2#comment-39639

My proposed fix after looking around and asking bookwar a few questions:
https://github.com/CentOS-PaaS-SIG/upstream-fedora-pipeline/pull/215

It's running a quick test to make sure the change doesn't break anything, and then I just need some peer review and we can see if this does the trick.

I think Jim's pull request will fix this issue. Since fonts-rpm-macros package was introduced ~1 month ago, it's probably not in the container image used by CI. Jim's PR will force rebuild of the image and thus it will pull the latest-greatest bits.

https://jenkins-continuous-infra.apps.ci.centos.org/job/fedora-stage-pipeline-merge/117/

https://src.fedoraproject.org/rpms/sil-abyssinica-fonts/pull-request/2

The PR was merged, and I re-ran the above PR, which created this job:

https://jenkins-continuous-infra.apps.ci.centos.org/job/fedora-rawhide-pr-pipeline/2876/

It ran until the very last stage, where it ran into an error:
https://jenkins-continuous-infra.apps.ci.centos.org/job/fedora-rawhide-pr-pipeline/2876/
https://jenkins-continuous-infra.apps.ci.centos.org/job/fedora-rawhide-pr-pipeline/2876/artifact/package-tests/logs/en_-usr-share-fonts-sil-abyssinica-err.log

find: ‘/usr/share/fonts/sil-abyssinica’: No such file or directory

That feels like a legitimate error, though hopefully you can let us know how things look for you now.

Indeed its legitimate error. The new fonts packaging guidelines changed the file path where font files are getting installed. I am now changing CI script in all font packages :)

Thank you Jim and other members of CI team for your help.

I confirm CI jobs are getting successfully completed for new fonts packaging changes.

Thanks! Closing this ticket. =)