Still failing at the moment so if there's a background process fixing this, it doesn't seem to run very often.

The error means that the https://src.fedoraproject.org/ returned 500 when calling the branch creation API.

The script that is creating the branch is not doing anything with koji, so this must be caused somewhere else then during creation of the branch. My assumption would be that the pagure API is enabling the koji for the package and this failed.

Let me check if that is true.

So the pagure API code doesn't touch koji at all, just trying to create the branch in pagure. So the koji enablement must happen somewhere else.

@carlwgeorge I think something is not correctly setup for epel10 yet, or is this expected behavior?

I don't think it's expected behavior because I've done dozens of epel10 branch requests over the last couple of weeks and only 2 have failed like this.

After the branch is created, is there a message sent out? I wonder if that message perhaps doesn't get sent out and then something else doesn't see the message and hence doesn't update koji?

The message is sent, you can see all the latest messages in datagrepper

EDIT: And here is the message about creation of the branch you requested.

The process that syncs projects/branches/ownership to koji is the koji_sync_listener service ( fm-consumer@koji_sync_listener.service ) running on bodhi-backend01.

It should be listening to those messages and acting on them. I don't know why it did not in this case.

It also runs once a day a 'full sync' where it checks all packages and syncs ownership/etc to koji.

There goes my theory then. No idea what's going wrong, though the traceback in the branch request does correlate with the koji issue manifesting.

Build still failing at the moment.

Ah, so does the once-a-day run take place sometime around 1900UTC? That could explain why my perl-Sereal build eventually worked yesterday.

Indeed the original error toddlers.exceptions.pagure_error.PagureError is because of the status code returned by pagure. I've seen it happen occasionally, but as of yet no one has done a deep dive investigation to figure out why. With the impending replacement of pagure I'm not entirely sure spending cycles on a rare intermittent error is worth the effort.

Allowing the package in koji is done in two ways:

• koji_sync_listener notices the message and runs owner-sync-pagure for the package. The name is a bit misleading, as the ownership is set in the tag allowlist, which is what permits the package to be built in that tag.
• There is a cron job that runs owner-sync-pagure for all packages once a day, to catch and clean up instances like this that were missed for whatever reason.

I believe the cron job would have picked this up and fixed it, but I went ahead and manually added the package to the tag allowlist to unblock you.

carl ~ 
❯ koji list-pkgs --tag epel10.0-testing-candidate --package perl-File-Find-Object
Package                 Tag                     Extra Arches     Owner          
----------------------- ----------------------- ---------------- ---------------
perl-File-Find-Object   epel10.0                                 pghmcfc

Thanks, build succeeded now. What time zone does that cron job run in? It'll be useful to know when to resubmit builds if this crops up again, which is quite likely given that I'm working through a lot of the perl stack for EPEL-10 at the moment. It'll save me having to keep re-submitting until it works.

It's UTC. All infrastructure machines are on UTC.

I'm a bit puzzled how this message wasn't processed, but I am not sure how to debug it any further. I looked for that message in the logs for the sync listener and didn't see it... ;(

I'm not sure how my perl-Sereal build from yesterday (https://koji.fedoraproject.org/koji/buildinfo?buildID=2539922) started working now. The successful build started at 1655 after several failures at various points earlier in the day. However, the cron job seems to be set to run at 0415 so it seems unlikely that that cron job is what fixed things.

Saw a different error today (https://pagure.io/releng/fedora-scm-requests/issue/66371):

Error happened during processing:

Traceback (most recent call last):
  File "/code/toddlers/plugins/scm_request_processor.py", line 189, in process
    self.process_ticket(issue)
  File "/code/toddlers/plugins/scm_request_processor.py", line 301, in process_ticket
    self.create_new_branch(issue, issue_body)
  File "/code/toddlers/plugins/scm_request_processor.py", line 780, in create_new_branch
    self.pagure_io.close_issue(
  File "/code/toddlers/utils/pagure.py", line 191, in close_issue
    raise PagureError(
toddlers.exceptions.pagure_error.PagureError: Couldn't close issue 'https://pagure.io/releng/fedora-scm-requests/issue/66371'

Request to 'https://pagure.io/api/0/releng/fedora-scm-requests/issue/66371/status':
{'status': 'Processed'}

Response:
None

Status code: 500

Don't know if this is blocking a build or not because there is another issue affecting the epel10 buildroot today (https://pagure.io/releng/issue/12305), which I'm hitting for all epel10 builds. It doesn't look promising though:

$ koji list-pkgs --tag epel10.0-testing-candidate --package perl-Cpanel-JSON-XS
(no matching packages)

Saw a different error today (https://pagure.io/releng/fedora-scm-requests/issue/66371):
```
Error happened during processing:

Traceback (most recent call last):
File "/code/toddlers/plugins/scm_request_processor.py", line 189, in process
self.process_ticket(issue)
File "/code/toddlers/plugins/scm_request_processor.py", line 301, in process_ticket
self.create_new_branch(issue, issue_body)
File "/code/toddlers/plugins/scm_request_processor.py", line 780, in create_new_branch
self.pagure_io.close_issue(
File "/code/toddlers/utils/pagure.py", line 191, in close_issue
raise PagureError(
toddlers.exceptions.pagure_error.PagureError: Couldn't close issue 'https://pagure.io/releng/fedora-scm-requests/issue/66371'

Request to 'https://pagure.io/api/0/releng/fedora-scm-requests/issue/66371/status':
{'status': 'Processed'}

Response:
None

Status code: 500
```

This error just means that the script couldn't close the ticket as processed, so it doesn't affect anything.

Well that's what I'd have thought too but the ticket did in fact get closed and perl-Cpanel-JSON-XS doesn't seem to be in the list for epel10.0-testing-candidate.

Yeah, It seems that the ticket was closed and the error happened in the code after closing the ticket.

Not unexpectedly, my perl-Cpanel-JSON-XS build (branch request ticket had error this morning) failed:

Building perl-Cpanel-JSON-XS-4.38-1.el10 for epel10-candidate
Created task: 122950817
Task info: https://koji.fedoraproject.org/koji/taskinfo?taskID=122950817
Watching tasks (this may be safely interrupted)...
122950817 build (epel10-candidate, /rpms/perl-Cpanel-JSON-XS.git:94c4d9f800f5f168105447263e223caefe09751f): free
122950817 build (epel10-candidate, /rpms/perl-Cpanel-JSON-XS.git:94c4d9f800f5f168105447263e223caefe09751f): free -> open (buildvm-a64-11.iad2.fedoraproject.org)
  122950825 buildSRPMFromSCM (/rpms/perl-Cpanel-JSON-XS.git:94c4d9f800f5f168105447263e223caefe09751f): free
  122950825 buildSRPMFromSCM (/rpms/perl-Cpanel-JSON-XS.git:94c4d9f800f5f168105447263e223caefe09751f): free -> open (buildvm-x86-22.iad2.fedoraproject.org)
122950817 build (epel10-candidate, /rpms/perl-Cpanel-JSON-XS.git:94c4d9f800f5f168105447263e223caefe09751f): open (buildvm-a64-11.iad2.fedoraproject.org) -> FAILED: BuildError: package perl-Cpanel-JSON-XS not in list for tag epel10.0-testing-candidate
  0 free  1 open  0 done  1 failed
  122950825 buildSRPMFromSCM (/rpms/perl-Cpanel-JSON-XS.git:94c4d9f800f5f168105447263e223caefe09751f): open (buildvm-x86-22.iad2.fedoraproject.org) -> closed
  0 free  0 open  1 done  1 failed

I'll try again later/tomorrow, by which time the cron job or whatever should have fixed the issue.

Either the cron tab takes a very long time to run or it has not resolved the issue:

$ koji list-pkgs --tag epel10.0-testing-candidate --package perl-Cpanel-JSON-XS
(no matching packages)
$ TZ=UTC date; fedpkg build
Thu  5 Sep 06:02:51 UTC 2024
Building perl-Cpanel-JSON-XS-4.38-1.el10 for epel10-candidate
Created task: 122970998
Task info: https://koji.fedoraproject.org/koji/taskinfo?taskID=122970998
Watching tasks (this may be safely interrupted)...
122970998 build (epel10-candidate, /rpms/perl-Cpanel-JSON-XS.git:94c4d9f800f5f168105447263e223caefe09751f): free
122970998 build (epel10-candidate, /rpms/perl-Cpanel-JSON-XS.git:94c4d9f800f5f168105447263e223caefe09751f): free -> open (buildhw-a64-20.iad2.fedoraproject.org)
  122971043 buildSRPMFromSCM (/rpms/perl-Cpanel-JSON-XS.git:94c4d9f800f5f168105447263e223caefe09751f): free
  122971043 buildSRPMFromSCM (/rpms/perl-Cpanel-JSON-XS.git:94c4d9f800f5f168105447263e223caefe09751f): free -> open (buildvm-a64-24.iad2.fedoraproject.org)
122970998 build (epel10-candidate, /rpms/perl-Cpanel-JSON-XS.git:94c4d9f800f5f168105447263e223caefe09751f): open (buildhw-a64-20.iad2.fedoraproject.org) -> FAILED: BuildError: package perl-Cpanel-JSON-XS not in list for tag epel10.0-testing-candidate
  0 free  1 open  0 done  1 failed
  122971043 buildSRPMFromSCM (/rpms/perl-Cpanel-JSON-XS.git:94c4d9f800f5f168105447263e223caefe09751f): open (buildvm-a64-24.iad2.fedoraproject.org) -> closed
  0 free  0 open  1 done  1 failed

122970998 build (epel10-candidate, /rpms/perl-Cpanel-JSON-XS.git:94c4d9f800f5f168105447263e223caefe09751f) failed

And at last, I was able to build perl-Cpanel-JSON-XS. No idea what happened to change things.

Today's example: perl-Module-Find

Building perl-Module-Find-0.16-3.el10 for epel10-candidate
Created task: 122988156
Task info: https://koji.fedoraproject.org/koji/taskinfo?taskID=122988156
Watching tasks (this may be safely interrupted)...
122988156 build (epel10-candidate, /rpms/perl-Module-Find.git:28107c20603d2f7f111e6762f23eb43c068ca07f): free
122988156 build (epel10-candidate, /rpms/perl-Module-Find.git:28107c20603d2f7f111e6762f23eb43c068ca07f): free -> open (buildvm-a64-06.iad2.fedoraproject.org)
  122988159 buildSRPMFromSCM (/rpms/perl-Module-Find.git:28107c20603d2f7f111e6762f23eb43c068ca07f): free
  122988159 buildSRPMFromSCM (/rpms/perl-Module-Find.git:28107c20603d2f7f111e6762f23eb43c068ca07f): free -> open (buildvm-a64-17.iad2.fedoraproject.org)
122988156 build (epel10-candidate, /rpms/perl-Module-Find.git:28107c20603d2f7f111e6762f23eb43c068ca07f): open (buildvm-a64-06.iad2.fedoraproject.org) -> FAILED: BuildError: package perl-Module-Find not in list for tag epel10.0-testing-candidate
  0 free  1 open  0 done  1 failed
  122988159 buildSRPMFromSCM (/rpms/perl-Module-Find.git:28107c20603d2f7f111e6762f23eb43c068ca07f): open (buildvm-a64-17.iad2.fedoraproject.org) -> closed
  0 free  0 open  1 done  1 failed

122988156 build (epel10-candidate, /rpms/perl-Module-Find.git:28107c20603d2f7f111e6762f23eb43c068ca07f) failed

No indication that anything went wrong in the branch request (https://pagure.io/releng/fedora-scm-requests/issue/66471).

Now managed to build perl-Module-Find. Wasn't able to a couple of hours ago. Don't know what changed.

Are you still seeing this?

Seen it once since perl-Module-Find and it resolved itself after a day or so (https://pagure.io/releng/fedora-scm-requests/issue/67247).

I've done the bulk of my packages in the perl stack now so my request frequency is much lower in the last couple of weeks.

ok, lets close this... if you see it again, can you please re-open and we can try and track it down while it's fresh.