Nice finding!

Yeah, this is because koji doesn't even send the 'start' fedmsg until after this, so there's not an indication a build was attempted.

Perhaps we could add in another set of messages there before/after srpm creation?

not sure how else to detect them. ;(

Well we do attempt to build them, so a check after the mass rebuild for packages we never heard back from would do.

Not entirely the same failure but it seems tcl wasn't bumped at all.

tcl has https://src.fedoraproject.org/rpms/tcl/blob/master/f/noautobuild - something we should probably stop using.

Filed upstream ticket - https://pagure.io/koji/issue/2825

@humaton Just FYI, this is related to the MLT rebuild we have discussed via IRC

This could be solved by not only reporting bugzillas for packages listed in 37-failures.html, but also the remaining packages listed only in f37-need-rebuild.html. The bugzillas could be slightly different (they could say: "Your package was not rebuilt in the mass rebuild, investigate why"). That could also help discover packages like https://pagure.io/releng/issue/10920

I created a test cli application that can get this data out of koji (how old buildtime data is for each package in a tag): https://github.com/CentOS/sync2git/commit/1cba9c84b1587d633d3183aa147c8b25a0aadafa

...obviously can do a lot of changes from here, at the moment it will give you lists of packages with a buildtime older than a certain duration or a historgram of the buildtime of all packages in a tag. Eg. Fedora-38:

2021-08-13     72 
2021-09-12    136 
2021-10-13      0 
2021-11-12      0 
2021-12-12     57 
2022-01-12   1054 
2022-02-11    116 
2022-03-13      7 
2022-04-13     19 
2022-05-13      6 
2022-06-12    177 
2022-07-13   2238 
2022-08-12    291 
2022-09-11    415 
2022-10-12    358 
2022-11-11    714 
2022-12-11    684 
2023-01-11 214615 ##################################################
2023-02-10  38332 ########
2023-03-12  29175 ######

So ~208 packages with buildtimes before 2021-10-13. Note that I don't filter to .src packages atm. but you can do that in the lists easily:

% wc -l work/lp-f38-50w
4151 work/lp-f38-50w
% fgrep .src work/lp-f38-50w | wc -l
380

...anyway, I guess the next step is working out if we want to tweak how we present the data so a human can look at it or add code so it can just email the owners on it's own?

FYI, filtered to source packages (f38):

2021-08-13     6 
2021-09-12     7 
2021-10-13     0 
2021-11-12     0 
2021-12-12     4 
2022-01-12   123 
2022-02-11     8 
2022-03-13     2 
2022-04-13     2 
2022-05-13     1 
2022-06-12    11 
2022-07-13   198 
2022-08-12    19 
2022-09-11    37 
2022-10-12    23 
2022-11-11    23 
2022-12-11    32 
2023-01-11 19257 ##################################################
2023-02-10  2102 #####
2023-03-12  1253 ###

Thanks for working on this!

Ideally, next step would be to use this information and adjust the existing https://pagure.io/releng/blob/main/f/scripts/mass_rebuild_file_bugs.py script to use it.

ie, in addition to the ones that it finds that failed, also find the ones that haven't been rebuilt since the mass rebuild started and report bugs on them as not having been built. That might catch more than the ones that just failed to make a src.rpm in koji, but ideally any that were skipped for any reason should get a bug so the maintainer(s) can fix it.

We may want to I guess before that mail the devel list with the list we have and ask them if there's any false positives we need to exclude?

The only one that leaps to mind is shim...

Is this work still in progress?

cc @james