Forwarded from https://pagure.io/modularity/issue/179, CC @mschorm.
MBS sometimes takes too long to realize all RPM build tasks of a group finished and a new building group can commence. This is especially frequent for packages following the initial module-build-macros package.
Example is https://mbs.fedoraproject.org/module-build-service/2/module-builds/14376 where RPM build task for module-build-macros https://koji.fedoraproject.org/koji/taskinfo?taskID=86359901 finished at Thu, 28 Apr 2022 16:18:02 UTC. A subsequent RPM build for galera package https://koji.fedoraproject.org/koji/taskinfo?taskID=86423098 was created at Sat, 30 Apr 2022 00:42:09 UTC. That's more than 24 hours later.
Could you investigate whether there was some outage or it is a bug in MBS? This could a duplicate of issue #10637.
CC: @breilly
Metadata Update from @kevin: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: medium-gain, medium-trouble, ops
I think this has been fixed also... at least looking at some recent builds they seemed to flow though pretty normally.
Can you confirm?
I cannot confirm it. I haven't built a module since then.
Looking at a difference between finishing module-build-macros and starting a next build for the recent module builds:
sway:rolling:3520220605232611:fe74cceb 3 minutes sway:rolling:3620220605232611:eef18be9 3 minutes nginx:mainline:3520220603160158:f27b74a8 3 minutes nginx-mainline:3720220603160158:9e842022 4 minutes nginx:1.20:20220603154534:8b234a03 3 minutes avocado:latest:3620220526162954:5e5ad4a0 75 minutes avocado:latest:3520220526162954:f27b74a8 7 minutes testmodule:rawhide:3520220525145817:ab7e7bfe 3 minutes testmodule:rawhide:3420220525145817:866eb18a 58 minutes
So it looks pretty good except for the 75min and 58 min ones. ;(
@breilly is this normal? Or can you investigate?
This should all be working now.
Please let us know if you still see any issues.
Metadata Update from @kevin: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
I can see similar problem again. I spawned multiple module builds and all of them are stuck just after finishing module-build-macros package. An example:
https://mbs.fedoraproject.org/module-build-service/2/module-builds/15027 finished https://koji.fedoraproject.org/koji/taskinfo?taskID=90956511 at 2022-08-18T06:30:32, the build emerge in the build target module-ghc-9.2-3620220818061242-d099bf28-build, but the module build did not advance to a next package.
The same issue with MBS tasks 15033, 15035, 15043.
The only different one is 14982 which I restarted after a normal RPM build failure and it's in a strange state.
You can see a complete list of stuck MBS tasks with "fedpkg module-overview --unfinished" command.
Metadata Update from @ppisar: - Issue status updated to: Open (was: Closed)
I restarted things and I think it's processing now.
Please check this when you are next around and see if it's working...
Those tasks finished after your restart.
I retried some of them and I submitted new ones. Most of them are stuck again: 15045 (5 hours), 15004 (5 hours), 15050 (5 hours), 15074 (12 minutes).
In addition, submitting swig:4.0 reports:
$ fedpkg module-build Submitting the module build... Could not execute module_build: The build failed with: None of the base module (platform or bootstrap) streams in the buildrequires section could be found
Isn't PDC down?
Also accessing MBS web is quite slow.
PDC is definitely not down. If it was, no one would be able to commit anything to dist-git. ;)
I'm again at a loss for whats going on here.
I am running the playbook now to see if that fixes anything. I do see some mbs builds in progress now... perhaps it's moving forward again?
Perhaps @breilly could take a peek?
I think this got fixed up a while ago. I see 0 unfinished builds right now.
Please feel free to re-open or file a new ticket if you see this again/still see this somewhere.
Login to comment on this ticket.