#6898 Fedora 27 mass rebuild
Closed: Fixed 6 years ago Opened 6 years ago by mohanboddu.

The fedora 27 schedule[1] has a mass rebuild scheduled for Jul 12, We need to plan and coordinate all tasks in preparation for it.

[1] https://fedoraproject.org/wiki/Releases/27/Schedule


Please note: https://pagure.io/fesco/issue/1735 before starting the mass rebuild.

Did this mass rebuild happen? I am looking into the scalability of the Fedora koji deployment and am curious how many packages were rebuilt and how long that took. Any details you have would be appreciated. Thanks.

@dgregor We are waiting on https://pagure.io/fesco/issue/1735 and once thats cleared I can inform you when we are going to start and we can get some statistics out of it.

@mohanboddu: what is the status of the mass rebuild ? https://pagure.io/fesco/issue/1735 does not seem to block it anymore.

@jkurik Mass rebuild started on Tue night US EDT time. Its still running, its going to take around 5-7 days.

Please may we have another mass rebuild ?

I have updated the rawhide binutils to implement the change suggested by Florian, but in order for it to take effect all libraries and compiled executables need to be rebuilt. If it helps, the rebuild is only necessary for the PowerPC architecture, not any of the others.

Thanks.

@nickc We are planning on doing another mass rebuild skipping noarch packages. But it has to wait until the current mass rebuild is done.

Starting another mass rebuild immediately following the one we just had will affect a number of critical infrastructure team projects, such as the pagure-dist-git deployment. That is critical for a number of other people outside our teams. Can we schedule the mass rebuild to follow that?

Rawhide is currently in a really bad shape on ppc64le because key libraries were mislinked. We'll have to recompile an unknown number of components manually, but it should be doable. This work is is required for the second mass rebuild anyway, otherwise there will be too many failures.

If you could let us know here when those key packages are done we can figure out when to do the next mass rebuild.

I will note that the only interaction between the mass rebuild and the pagure over distgit work is committing the spec bump to distgit. Once those are done the pagure over distgit work could go ahead and the mass rebuild will just build everything its already building. Also the next mass rebuild only will need archfull packages which will cut down the time/number involved a great deal.

If you could let us know here when those key packages are done we can figure out when to do the next mass rebuild.
I will note that the only interaction between the mass rebuild and the pagure over distgit work is committing the spec bump to distgit. Once those are done the pagure over distgit work could go ahead and the mass rebuild will just build everything its already building. Also the next mass rebuild only will need archfull packages which will cut down the time/number involved a great deal.

Right -- that's how I understood the issue as well. The only concern I have is giving @pingou enough clearance to do his work. AIUI there have already been some delays and each additional one at this point makes it more unlikely we can hit milestones by Flock time. It doesn't seem like there's a good alternative where everyone gets things quickly, sorry for that. My hope is that @pingou, @mohanboddu, and other folks can find the least-bad solution.

What's the repository used for the current ppc64le buildroot? I tried this:

But it's not what the builders use. For example, they install rpm-build 4.13.0.1-40, but the repository above currently has version 4.13.0.1-39 only.

I need the right repository to minimize rebuilds, otherwise I'm down to trial-and-error.

I'm now using

for reference. It seems more recent than the rawhide repository, so maybe this is the right one.

There seems to be an ongoing issue with the buildroot generation, and new packages do not show up in it. Until that happens, I cannot start the targeted rebuild effort in earnest. (The immediate goal is to unbreak cmake.)

f27-build is correct. "rawhide" is just one we have as an alias.

While the mass rebuild was finishing it may have been a bottleneck with signing packages. I am not sure what it is currently, but will investigate.

So what is the status on this? Do we start a new mass-rebuild? When? How long will it take? (time-wise what impacts me is how long it takes to commit and queue all the builds, not actually finish the builds)

I saw some discussion on the devel list about progress being made over the week-end but it wasn't really clear for me what the outcome was :s

My targeted rebuilds for the ppc64le issue are still ongoing, but are nearing completion.

My targeted rebuilds for the ppc64le issue are still ongoing, but are nearing completion.

So all interaction with git is done? There is no script running that would crash is git was not accessible for a minute or two?

@pingou, it's mostly manual anyway. I don't need dist-git right now.

I believe the ppc64le cleanup is done for now (as long as there aren't any untagged ppc64le builds waiting to be tagged into f27-build at this point).

Just to keep this ticket updated:

  • all the noarch packages that were rebuilt in the rebuild last week have been tagged into rawhide.
    (todays rawhide compose failed, but they should show up soon).

  • All the archfull packages are being mass rebuilt again now in a new tag.

This should be complete now. Please re-open if there's anything further to do here.

:checkered_flag:

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

6 years ago

Login to comment on this ticket.

Metadata