#4224 break build inheritance between F14 and F15 and update deltarpm files

Created 6 years ago by toshio
Modified a month ago

After talking with jnovy and jdieter about the new xz and its effect on deltarpms we've come up with this plan:

  1. rpm moves to the new xz as soon as possible. This should just be a rebuild. It's needed so that the rpms being created from that point forward all use the same version of xz for compression rather than having incompatible client-side and server side tools. Doing this step soon cuts the number of packages built with the old xz and thus getting the hash warning for a second and third time.
  2. Break build inheritance from F14 to F15. This is where releng comes in. This could be done with a mass rebuild or just by making a setting change in koji.
  3. If a setting change is used rather than a mass rebuild, remove the current F15 deltarpms so only deltarpms that match with the xz people have on their systems will be available in the future.

These steps protect against the scenario where the new xz is installed clientside but the packages being installed via deltarpms were generated with the old xz (for instance, via build inheritance from f14). Rawhide users will still get a warning in the transaction where they update from the rpm/deltarpm using the old xz to the one using the new xz.

Assuming this looks okay to releng, jnovy could you update this ticket when the new rpm and deltarpm packages are built?

Breaking build inheritance has no effect on what deltarpms are produced, FYI. For rawhide, it's merely just against rpms in the prior rawhide tree.

New rpm-4.8.1-6 is now built.

Breaking build inheritance affects what rpms are in the update, though, correct? The problem that breaking inheritance solves is: Clients on F15 have been updated to new xz. gcc-4.1-1.fc14 is in F14 and inherited into F15. gcc-4.1-2.fc14 is built for F14 and inherited into F15. A delta is generated between those two. When the delta is applied on the F15 client, the generated package's hash will not match the gcc-4.1-2.fc14 package. Breaking build inheritance would mean that to update gcc in F15, a package using the F15 tools would have to be used which would use the new xz.

Does that sound correct?

OK, when phrased that way it makes sense. However, the issue I see is that that's a break of previous policy on inheritance... it would certainly need announced at a minimum, and/or run by FESCo.

So trying to wrap my head around this, what is it specifically you are trying to avoid by breaking the inheritance?

Looking at the options, I think a mass /tag/ of packages and then a koji config change would be the quickest and least intrusive way to accomplish this. It would take every build that is currently being inherited into dist-f15 (7047 builds as of this morning) and explicitly tagging them with dist-f15. Then we would make the config change on dist-f15 to stop inheriting builds from dist-f14-updates (and dist-f15-build from dist-f14-build). No rawhide churn in the package n-v-rs.

By scrubbing the existing deltas, I think you'd trigger the compose tools to attempt to create new ones. However rawhide compose scripts only look at the previous day's rawhide to build deltas against, which in this case would be unchanged, so you wouldn't have any deltas. The only time you'd get deltas is when a new build happens.

Now, even with inheritance, lets say a new f14 build gets inherited into rawhide. The rawhide compose would generate a delta against the previous f14 build, using whatever tools are currently in rawhide to generate that delta. It does not re-use any delta previously made. The deltas do not exist in koji.

So I'm not sure what this would accomplish, I need to know more about what you're trying do avoid.

The deltas themselves don't hold any information that's problematic but the existence of a delta is the trigger for reconstructing an rpm and the reconstructed rpm can lead to the hash mismatch in certain circumstances:

  1. when the client has the old xz/rpm/deltarpm but the new package has been built by rpm with the new xz the client will construct a package with a different payload than the new package leading to a hash mismatch. This is usually a one-time occurrence in the yum update that pulls in the new xz/rpm/deltarpm combination. I can't think of a way to address this but it doesn't affect people using media to upgrade from F14 to F15 as they aren't using deltarpms in order to get the new rpm. (Would this affect preupgrade or other yum upgrade paths? I don't think we have deltarpms between releases but I'm not sure.)
  2. when the client has the new xz/rpm/deltarpm but the package has been built by rpm with the old xz the same hash mismatch will occur. AFAICT, this occurs in two cases:
    1. if the client updates xz/rpm/deltarpm specifically and then updates a package where the update was built by the old rpm/xz combination. This is taken care of by removing the deltarpms for F15 generated before the new rpm/xz combo is used in the buildroots. As there's no longer a deltarpm present, there's nothing to trigger recreating the rpm on the client's machine.
    2. if the update is built using the old xz/rpm combination on F14 and build inheritance pulls that build into F15. This case is taken care of by breaking build inheritance with F14 so that an update with an old payload is never pushed to F15.

Ok, so it sounds like a mass tag, break inheritance, then scrub deltas is the way to go forward. Quite unfortunate.

I believe this should go through FESCo, and the rpm/xz update in rawhide should be delayed until FESCo approves this plan and there is proper warning sent out to developers.

Given that deltas automatically fallback to the full RPM, I'm not sure that the update needs delayed - it results in a suboptimal user experience, but nothing actually is a hard break.

All the inherited builds have been specifically tagged into dist-f15. I'm now building a mash that will not use inheritance when composing rawhide.

Mash building, this should be used in the next rawhide compose.

I'll send an announcement about this later today.

Would it be feasible to avoid this extra trouble in the future by waiting until a mass rebuild is about to happen for some other reason, then do the compression change (which doesn't happen often anyway) just before that? For example, there's supposed to be a mass rebuild due to GCC 4.6 - see [https://fedoraproject.org/wiki/Features/GCC46].

a month ago

Metadata Update from @toshio:
- Issue assigned to jkeating

Login to comment on this ticket.