Currently, RPM in RHEL 8 has the code to support zstd compression but it is not compiled in. I have spoken to the Red Hat Product Manager for this area and he said if FESCo approves this proposal, he will try to get some level of support in RHEL 8. i would not expect RHEL 7 to have support added.

I haven't yet read trough all the discussion on devel, but it seemed to me at first that the benefit here is very questionable and the problems might not be worth it.

I'm also likewarm on this. It would be a lot of incompatibility and work so it would really need to be clearly worth it.

@churchyard I don't think that benefit is questionable. In ideal conditions, decompression speed is about 15x faster. The improvement is not that great in real world - there's definitely space for improvement in rpm's install code, but that's something we can work on with no Fedora Changes needed.

Last week I tried to re-compress RPMs on RHEL installation DVD and ended up with roughly 8m default installation (~1200 RPMs) instead of 10m. 20% down is not entirely bad, isn't it?

@churchyard @kevin could you be more specific about the problems?
I'm currently aware of 2:

• RHEL & CentOS unable to install Fedora RPMs due to zstd compression (we can eventually fix that by backporting zstd support into el8)
• deltarpm will need zstd support; there was discussion whether we shouldn't drop deltas as they bring only little benefit (that would probably need further discussion)

None of them doesn't seem to be a showstopper to me.

Could you also share your criteria when it's "worth it"?

Currently, RPM in RHEL 8 has the code to support zstd compression but it is not compiled in. I have spoken to the Red Hat Product Manager for this area and he said if FESCo approves this proposal, he will try to get some level of support in RHEL 8. i would not expect RHEL 7 to have support added.

Actually, this is not accurate. Modularity in Fedora and RHEL 8 has reminded us that there are ripple effects of management and build systems on older releases to have at least some level of compatibility when enabling new functionality. This is particularly true for the low level content.

For example, we had to scope out the impact for Satellite 5.x running on RHEL 6 to have minimal support for RHEL 8 modular content.

So there are ripple effects for Satellite 6 components (formean, pulp, and others) to understand the impact and plan for future changes to support said content. We fully anticipate Sat 6 running on RHEL 7 to support future content changes of RHEL 8++.

So you might ask yourselves, how do users manage Fedora Server at scale? Do you have an impact to Puppet, Chef, Salt, Ansible, Satellite/Forman when those management tools might be running on an older CentOS or even Fedora? Users may not want to immediately update their management tools to accommodate the new changes, so it delays adoption of your future releases.

Lastly, it will help to tell a story about how the benefits of the new technology weighs against the effort of change required for the ecosystem to support it.

I am not at all suggesting you do not proceed. Just trying to help by sharing these type of impacts on users and developer communities.

@tbowling note that this change is about compression of payload. Any tools will be able to read headers (pulp, createrepo, whatsoever). This does not have any impact apart from unpacking / installing RPMs on old distributions.

But thanks for bringing this up, this is definitely something to consider. But in any case, you can't even build Fedora RPMs on EL6 because of rich dependencies. There there are much more things which are not supported by older distros like caret versions, dynamic BuildRequires.

So you might ask yourselves, how do users manage Fedora Server at scale? Do you have an impact to Puppet, Chef, Salt, Ansible, Satellite/Forman when those management tools might be running on an older CentOS or even Fedora? Users may not want to immediately update their management tools to accommodate the new changes, so it delays adoption of your future releases.

As one of said users, I can tell you it's a major PITA when management tools are not backported timely in EL. By all means be very conservative on all the parts apps use (apache, postgresql, etc).

But system management tools? That just costs a huge amount of money when not backported (you end up having to use different tools for different releases, or forcing operators to give up on time-saving enhancements to process everything with the oldest tooling stack you have. Either solution is not cost effective).

From today's meeting:

  * AGREED: F31 System-Wide Change: Switch RPMs to zstd compression is
    approved. (+5, 0, -0)  (contyk, 15:43:34)

We had some concerns regarding the impct on various infra services we run, from builders to COPR. But coordinating all of that is out of scope for FESCo. I'd suggest you pull some stakeholders into the releng ticket to make sure everything's fine: https://pagure.io/releng/issue/8395