#2873 FESCo blocker bug: glibc missing DT_HASH data, breaking games using Easy Anti-Cheat
Closed: Accepted 2 years ago by zbyszek. Opened 2 years ago by ngompa.

With glibc 2.36, glibc upstream dropped building its own binaries with DT_HASH data as specified by the System V ELF ABI specification. The consequence of this is that video games from Steam increasingly stopped working. Notably, games failed to load because the anti-cheat module could not run, because it relied on this data.

The whole background on this can be read here:

The discussion on @devel ended with @fweimer saying "We are working on this and hope to have an update soon."

Unfortunately, we're getting close to Final Freeze and nothing has materialized.

Thus, I've filed RH#2129358 and prepared pull requests to fix the problem for us in F37 and in Rawhide.

Since we don't have any criteria to block the release on this, I'd like FESCo to step in here.

Proposal: Make RHBZ#2129358 a Fedora Linux 37 Final Blocker per FESCo.


Metadata Update from @ngompa:
- Issue tagged with: fast track

2 years ago

I'm requesting fast-track so that we can get this shipped in time before Final Freeze starts on October 4.

+1 and +1 to fast-track.

I think this is an appropriate measure to take downstream for the time being, and the PRs look straightforward. Furthermore, landing this in F37 will save people a lot of frustration.

Therefore, +1 for both both blocker and fast-track status.

I’m +1 to declaring this a blocker.

Just to be clear, I’m not dictating a specific solution to the problem (glibc upstream may have a different patch planned).

As for Fast-Track, once nominated by a FESCo member, a fast-track ticket passes the moment it reaches +7 without any -1 votes. No need to specifically vote in favor of using the FT process.

Just to be clear, I’m not dictating a specific solution to the problem (glibc upstream may have a different patch planned).

This solution I've proposed is in the event no solution comes from upstream. It's the solution other Linux distributions have gone with, so in the absence of a better solution, we can at least do this.

We're running out of time, so I want something to land soon.

Leigh, as you are not a member of FESCo, you cannot vote here.

We are pre-freeze and the fix is ready. What is the reason to block the release on this? Are you worrying the fix will not be merged if this is not a blocker?

We are pre-freeze and the fix is ready. What is the reason to block the release on this? Are you worrying the fix will not be merged if this is not a blocker?

Yes.

I am not entirely convinced we should block a Fedora release on this. I abstain. 0

I am not entirely convinced we should block a Fedora release on this. I abstain. 0

The primary motivation for blocking the release on this being fixed somehow is that it will be a PR nightmare if we don't have this resolved so that games work. Gaming capability is a serious aspect of how people review Fedora Linux these days, and we will be steamrolled when games from Steam are discovered to not work on Fedora while they work on other distros that either have glibc < 2.36 or have reverted the breaking change in glibc 2.36+.

It was already kind of bad that the beta is broken this way, but we can at least handwave it as a "common bug" if we can promise it's fixed by final release. Reviewers will try out the GA release and expect everything to work.

We provide a two-click setup for Steam and easy access to the Steam library, so it would be very bad if this remains broken for those games.

I agree that this needs to be fixed. I am just not convinced we should block the release on this. If we do, I'd rather come up with some gaming release criterion (it's possible to install Steam and play games with declared Linux support) than declaring this one particular bug a FESCo blocker.

Just to be clear, I’m not dictating a specific solution to the problem (glibc upstream may have a different patch planned).

This solution I've proposed is in the event no solution comes from upstream. It's the solution other Linux distributions have gone with, so in the absence of a better solution, we can at least do this.

As far as I understand it, an upstream fix has been released by the vendor. I don't think a FESCo ticket is the right place to discuss the development processes of ISVs.

As far as I understand it, an upstream fix has been released by the vendor.

Has it? And has Epic notified game developers to update their EAC modules to fix it? Because Epic making a release is only one part of the equation.

I don't think a FESCo ticket is the right place to discuss the development processes of ISVs.

Probably not, but it is appropriate for us to discuss how we handle breakages that we would get blamed for.

Look, a big part of what I'm working on in Fedora these days is around gaming and game development. Naturally, that means I'm keeping an eye out for things that can cause Fedora to look bad in that context.

If it is true that Epic has released a fix for EAC, then we can probably skip merging my PR for Rawhide, but I'd still like to have it for F37 so that games still work while they work through updating their EAC modules.

I think we shouldn't release F37 without an update with the fix in place. From users' POV, this is a regression. Waiting for fixes from ISVs is completely infeasible — it'll be at best months before a significant number of games is fixed. I think we should merge the work-around patches ASAP, to confirm that they really fix things and that there are no regressions. Thus, +1 to declare this a blocker.

Actually, based on @zbyszek's comment, it may still make sense to merge it in Rawhide too.

There are a number of EAC users that are unlikely to receive an update within the next year or so, and breaking the ability to play those games would seriously suck. And we really don't take a penalty for shipping the DT_HASH data for the glibc ELF binaries.

The PR, perception, and reviewer concerns over this I think outweigh the technical concerns. I agree that F37 should include a fix for this or else we will forever be answering the question "hey remember that Fedora release where you guys broke Steam?" or something like that. I really don't want to see Fedora get that treatment, but no amount of explanation will ever change the public perception. I'm confident the glibc team will address this, we just need to wait for their fix to get incorporated.

+1 to declare this a blocker

Would flatpak steam be affected by this?, I believe flatpak bundles everything including glibc.
I could pull the rpmfusion steam package from the f37 repo until it's fixed.

Would flatpak steam be affected by this?, I believe flatpak bundles everything including glibc.

It'll be affected once they upgrade glibc in their runtime.

I could pull the rpmfusion steam package from the f37 repo until it's fixed.

That would break people's ability to upgrade and remove access to a lot of games, even ones that aren't affected by this problem. When the fix is incredibly easy, it's not worth considering that.

I agree that this needs to be fixed. I am just not convinced we should block the release on this. If we do, I'd rather come up with some gaming release criterion (it's possible to install Steam and play games with declared Linux support) than declaring this one particular bug a FESCo blocker.

That's possible to do for F38, but new criteria don't get added to current releases, so if it's going to block, it needs FESCo to say so.

As of now, we have (+6, 1, 0) which means that this will be a blocker once the one-week timer runs out, unless one more FESCo member offers up a +1 (and no one pulls out a -1, forcing it to go to a meeting).

Metadata Update from @zbyszek:
- Issue tagged with: meeting

2 years ago

Late +1 here from me.

I agree with the comments around perception. The PR looks pretty straightforward and it's a clean revert for glibc.

So we're now at (+7, 1, 0) and nobody has no members have opposed the fast-track procedure. (I set the meeting tag in preparation for today's meeting, but now that we have the votes, we don't need to talk about it, so I'll withdraw it.)

APPROVED: (+7, 1, 0)

EDIT: I adjusted to text to not say "nobody". I saw Leigh's comment, and I didn't ignore it, but it doesn't count towards the vote.

Metadata Update from @zbyszek:
- Issue untagged with: meeting
- Issue tagged with: pending announcement

2 years ago

Metadata Update from @zbyszek:
- Issue untagged with: pending announcement
- Issue close_status updated to: Accepted
- Issue status updated to: Closed (was: Open)

2 years ago

Login to comment on this ticket.

Metadata