Well, we pretty much have our decision made for us, as there is a mesa now in stable updates that needs the new libglvnd. Since we have no way to remove stable updates, we must allow this particular libglvnd update to stable, IMHO.

Longer term I would need to look more at the assocated bug and changes...

So, we need to allow a package that will break Fedora for many people because if we don't Fedora is (already) kind of broken for everyone... That's awesome!

Updates can not be unpushed from stable, but a new update can be pushed (and eventually an epoch can be added if a downgrade is needed)... Just saying..

@fale can you expand on 'many people' ? From what I can tell reading the bug and update the only (known) breakage is sway, and @jwrdegoede has agreed to look at that as soon as he finishes traveling tomorrow. Are there other packages/users broken by this?

I don't know if there are, but even if it's only sway, is still "enough" people imho. Also being sway a Window Manager this could mean that some users will not be able to run any graphical session on their computers. Also, I think that updating such critical piece (mesa is in the critical path) should require even more attention.

The fact that @jwrdegoede agreed to have a look at this tomorrow is good, and I think until the sway problem gets fixed, this update should not land in stable (otherwise users will be affected)

@fale When I checked earlier you hadn't even filed an upstream bug report (sway/wlc)!, when do you intend to do so?

Saying what exactly? When I install Hans version of mesa everything breaks?
We are not talking about rawhide here, we are talking about F25 that should never be broken.

Also, this week is Devconf week and I've been traveling for the whole week (as many others people)

Sway doesn't equal everything.
I haven't seen any other programs or DE's broken by this change yet.

No package in Fedora 25 should break, and if sway is a package, it should not be broken.

Said so, if sway is the only package to be broken by this update is good, because after sway gets fixed the update is clear to go ahead, imho

IMHO that should have been introduced as a change for Fedora 26 in rawhide, not in a stable release. First of all it breaks sway (which I'm using, have to move to i3 now, so back from Wayland to X), which means we have some kind of break here and therefore some change with fallout. Second: How do we know that nothing else breaks? As a maintainer of a package broken by such a change I'd be very upset. You have quite no chance to fix that in time if bigger changes to the package or even harder source code are required. Thats exactly what we have rawhide and change proposals for… To investigate such things with enough time and avoid breakage @users.

IMHO the best solution would be undoing that change for 25 by an update of mesa undoing the changes.

+1 to what @lupinix wrote above.

According to new comment in https://bodhi.fedoraproject.org/updates/FEDORA-2017-9f6383547e the update also breaks other stuff (stream in this case). I know that steam is not a Fedora package, but this case shows once more that this update has high potential to break things.

We appear to be in a situation where updating F25 Workstation installs renders users systems unbootable. See e.g. https://bugzilla.redhat.com/show_bug.cgi?id=1417431#c9 or https://plus.google.com/+AceKC/posts/CYignV3M2xF

This seems to be caused by having half of the update (mesa) in stable, and the other half (libglvnd) stuck in updates-testing. I didn't comment on this earlier because it seemed to be on its way to getting resolved today, but apparently the libglvnd update didn't make it to today's stable push either, as per https://bodhi.fedoraproject.org/updates/libglvnd-0.2.999-7.gitdc16f8c.fc25#comment-555869

Since mesa is split up into a number of smaller subpackages, it's only a subset of these that are being kept back by broken deps and users are now getting partial mesa updates. It was during yum times that such dependency issues held back the entire transaction; these days both DNF and PackageKit have more advanced depsolvers that manage to go on despite broken dependencies and do a partial updates. And this appears to be breaking updated systems in our default install.

This needs to be solved now. Temporarily broken steam is OK. Workstation updates rendering user systems unbootable is not.

I think there's two ways to solve this. One is to fast track the libglvnd update to stable and then continue discussing if it should stay like this. The other way is to fast track a mesa revert and then continue discussing.

We need one of these NOW, it can't take a few days. And then we can take time and figure out how to make steam/sway properly work again.

CC @jwrdegoede

The other way is to fast track a mesa revert and then continue discussing.
We need one of these NOW, it can't take a few days. And then we can take time and figure out how to make steam/sway properly work again.

IMHO that's exactly what we need, a mesa revert and then we have to investigate the issues in detail.

Right,

I already mailed Kevin Fenzi earlier today to also unpush the mesa version since he has unpushed the libglvnd update:

"Hi,

As you've unpushed the libglvnd update, can you please also unpush
the matching mesa update:

https://bodhi.fedoraproject.org/updates/FEDORA-2017-9c9c0899f9

And then add the mesa pkg from that one to the libglvnd bug:

https://bodhi.fedoraproject.org/updates/FEDORA-2017-9f6383547e

So that both mesa and libglvnd are again one update in bodhi as
was the intention all along ?

Then I'll look into fixing sway and update the libglvnd update
with fixed pkgs, iow I intend to use the libglvnd update as
the single update to track this.

Regards,

Hans "

I still believe this is the best resolution to fix the dependency issues now, then I can go and fix sway, wait a bit for testing and then push things to stable again when everything is in order.

@jwrdegoede That mesa update is in stable repo, Kevin says it can't be unpushed.
There are 2 choices

1. push libglvnd to stable and try and fix sway (high motivation).
2. push a reverted mesa and try and fix sway (zero motivation).

I'm +1 for solution #1

As noted we cannot unpush stable updates. That is not an option. Sorry.

The other day I wasn't able to get dnf to partially update things, but starting from the GA packageset I can get it to just partially update things. So this does break new installs.

So, based on that we need to fix it faster.

Here's a master plan we came up with after an IRC discussion: (copy-paste from IRC)

21:37 < kalev> 1) I'll go and build a mesa -5 that reverts the libglvnd support
21:37 < kalev> 2) nirik cancels ongoing pushes
21:37 < kalev> 3) I'll go and submit a bodhi update with mesa -5
21:37 < kalev> 4) folks will karma it up
21:37 < kalev> 5) we'll queue it directly to stable (I believe bodhi allows that if it has karma)
21:37 < kalev> 6) nirik does a stable push with just a single mesa -5 package
21:37 < kalev> 7) meanwhile I'll go and do a -6 mesa build that adds the libglvnd support again
21:37 < kalev> 8) nirik unlocks the libglvnd update so that I can edit it
21:37 < kalev> 9) I'll add the -6 mesa build that adds libglvnd support to libglvn update and queue it for testing
21:37 < kalev> 10) nirik resumes normal pushes
21:37 < kalev> with this we'll have reverted mesa in stable and an atomic libglvnd update in testing

A quick status update here:

We've fast tracked mesa -5 build, https://bodhi.fedoraproject.org/updates/mesa-13.0.3-5.fc25 to stable, which reverts the libglvnd support in order to fix the broken deps in stable and hopefully avoids any more users breaking their systems with the update.

A mesa -6 build adds the libglvnd support back and is on its way to updates-testing, this time bundled together with the libglvnd update so that users should hopefully get the two in lock step. https://bodhi.fedoraproject.org/updates/FEDORA-2017-9f6383547e

FWIW ack for the plan from irc.

Thank you for taking care of this, and sorry for causing such a big mess :|

HI all,

I've fixed wlc (and thus sway) and a fixed wlc is building for f25+ now:
https://koji.fedoraproject.org/koji/taskinfo?taskID=17524308
https://koji.fedoraproject.org/koji/taskinfo?taskID=17524338

I've also submitted a pull-req with the fix upstream:
https://github.com/Cloudef/wlc/pull/233

And I will add the f25 build to the libglvnd+mesa update in bodhi when it is done building:
https://bodhi.fedoraproject.org/updates/FEDORA-2017-9f6383547

Regards,

Hans

I still think that this is not a suitable update for a stable release.

Why is this marked for the FESCo meeting? There doesn't seem to be anything actionable here outside of what has already been done, so I'm confused what we need to discuss.

Why is this marked for the FESCo meeting? There doesn't seem to be anything actionable here outside of what has already been done, so I'm confused what we need to discuss.

It was unclear whether we wanted to
1. Assert that this change was unacceptable for a stable release and force a downgrade.
2. Figure out if there's anything we can do to avoid similar issues in the future.

I want to make a general statement here from the viewpoint of the Workstation Working Group and that is that this is a crucial feature for a lot of people on the desktop, in fact I think it is quite uncontroversial to say that the amount of people negatively impacted by the current state of things in regards to hybrid graphics support outnumber the amount of people using sway or similarly obscure pieces of software by a ratio of a hundred to one. So while there might have been some snafus here in terms of clearly stating the plans in the tickets and wiki pages due to the transition of ownership of this issue from Adam Jackson to Hans I do feel that the plans where very publicly communicated and Hans did leave the update in testing for a long time here to ensure people had time to test this, to try to ensure we catch any bugs triggered by this change.

So I do hope that when the time comes for FeSCO to look at this again there is a clear consideration of the amount of people positively impacted by this change vs the number of people likely to be negatively impacted.

I want to make a general statement here from the viewpoint of the Workstation Working Group and that is that this is a crucial feature for a lot of people on the desktop, in fact I think it is quite uncontroversial to say that the amount of people negatively impacted by the current state of things in regards to hybrid graphics support outnumber the amount of people using sway or similarly obscure pieces of software by a ratio of a hundred to one.

1. Are you sure you can speak on behalf of the Workstation Working Group? If you are summing up the Group opinion that was expressed formally in a meeting, please specify it (and link it).

2. I think that continuing to say that the number of users affected by this change in a positive way is much bigger than the number of users affected in a negative way is absolutely pointless. We are talking about a feature introduction and a big change to how stuff gets rendered in Fedora. Fedora has policy and guidelines around those big changes and this update IS NOT respecting those. If there would be 0 users affected negatively could be a different situation, but since there are, I think this change should NEVER reach F25.

So while there might have been some snafus here in terms of clearly stating the plans in the tickets and wiki pages due to the transition of ownership of this issue from Adam Jackson to Hans I do feel that the plans where very publicly communicated and Hans did leave the update in testing for a long time here to ensure people had time to test this, to try to ensure we catch any bugs triggered by this change.

1. The fact that this issue got moved from Adam to Hans is not a justification of what happened, IMHO.

2. Reading the bugzilla bug, this FESCo ticket and the bodhi change, is clear that Hans was in hurry to push this update to stable and this ticket is the only reason why that did not happened. So, no I think we can not say that "Hans did leave the update in testing for a long time here to ensure people had time to test this"

So I do hope that when the time comes for FeSCO to look at this again there is a clear consideration of the amount of people positively impacted by this change vs the number of people likely to be negatively impacted.

As said, I think this is absolutely irrelevant.

Reading the bugzilla bug, this FESCo ticket and the bodhi change, is clear that Hans was in hurry to push this update to stable and this ticket is the only reason why that did not happened. So, no I think we can not say that "Hans did leave the update in testing for a long time here to ensure people had time to test this"

That is simply not true. I had disabled auto-karma explicitly (it is on by default) to allow for ample testing and I must say do not like being accused in this way, please stop assuming bad faith where there is none.

I want to make a general statement here from the viewpoint of the Workstation Working Group and that is that this is a crucial feature for a lot of people on the desktop, in fact I think it is quite uncontroversial to say that the amount of people negatively impacted by the current state of things in regards to hybrid graphics support outnumber the amount of people using sway or similarly obscure pieces of software by a ratio of a hundred to one.

Are you sure you can speak on behalf of the Workstation Working Group? If you are summing up the Group opinion that was expressed formally in a meeting, please specify it (and link it).

This is part of the formal Fedora Workstation plans and have been for a long time -
https://fedoraproject.org/wiki/Workstation/Tasklist

I think that continuing to say that the number of users affected by this change in a positive way is much bigger than the number of users affected in a negative way is absolutely pointless. We are talking about a feature introduction and a big change to how stuff gets rendered in Fedora. Fedora has policy and guidelines around those big changes and this update IS NOT respecting those. If there would be 0 users affected negatively could be a different situation, but since there are, I think this change should NEVER reach F25.

When this was discussed with our Mesa developers and domain experts it was decided that for users of the open source drivers this would not be a big change. You are of course entitle to subjectively disagree about the scope, but please don't assume that the decision process was 'lets push this gigantic change just for the heck of it'

So while there might have been some snafus here in terms of clearly stating the plans in the tickets and wiki pages due to the transition of ownership of this issue from Adam Jackson to Hans I do feel that the plans where very publicly communicated and Hans did leave the update in testing for a long time here to ensure people had time to test this, to try to ensure we catch any bugs triggered by this change.

The fact that this issue got moved from Adam to Hans is not a justification of what happened, IMHO.

Reading the bugzilla bug, this FESCo ticket and the bodhi change, is clear that Hans was in hurry to push this update to stable and this ticket is the only reason why that did not happened. So, no I think we can not say that "Hans did leave the update in testing for a long time here to ensure people had time to test this"

Hans already responded to this attack.

So I do hope that when the time comes for FeSCO to look at this again there is a clear consideration of the amount of people positively impacted by this change vs the number of people likely to be negatively impacted.

As said, I think this is absolutely irrelevant.

Well I am not going to tell you what you can or can not think

That is simply not true. I had disabled auto-karma explicitly (it is on by default) to allow for ample testing and I must say do not like being accused in this way, please stop assuming bad faith where there is none.

It was not meant to be an accuse, and I'm not assuming bad faith. I'm sorry if this is what passed in my message

• AGREED: Allow the update, encourage everyone to work through the process better next time (7,0,0) (jforbes, 16:51:10)