#131 Red Hat Bugzilla: GNOME packages are in bad shape
Opened a year ago by catanzaro. Modified 5 months ago

I think most Fedora GNOME developers have simply given up on Red Hat Bugzilla. Currently, it's where bugs go to be ignored and receive no response, including serious bugs. This status quo is unfair to users.

Downstream, we are generally only passingly familiar with the packages we own, and we do not have time to even look at most bugs, let alone triage them. I checked a couple other GNOME maintainers to see how many bugs are assigned to them and found: 260, 182, 420, 511, 372. The only reason I don't have this many myself is that I don't own dozens of packages, unlike most of the other GNOME maintainers. These numbers are way too high for us to practically deal with triaging them. And we can't practically move the bug reports upstream either; it just takes too long, we'd spend so much time moving bugs that we'd never get anything else done. We need the bug reporters to do that for us.

So reality is: we need to get the bugs upstream to be able to not ignore them. Upstream maintainers generally have dramatically fewer packages to maintain (usually only a couple packages rather than dozens) and much more specialized expertise in their packages. Upstream, there's a decent chance the bug reports won't be ignored, if the maintainers are good about managing their issue trackers. Some GNOME maintainers are good at this. Some are not. To the extent upstream maintainers ignore their issue trackers, that's a separate issue and one that is more solvable than the mess we currently have on Red Hat Bugzilla. Upstream, the bug reports at least go straight to the right developers. Downstream, on Red Hat Bugzilla, there's just no chance. We'd need way more downstream developers just to look at all the bug reports we receive.

I suggest we set up an automated script to close all bugs in GNOME packages, except bugs that are marked with some flag or keyword to indicate the issue is (a) a downstream packaging issue, (b) a proposed blocker, freeze exception, or Fedora priority bug.

This is related to #130.


Metadata Update from @chrismurphy:
- Issue untagged with: meeting
- Issue tagged with: meeting-request

a year ago

Is GNOME the only upstream where this is an issue?

No, this is also a problem with KDE. I'm pretty sure @rdieter would like to see this work for submitting to KDE Bugzilla too.

However, I think it makes more sense to file the bugs both upstream and downstream, and have linked bug reports in both directions. That way Fedora processes still work, and GNOME/KDE people can focus on upstream reports.

I suppose it might also be a problem for Pantheon (@decathorpe can chime in on this)...

I suppose it might also be a problem for Pantheon (@decathorpe can chime in on this)...

To some extent, yes. (TL;DR at the bottom)

For example, GNOME upstream changes DBus interfaces or GSettings schemas with almost every major release, and those changes often introduce crasher bugs in Pantheon / elementary apps. And since new GNOME versions always land in fedora at the last possible minute, this makes it pretty hard for me to keep Pantheon working on stable releases ...

These changes are not necessarily upstream "bugs", but they still frequently break things in fedora - so I always thought that reporting them in GNOME GitLab was the wrong place and opened bugs in RHBZ instead.

However, it often takes weeks for somebody to look at those bugs (if at all), at a time where fixes would be time-critical (just before the final freeze, usually). For fedora 31, I was only able to fix Pantheon after GA (due to the switch to systemd user session stuff completely breaking Pantheon), which was ... not good. It is often more productive to ask elementaryOS upstream devs for help to come up with workarounds than to wait for somebody to respond to my RHBZs.

(Just to clarify: I think that elementary upstream should fork gnome-session, gnome-settings-daemon and mutter, to be rid of those frequent issues. They're still stuck at support for mutter <= 3.28 - almost two years old now - because every mutter release starting with 3.30 included major breaking changes ... But they just don't have the manpower to maintain those desktop components themselves.)


TL;DR: Basically, I'm usually the first person to notice that some change from GNOME upstream breaks Pantheon, GNOME GitLab is the wrong place to report those breaking changes since they're just "features, not bugs" (so I've been told), elementaryOS is struggling to keep up to support new GNOME releases, and I'm their first line of defense, without even wanting to be in that position - and it doesn't help that bug reports for "this fedora update broke my packages" are most often ignored.

However, I think it makes more sense to file the bugs both upstream and downstream, and have linked bug reports in both directions. That way Fedora processes still work, and GNOME/KDE people can focus on upstream reports.

There is significant value in having only bug reports that we actually plan to work on open in downstream Bugzilla. For example, I currently have saved searches to display gnome-shell and gnome-software bugs, but only RHEL bugs, because there are too many Fedora bugs. This means that all downstream bugs are ignored. If there were only a few Fedora bugs, then I could include the Fedora bugs in my search results so that real downstream issues wouldn't be ignored.

That said, reporting the issues upstream, in addition to downstream, would be a significant improvement over the status quo. But the downstream bug is really negative value. The only Fedora processes that I'm aware of that require downstream bug reports are the blocker, FE, and PrioritizedBug processes, and those I've already stated would remain open downstream. (E.g. our script for closing bugs could look for the relevant keywords and avoid closing bugs that are proposed/accepted blocker/FE or PriorizitedBug.)

But the downstream bug is really negative value.

This is not true. It's important to note that downstream bugs are the inputs for the characterization processes (PrioritizedBug, BlockerBug, FreezeException). If the bug doesn't exist in the first place, there's no way for that to be tagged for those processes.

Essentially, you cannot be auto-closing bugs (or worse, not even filing them) downstream without breaking how we even do distro development.

Metadata Update from @ngompa:
- Issue untagged with: meeting-request

a year ago

Metadata Update from @ngompa:
- Issue tagged with: meeting-request

a year ago

This is not true. It's important to note that downstream bugs are the inputs for the characterization processes (PrioritizedBug, BlockerBug, FreezeException). If the bug doesn't exist in the first place, there's no way for that to be tagged for those processes.

If you want a bug to block the release, you'll need to create it and link to an upstream bug report instead of finding an existing downstream bug report to use for the purpose, but this does not seem so hard.

Essentially, you cannot be auto-closing bugs (or worse, not even filing them) downstream without breaking how we even do distro development.

This proposal is primarily intended for maintainers who are currently ignoring downstream Bugzilla. I don't see how it would break workflows. If you currently read downstream bugs, obviously you would probably not want to auto-close bugs for your packages.

Metadata Update from @catanzaro:
- Issue untagged with: meeting-request

a year ago

Metadata Update from @catanzaro:
- Issue tagged with: meeting-request

a year ago

(Created https://pagure.io/pagure/issue/4749 for our fun with the meeting-request tag)

I certainly, over the years, have been a major culprit as "Fedora maintainer ignoring hundreds of bugs assigned to them". Generally, when I'd spend a day looking at Fedora gnome-shell and mutter bugs in Red Ha bugzilla, I'd find lots of interesting and intriguing stuff, and sometimes find fixes to make upstream, but getting bugzilla anywhere close to clean would have required me to be close to full-time bugzilla triager, and not get anything else done.

Crashes

I, generally, do not think that getting ABRT to report bugs upstream is a good idea. Back in the days of "bug buddy", parts of GNOME bugzilla were a wasteland. Having automated or semi-automated bug reports mixed together with human-curated things is a not a good idea.

My feeling general feeling about the approach that would work best:

  • Double down on retrace.fedoraproject.org - if we think there are things that people can do manually to make a better bug report, don't do that in bugzilla, add a way to annotate/extend a crash in retrace.fedoraproject.org. Or at least, make associating with a crash in retrace.fedoraproject.org an essential part of filing a crash bug in bugzilla, and be able to find those bugs easily

  • Put resources into making triaging crashes on retrace.fedoraproject.org a rewarding and rewarded activity. Have guides to what you should do, collect stats, publish leaders, and make sure that it from a UI perspective doesn't feel Sisyphean - it should look good if the top 10 crashes are properly triaged, even if there is a long tail of 100 more odd stack traces that haven't been triaged.

Everything else

Somehow make part of the filing experience for GNOME components, pre-filing:

"If you do not think this is a packaging problem or other Fedora specific problem, please file it upstream"

because while automatically filing crashes upstream (or anywhere) is hard to manage, requests for UI changes, for example, are better handled upstream.

The only Fedora processes that I'm aware of that require downstream bug reports are the blocker, FE, and PrioritizedBug processes, and those I've already stated would remain open downstream. (E.g. our script for closing bugs could look for the relevant keywords and avoid closing bugs that are proposed/accepted blocker/FE or PriorizitedBug.)

Another exception: security tracker bugs. Just got pointed to yet another security bug we had ignored. Currently they get ignored along with everything else. If we could turn off the fire hose, maybe these wouldn't slip by....

I, generally, do not think that getting ABRT to report bugs upstream is a good idea. Back in the days of "bug buddy", parts of GNOME bugzilla were a wasteland. Having automated or semi-automated bug reports mixed together with human-curated things is a not a good idea.

We know ABRT reports to Red Hat Bugzilla work quite well, though (at least up until it started marking everything as private and intentionally reporting duplicate issues for private bugs, both solvable problems).

I think the problem with Bug Buddy was simply that Bug Buddy would file dozens upon dozens of duplicate reports for the same issue. ABRT is basically already smart enough to not do this; it just needs to trust its own heuristics a bit more and more aggressively assume bugs with similar-looking backtraces are probably duplicates rather than opening a new bug and marking it as a "Possible Duplicate" for a human to review. Having high-quality crash reports in the main project issue tracker seems extremely valuable. I don't think the situation is comparable to Bug Buddy.

Although I agree with your suggestions for retrace.fedoraproject.org, I think we'd need further planning to determine a path towards making it useful for upstream developers who don't use Fedora and are not interested in downstream tooling.

This involves a few elements: retrace, GNOME gitlab, RHBZ, Fedora QA, ABRT devs. But what is the WG role? Are there UX choices that the WG should have a say in? Is this ready for meeting time?

The WG certainly has a say in how we handle bugs for desktop packages. I think we're ready for meeting time.

ABRT is basically a separate issue. Ernestas has already indicated interest in reporting directly to GNOME GitLab, should we decide that's what we want to do.

And this has basically nothing to do with Fedora QA. I'd say any bug that's important enough for Fedora QA to take interest can continue to be tracked downstream. We don't have to close everything. The goal is to just reduce the size of the firehose.

I'm following this issue in Pagure, but if I can provide any help as Program Manager, please feel free to pull me in to meetings or ping me here or via email.

Metadata Update from @chrismurphy:
- Issue untagged with: meeting-request
- Issue tagged with: meeting

a year ago

So we've agreed the GNOME packages are in bad shape in Red Hat Bugzilla, and we should probably not leave them like this. We don't have consensus on how to handle the situation.

Metadata Update from @chrismurphy:
- Issue untagged with: meeting

a year ago

As a first and trivial step, maybe ask users to file bugs in upstream bugzilla:
1. add add to Description in packages:
"
Please report bugs in https://bugzilla.gnome.org/...?component=foobar"
2. change Bug URL to point to the upstream tracker.

GNOME stuck a knife into its Bugzilla instance, so you have to figure out where in their GitLab each component is.

I'm sure the majority of components can be found in https://gitlab.gnome.org/gnome/foobar

An officially maintained mapping would be a pre-requisite for this. If there isn't one, then this will not be great.

I'd think that it'd be enough if the maintainers would just verify the URL when changing it. It's a one time thing.

By the way, tools such as ABRT could be capable of handling appstream files. In these, the <url type="bugtracker"> node is set by upstream. https://www.freedesktop.org/software/appstream/docs/chap-Metadata.html#tag-url

The new service Packager Dashboard illustrates the problem here quite nicely.

This is one of these reoccurring issues that goes back more than a decade ( we discussed this same issue in QA when James Laska was there ) the solution we came up with to be the best way to address this was to extend RHBZ to be able to communicate with upstream BZ then downstream package maintainers could just "forward" the report upstream once they had triage it if relevant.

That idea was shutdown by RHBZ maintainers at that time due to mainly security concerns ( if I can recall correctly ) because RHEL ( and thus it's customers ) shared the issue tracker with Fedora.

I personally have administrated an issue tracker ( jira ) in which we where doing exactly that for over 1000 projects ( not exactly the size of Fedora but measurable never the less and granted between jira instances but basically you are just sending mail between those instances ) with no problems so this problem can most certainly be solved on a technical level and tailored to each projects workflow for that matter however that solution does require Fedora to gain it's own issue tracker for the project separated from RH and thus can be tailored to the projects needs and not be a security risk to RH customer base and b) bear in mind this does not mean the issue has been resolved as in you still need to have enough resources ( manpower ) on the receiving end to address the issue so if a project is already starved of resources it does not matter where it's issue resides.

What has change since back in those days is that reporters by large, regardless of distribution have started to file reports directly upstream if the project resides on github due github currently being the largest social development platform on the planet so Gnome might want to consider switching the roles for it's repositories as in Gitlab becomes the mirror while Github becomes the primary which could to a certain extent solve this problem while also attract new people and contribution to the project.

I probably will get flamed for saying this but arguably Fedora as a project should do exactly the same thing, stop this outdated nonsense of having to run it's own infrastructure on self made and coupled together solution that never quite work out or gain popularity and cost tremendous resources ( and money for RH ) and move itself closer to where people ( and upstream ) actually are and I strongly recommend that the leaders of the project along with relevant parties high enough within RH sit down and have a serious discussion about doing this for the long term sake of the project.

If we want different results, we have to try different approaches. It's that simple.

So we've agreed the GNOME packages are in bad shape in Red Hat Bugzilla, and we should probably not leave them like this. We don't have consensus on how to handle the situation.

Last time we discussed this, we agreed we could do a small trial with a few GNOME packages to see how it goes, but no more than that. Now ABRT developers have actually implemented this feature. I've volunteered epiphany and glib-networking packages to begin this trial, and Ray has volunteered gdm and accountsservice. This isn't really going to be a terribly interesting trial to start with, because these packages have almost no ABRT reports from the past couple years, but that's because ABRT has been in such bad shape that reporting almost always failed (#130) even prior to #156. But it should be a very low-risk way to start experimenting with upstream reporting.

FYI: tpopela determined we had 33 unresolved desktop security bugs in Fedora, which had been ignored by the Bugzilla assignees. Many of these are 2+ years old. I've closed most of them just now, except for a couple that haven't yet been handled by upstream rebases. But this was all one-time work: we have no way to track such problems going forward.

@catanzaro the list that you got was probably a half of what it was last week before I (and others including you) cleaned them as well. So it was probably 60+.

So I managed to get the original list and it was 115 security bugs (not all include all the GNOME packages, there was Qt, grub2, ..).

Login to comment on this ticket.

Metadata