#3512 Let's do something about ignored CVE bugs
Opened 2 months ago by catanzaro. Modified 2 days ago

Fedora currently has 6664 unresolved bugs with "CVE" in the title (query). I was expecting this number to be high, but not this high.

It's currently common for Fedora maintainers to ignore CVE bug reports (first example I found) (we even completed the nonresponsive maintainer process for this maintainer, but that does not result in reassignment of bug reports). It's entirely up to discretion of the package maintainer as to whether to handle a bug report or not.

It's not always possible to resolve CVEs. Some are very hard to fix, and we just don't have time. But we probably should not be completely ignoring CVEs.

Proposal: if a CVE bug report is still in NEW state after 6 weeks, some automation should set needinfo?. After 8 weeks, the automation should kickstart the nonresponsive maintainer process. Package maintainers should move the bug to ASSIGNED state (indicating they acknowledge the bug report) to avoid being caught by the automation.


The process that is used to file these bugs is kind of broken. There was a mailing list thread a couple years ago where there were multiple examples of useless CVE bugs being filed. There are bugs filed against the wrong package, filed multiple times, about issues that have already been fixed, issues that are not practicably exploitable in that specific package, or that are otherwise incorrect.

I don't think the bug reports should be any louder. If there a security bug reports that an actual person (and not a throw-it-over-the-wall script) determines are valid and the maintainer doesn't respond to it after being contacted, then the usual nonresponsive processes apply. These security bugs shouldn't have the same weight as FTI/FTBFS bugs which are always actionable and filed in a reasonable way.

Also, it might be good to post proposals like this to the mailing list first so it can be discussed more widely before being voted on.


P.S. for statically linked ecosystems like Go, the bugs are especially useless, and hundreds or thousands are filed, a bug against every package that depends on golang, after each golang CVE (usually at least one per monthly Go release), whether or not the package actually uses the impacted component. This results in thousands of bugs. We could be super safe and mass rebuild every Go package every time but this isn't practical to do on stable branches, only provenpackagers can do it properly since some Go packages have the PR required feature turned on, and it takes a lot of time.

We could be super safe and mass rebuild every Go package every time but this isn't practical to do on stable branches, only provenpackagers can do it properly since some Go packages have the PR required feature turned on, and it takes a lot of time.

Yeah, this part tanks. I'd like to see if we can make progress on changing this. If we can integrate an approach on top of #3501 in Koji, then rebuilds will not require provenpackager powers, since anyone can build any package in Koji as long as there's no need to change anything in Git.

Fedora currently has 6664 unresolved bugs with "CVE" in the title (query).

This is also ... quite misleading. For bugs that are filed by automation, there are a multiple bugs filed: 1 toplevel tracking bug + (1 bug for each active Fedora (and EPEL) branch) × (1 + number of packages where really really dumb "heuristics" determine that the project is "bundled"). So that number can grow pretty quickly without the impact actually being big.

For example, we had a CVE for the idna Rust crate, which had a (low severity issue). This resulted in dozens of bugs being filed, about half of which were bogus because the heuristics for "is this project bundled here?" were just wrong.

We tried cleaning up CVEs in EPEL a few years ago and after a while the effort fizzled, partly because a lot of these are, as noted, false positives and it got tedious.

What we might want to do is follow up with the team generating these CVE bugs - I have not gotten traction following up individually - because getting tagged for Javascript CVEs that are in documentation code that is not even being shipped is just silly at first, and downright tedious the 1000th time.

And we don't have automation to auto-close such bugs (ideally with an email to the person submitting it so they ... get to feel some of the impact)

I have gotten to the point of ignoring ffmpeg related CVE bugs because they're filed on everything and they are very often wrong too. It's just so much pain to slog through.

Based on the comments so far, the situation is complicated. Let's add it to the agenda for today, maybe we can brainstorm some steps to take.

Metadata Update from @zbyszek:
- Issue tagged with: meeting

2 months ago

We discussed this during the meeting today:
* ACTION: Conan Kudo to talk to the fedoracve.org owner/maintainers about trademark use (@zbyszek:fedora.im, 17:49:52)
* ACTION: Jef Spaleta to talk to RH ProdSec about the quality and procedures for CVE filing (@zbyszek:fedora.im, 17:50:31)

Let's wait the results of those actions.

Metadata Update from @zbyszek:
- Issue untagged with: meeting

2 months ago

I looked at some of the high-severity bugs on the list, and all of them turned out to be invalid. qtwebkit and rekonq have been retired, finally. A few other bugs that I looked at were solved through updates years ago so I closed them.

Then we have hundreds of bugs for firefox [1] and the kernel [2]. Both packages are regularly updated to latest upstream versions and any fixes would have to come from there. I don't think those hundreds of bugzilla entries are useful. I think we should close them all and say that they're being handled upstream (and do the same for future bugs). @stransky, @jforbes WDYT?

[1] https://bugzilla.redhat.com/buglist.cgi?bug_status=NEW&bug_status=ASSIGNED&bug_status=POST&bug_status=MODIFIED&bug_status=ON_DEV&bug_status=ON_QA&bug_status=VERIFIED&bug_status=RELEASE_PENDING&columnlist=product%2Ccomponent%2Cassigned_to%2Cbug_status%2Cshort_desc%2Cchangeddate%2Cbug_severity&component=firefox&f0=OP&f1=OP&f2=product&f3=component&f4=alias&f5=short_desc&f6=status_whiteboard&f7=CP&f8=CP&j1=OR&list_id=13632935&o2=substring&o3=substring&o4=substring&o5=substring&o6=substring&order=product%2C%20status%2C%20&product=Fedora&query_format=advanced&short_desc=CVE&short_desc_type=allwordssubstr&v2=CVE&v3=CVE&v4=CVE&v5=CVE&v6=CVE
[2] https://bugzilla.redhat.com/buglist.cgi?bug_status=NEW&bug_status=ASSIGNED&bug_status=POST&bug_status=MODIFIED&bug_status=ON_DEV&bug_status=ON_QA&bug_status=VERIFIED&bug_status=RELEASE_PENDING&columnlist=product%2Ccomponent%2Cassigned_to%2Cbug_status%2Cshort_desc%2Cchangeddate%2Cbug_severity&component=kernel&f0=OP&f1=OP&f2=product&f3=component&f4=alias&f5=short_desc&f6=status_whiteboard&f7=CP&f8=CP&j1=OR&list_id=13632935&o2=substring&o3=substring&o4=substring&o5=substring&o6=substring&order=product%2C%20status%2C%20&product=Fedora&query_format=advanced&short_desc=CVE&short_desc_type=allwordssubstr&v2=CVE&v3=CVE&v4=CVE&v5=CVE&v6=CVE

(I forgot to add: I had one bug for my own package in that list. After scratching my head for a few minutes, I found out an unused jar file in sources and closed it as NOTABUG too.)

If there any opened CVE for Firefox we can close them - we follow upstream closely and we don't have any known open vulnerabilities there.

The 6,664 open CVE bugs show this is a systemic workflow issue, not a maintainer time problem. Your finding that many reports are unactionable confirms the quality gap.

Security Engineers creates these bugs with best-effort data, but without deep per-package analysis of Fedora’s build configs. This means many bugs represent potential exposure, not confirmed impact, the main source of noise. Even if we cleaned all ~6,664 bugs today, the backlog would return unless the workflow changes.

The long-term fix is adopting SBOM + VEX.

SBOMs define exactly which components Fedora actually ships. For example, Firefox bundles dozens of libraries, but many vulnerable ones may be disabled or patched at build time. SBOM metadata lets automated scanners ignore components not present, eliminating false positives caused by broad CPE/text matching.

VEX provides machine-readable impact states—e.g., “Not Affected: component not present,” “Not Affected: code path unreachable,” “Patch applied,” or “Affected.” This stops irrelevant CVEs from becoming bugs and removes most manual triage.

Together, SBOM + VEX ensure only real Fedora-affected packages get CVE trackers, improving data quality and preventing backlogs.

The goal is not to hide numbers, but to make the numbers meaningful. Transparency is a strength, not a weakness, So Improving the workflow with SBOM and VEX gives Fedora accurate, high signal vulnerability tracking without placing an unbounded triage burden on maintainers.

To move forward, we should reactivate the Security SIG to formalize and drive this SBOM/VEX workflow. FESCo will ultimately need to decide what to adopt.

I remember discussing SBOMs with @richardfontana years ago at a Red Hat Summit when we talked about SPDX compliance stuff, and I'll point out now as I did then: we cannot expect packagers to provide accurate data if it needs to be done by hand.

It's already hard enough as it is with the existing bundled libraries policy to audit and document things, because it requires really digging into the sources to figure it out.

I think us adopting this stuff is going to require significantly more investment into automation and figuring out how to blend that information into our repository metadata for consumers to fetch as needed.

I closed all the firefox cve bugs now.

One particularly bad case that I was just reminded of (got a notification email because it also "affected" firefox): https://bugzilla.redhat.com/show_bug.cgi?id=2357560 (for a use-after-free bug in rust-openssl, the Rust bindings for OpenSSL)

The bugs opened by a bot (OSIDB Bzimport, whatever that is) that appears to not get monitored. 24 bugs opened for "affected" packages that were either

a) not depending the affected code at all (some were not even written in Rust, like mingw-openssl),
b) did depend on rust-openssl (directly or indirectly) but did not use the affected functionality.

On the other hand, some packages that do depend on rust-openssl and do use the affected functionality did not get bugs filed for them.

I didn't check all 24 bugs, but it looks like all of them were wrong, and the two packages that should have gotten bugs filed for this issue - rust-openssl itself (!) and python-cryptography - did not get bugs filed for them.

Seriously, what am I as a package maintainer supposed to do in this case? That's a phenomenal 100% false positive and 100% false negative rate. At this point it just makes me want to ignore automated security bugs that are filed against my packages, or close them as "CLOSED IDONTCARE".

Thought: Can we automatically close CVE bugs when we detect that a package is rebased? This is how most of the mingw CVEs get handled in practice.

Metadata Update from @fale:
- Issue tagged with: meeting

2 months ago

Not if you want the data to be accurate. E.g. WebKitGTK 2.50.2 just fixed most outstanding CVEs, except for one. It also fixed a few CVEs that don't have bug reports yet, but will soon. Human judgment is required.

Hmm, but that means that if we do close all the bugs, we'd more accurate than right now. If the choice is between having many false positives and a single false negative, we might as well go for the latter. Users usually don't care about specific vulnerabilities, but about the general state of having all the latest fixes. So updating to the latest upstream release and closing all extant security bug reports seems to be a good default policy. If maintainers of a package have the time and means to handle individual bugs separately, they would be welcome to do that.

Even valid CVE reports vary widely in how actionable they are. The best have a link in the CVE itself to a specific atomic commit that fixes the issue. The worst have a general description of the nature of the problem and perhaps a sample file that triggered a crash in fuzz testing, with no further analysis of the root cause and no indication that upstream has even looked at the report, let alone acted on it.

I would expect the package maintainer to attach all the CVE bug reports to the Fedora rebase update if desired. Then again, this is frequently impossible because it's very common for Fedora updates to be released before CVE bugs are created. In RHEL we go back in time and edit the previous update to add new CVEs. In Fedora, the released update is immutable, so you can't ever actually trust that the list of CVEs attached to any update is correct.

Another valid workflow would be: close the CVE bug report, perhaps with an explanation ("fixed in rebase") or ("not enough information to be actionable").

I don't think completely ignoring the bug reports is our desired outcome.

The long-term fix is adopting SBOM + VEX.

IMHO this would only fix the "problem" of not having enough bureaucratic paperwork to deal with. There are lots of reasons why a CVE may not apply to a given package, and few of then can be automagically determined by looking at SBOM info, even if we assume SBOM is fully up-to-date, complete and accurate, which it won't be.

Fedora currently has 6664 unresolved bugs with "CVE" in the title (query). I was expecting this number to be high, but not this high.

Putting aside the fact that many of the open bugs will effectively be NOTABUG, not all CVE bugs are of equal impact. Even when a CVE bug is valid and affects Fedora, this does not imply that we should aim to fix it. It is necessary to consider the severity of issues, and/or their CVSS score, to decide whether it is worth expending effort to even analyse the CVE reports, let alone invest in fixing the problem, and if so, on what timeframe.

Of those open CVEs currently we have, grouped by (bz severity / CVE severity)

  • urgent / critical => 28
  • high / important => 709
  • medium / moderate => 5435
  • low / low => 467

I'm more concerned about critical/important bugs being unfixed / ignored, than I would be about moderate/low bugs being ignored.

I wish it were possible to query a CVSS score across open CVEs, but that seems impossible in general as that info isn't provided on many of the bug reports. You have to either look at the NVD records for the CVE, or look at Red Hat's records if a fix happened to already be published in a product.

For context, IIUC, current Red Hat policy is only issue fixes for bugs with a CVSS score >= 7 (https://access.redhat.com/support/policy/updates/errata/ "Critical, Important, and Moderate CVEs with a CVSS score of 7 or higher").

Older variants of the policy did not guarantee Moderate/Low bug fixes at all (https://www.redhat.com/en/blog/do-all-vulnerabilities-really-matter "While fixing Moderate and Low severity issues are not part of the published product life cycle, they may be fixed when other non-security fixes are published.")

If Fedora had officially decided to follow that latter approach, then it would be valid to completely ignore those ~6000 bugs in Moderate/Low severity levels, or auto-close them as WONTFIX, on the assumption they'll get fixed in the background during normal course of updating packages across releases.

Proposal: if a CVE bug report is still in NEW state after 6 weeks, some automation should set needinfo?. After 8 weeks, the automation should kickstart the nonresponsive maintainer process. Package maintainers should move the bug to ASSIGNED state (indicating they acknowledge the bug report) to avoid being caught by the automation.

As a volunteer run project where no one is paying our maintainers to meet a guaranteed SLA for bug fixes (whether CVE or not), this policy proposal has a bad feeling to it. We need to be careful of not setting expectations on our volunteers that are benchmarked against expectations on commercial Linux vendors who pay their maintainers.

We ideally shouldn't ignore bugs entirely, but at the same time, just because a CVE is filed, doesn't mean it should automatically take priority over anything else a maintainer is doing. As a Fedora volunteer maintainer it is valid to ignore bugs whether they're CVEs or not, as there's only so much time people can volunteer towards Fedora.

If we demand a bug is moved to "ASSIGNED" to avoid triggering a nonresponsive maintainer process, that is unlikely to actually make people fix the bugs more quickly. Rather it is liable to push maintainers into malicious compliance where they just move the bug state and continue ignoring it. We'll end up with 6600 CVE bugs in ASSIGNED state instead of 6600 CVE bugs in NEW state, our users will be no better off in terms of security, and our maintainers will be less happy.

For context, IIUC, current Red Hat policy is only issue fixes for bugs with a CVSS score >= 7 (https://access.redhat.com/support/policy/updates/errata/ "Critical, Important, and Moderate CVEs with a CVSS score of 7 or higher").

This is a good rule of thumb.

That said, it's common for non-serious issues to receive a high CVSS score, and also not entirely uncommon for a serious issue to receive a low CVSS score. I'd say maintainer judgment is most important.

I'd really like to see this a wider discussion, the only people commenting here are people who know to look for fesco tickets.

That said, I'll note there's a bunch of history here folks could read. See https://pagure.io/fesco/issue/2956 and https://pagure.io/fesco/issue/2090 and https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org/thread/VYUGK76GLI2PSDVSCQEIEONP7YJP7NC2/

There was also discussion in https://lists.fedoraproject.org/archives/search?mlist=devel%40lists.fedoraproject.org&q=+CVE+Tracking+Bugs&page=1 from three years ago. I think we do now get less emails per each bug due to the way the trackers and Blocks are now handled.

But the barrage of low-quality security bugs filed by a group outside of Fedora that's not accountable remains. Even the team involved in creating these bugs recognize this, as all the trackers contain the following message:

Disclaimer: Community trackers are created by Red Hat Product Security team on
a best effort basis. Package maintainers are required to ascertain if the flaw
indeed affects their package, before starting the update process.

The following link provides references to all essential vulnerability
management information. If something is wrong or missing, please contact a
member of PSIRT.
https://spaces.redhat.com/display/PRODSEC/Vulnerability+Management+-+Essential+Documents+for+Engineering+Teams

and the link to report problems with these bugs is not even accessible to non-Hatters.

At this point, maybe low-medium severity security bugs should be treated like Upstream Release Monitoring bugs and be possible to opt-out of on a package-by-package basis.

As established, the Golang CVE bugs keep flooding our inboxes and until something like https://pagure.io/fesco/issue/3501 is implemented, we don't have a good way of handling them across branches. I think the best solution in this case would be to just turn them off. And the same for packages like ffmpeg or firefox or kernel that already keep up with upstream releases and don't need the additional burden of managing low-quality CVE bugs.

I also like having things on the lists instead of having to "intrude" on FESCo discussions, but right now, I don't think another long thread of complaints about the CVE bugs is going to be productive unless there's active involvement from Prodsec and additional concrete feedback from maintainers of different packaging ecosystems (which has already been provided in this ticket from Golang, Rust, Firefox, and Multimedia stack maintainers).

Another particularly bad example, fresh from earlier today: https://bugzilla.redhat.com/show_bug.cgi?id=2419093

I just closed 31 of the 37 bugs as NOTABUG because they were filed against packages where no action is needed (or possible).

And that was after going spelunking for where the actual issue was - because 1) the report does not explicitly mention which package has the affected code, 2) has no hyperlinks to the bug report or vulnerability whatsoever, and 3) uses a CVE number that has been reserved but no advisory has been published yet.

Seriously, at this point it would be better to not have any CVE bug reports at all.

Another particularly bad example, fresh from earlier today: https://bugzilla.redhat.com/show_bug.cgi?id=2419093

I just closed 31 of the 37 bugs as NOTABUG because they were filed against packages where no action is needed (or possible).

..snip..

Seriously, at this point it would be better to not have any CVE bug reports at all.

I wonder what a better solution would look like for Fedora CVE reporting, triage & tracking though ?

The Red Hat prod sec team don't have the resources to triage CVEs against every Fedora component, hence why they merely identify "possibly affected" components.

Package maintainers don't want to do triage from the huge number of BZs that prodsec is currently filing though.

If we don't have any BZ at all against Fedora, then most will never know CVEs even exist, the only fixes will be those that come in for free from rebases to new upstream releases.

If we accept the need to have some kind of reporting & tracking of CVEs, then it is inescapable that /someone / has to do triage of new CVEs at some point

Whom should be expected to do triage of CVEs, and what mechanism should Fedora use for tracking CVEs, instead of the BZs that are currently created ?

IMHO it is inescapable that package maintainers need to own the triage job as centralizing triage on to a small team cannot scale. I don't know what a better reporting & tracking approach would look like though, to make this job less awful ?

Metadata Update from @ngompa:
- Issue assigned to kevin

a month ago

This blog tells something that fedora(rawhide) already have for python

https://developers.redhat.com/articles/2025/12/15/how-reduce-false-positives-security-scans#

atleast we can start implementing VEX data for python issues, so those SBOM data can be used.

This blog tells something that fedora(rawhide) already have for python

https://developers.redhat.com/articles/2025/12/15/how-reduce-false-positives-security-scans#

"exists only in Fedora Rawhide as a technological preview"
"quick proof of concept"

This doesn't sound like something that you should already rely on.

Are we still waiting for the followups here?

I was going to try and gather some RH folks and see if we can open a dialog on this. I will try and do so this week.

Metadata Update from @decathorpe:
- Issue untagged with: meeting

9 days ago

If you need a recent example (from today):
https://bugzilla.redhat.com/show_bug.cgi?id=2429886

32 out of 36 filed bugs are bogus. There are zero hyperlinks (neither 1) the affected project, 2) a bug report, nor 3) the fix for the issue), it's only a wall of text.

AT LEAST the automatically generated hyperlink in the bug title is actually a valid page, often that's just a dead link to a nonexistent site on redhat.com ...

Now I'm going back to close 32 bugs as "NOTABUG", thanks for wasting my time, I guess :angry:

FYI, I have found the team working on this and have a meeting with them later this week.

Will report back after that and also ask them to engage here/etc.

Metadata Update from @sgallagh:
- Issue tagged with: stalled

2 days ago

Log in to comment on this ticket.

Metadata