#2333 mingw CVEs aren't getting fixed
Closed: Accepted 8 months ago by kevin. Opened 9 months ago by rharwood.

At time of writing, there are 459 open CVE bugs against mingw components, dating back at least as far as 2016: https://bugzilla.redhat.com/buglist.cgi?bug_status=__open__&classification=Fedora&f1=component&list_id=10797819&o1=substring&query_format=advanced&short_desc=CVE&short_desc_type=allwordssubstr&v1=mingw

Of those, 168 are assigned to FAS user @rjones : https://bugzilla.redhat.com/buglist.cgi?bug_status=__open__&classification=Fedora&email1=rjones%40redhat.com&emailassigned_to1=1&emailtype1=substring&list_id=10797813&query_format=advanced&short_desc=CVE&short_desc_type=allwordssubstr

On the (mistaken) theory that @rjones was inactive, I initiated the non-responsive maintainer process, and was told that they have no plans to fix or even triage these: https://bugzilla.redhat.com/show_bug.cgi?id=1794853#c1

It is my belief that if we are not going to provide security fixes for a package, it should not be in Fedora. I'm not aware of any reason mingw should be an exception here, and even if it is, the bugs should still be getting cleaned up (and communication with the security response team should occur so that they stop getting filed if they're somehow just noise).

More formally, package maintainer responsibilities indicate that "Package maintainer should handle security issues quickly" and more generally "Deal with reported bugs in a timely manner".


What are possible actions here? To me it is:

  • Ask the maintainer to handle security issues
  • Remove the affected mingw packages from the distribution
  • Make an exception for the mingw packages if we understand the context

As I understand things now, I would say the only option that makes sense is to remove the packages from the distribution. If we can get context on why CVEs are ignored and/or why that should be ok, maybe we can make another decision.

My view here is that we should not ship software with known vulnerabilities, certainly if there are fixes available. I do not know what exceptions to that would look like, but I am willing to believe there could be legitimate exceptions.

This isn't what I said at all. I happen to be co-maintainer on a bunch of these packages for historical reasons (I added many of them), but I handed over day to day maintenance of them to other people almost a decade ago. For this issue of mingw CVEs, please contact the actual maintainers, ie. kalev, sailer, fidencio, elmarco, teuf (and others I think).

Now I also know there was a plan discussed in Lyon last November to move the mingw-* packages into subpackages of the main packages. I don't know how far that's going, but it will solve the CVE duplication issue much more efficiently.

I should note the rather obvious point - these are not CVEs in Linux code and can't affect Fedora.

I handed over day to day maintenance of them to other people almost a decade ago

According to pagure, you are still the main admin on 62 mingw packages, including https://src.fedoraproject.org/rpms/mingw-openssl where the nonresponsive bug was opened.

@rjones Should I reassign them to a different account?

They should really be assigned to kalev in the first instance. I've asked around the team who actually use these packages at Red Hat if they'd like to take a look at these.

Not sure if I should reassign to @kalev or wait for the internal answer.

Some stats based on the versions involved:

  4 26
 62 27
 35 28
 42 30
 27 31
  5 el6
284 epel7

Fedora 26/27/28 are end of life. We might assume that the mingw versions of the packages in 26/27/28 have been updated since then & thus likely to have pulled in the fixes. That isn't always a valid assumption though, as historically mingw has tended to lag behind a little. Moving the mingw packaging into sub-packages of the main RPMs would be a massive win in that cases.

epel7 is clearly a disaster zone here, accounting for more than half the problems. What's worse is that EPEL7 packages don't get rebased frequently as it is a LTS stream, so that's where you really need CVE fixes backported the most. This is not a good look.

That leaves 42 CVEs on F30 and 27 on F31. This is still not a good look. One thing to bear in mind, however, is that AFAICT there's no analysis done on whether mingw is affected by any of these CVEs - the security people just automatically open CVEs for mingw packages regardless. For mingw-libvirt at least, we close pretty much every single CVE as NOTABUG because the code is usually not compiled in the mingw builds. This may or may not be the case for other packages, but something to bear in mind.

Finally here's a grouping of BZs per component

  1 mingw-flac
  1 mingw-fontconfig
  1 mingw-gettext
  1 mingw-harfbuzz
  1 mingw-hunspell
  1 mingw-lcms2
  1 mingw-libarchive
  1 mingw-librsvg2
  1 mingw-podofo
  1 mingw-qt5-qtimageformats
  1 mingw-qt5-qtsvg
  1 mingw-taglib
  2 mingw-freeimage
  2 mingw-gdb
  2 mingw-gstreamer-plugins-bad-free
  2 mingw-libffi
  2 mingw-libsoup
  2 mingw-nettle
  2 mingw-qt
  2 mingw-qt5-qtbase
  2 mingw-w64-tools
  3 mingw-bzip2
  3 mingw-gstreamer-plugins-base
  3 mingw-gstreamer-plugins-good
  4 mingw-gstreamer
  4 mingw-gstreamer1
  4 mingw-gstreamer1-plugins-base
  4 mingw-libtasn1
  4 mingw-libvorbis
  5 mingw-freetype
  5 mingw-gcc
  5 mingw-gdk-pixbuf
  6 mingw-libssh2
  9 mingw-cairo
  9 mingw-glib2
  9 mingw-gnutls
  9 mingw-postgresql
  9 mingw-SDL2_image
 10 mingw-expat
 10 mingw-libpng
 10 mingw-pcre
 10 mingw-webkitgtk3
 11 mingw-binutils
 12 mingw-libgcrypt
 13 mingw-libxslt
 13 mingw-webkitgtk
 15 mingw-icu
 18 mingw-curl
 19 mingw-libjpeg-turbo
 20 mingw-SDL2
 22 mingw-jasper
 25 mingw-libxml2
 30 mingw-openssl
 39 mingw-sqlite
 57 mingw-libtiff

And the same breakdown filtered only for CVEs against F30 & F31

  1 mingw-gstreamer
  1 mingw-gstreamer-plugins-base
  1 mingw-libarchive
  1 mingw-libgcrypt
  1 mingw-libsoup
  1 mingw-libtiff
  1 mingw-postgresql
  2 mingw-curl
  2 mingw-freeimage
  2 mingw-gcc
  2 mingw-gdb
  2 mingw-glib2
  2 mingw-gstreamer1
  2 mingw-gstreamer1-plugins-base
  2 mingw-libffi
  2 mingw-libssh2
  2 mingw-pcre
  3 mingw-expat
  3 mingw-SDL2_image
  3 mingw-webkitgtk
  4 mingw-binutils
  4 mingw-libxslt
  5 mingw-libjpeg-turbo
  6 mingw-libxml2
  7 mingw-openssl
 10 mingw-SDL2
 18 mingw-sqlite

They should really be assigned to kalev in the first instance. I've asked around the team who actually use these packages at Red Hat if they'd like to take a look at these.

I don't have the capacity to handle issues in other people's mingw packages. I don't understand why you think I should be handling all of that. Please do not reassign other people's bugs to me. Thanks.

IMHO The best course of action if @rjones is not interested in maintaining packages formally assigned to them is to orphan them all. I can do that in bulk as well.

Can you wait a little on this. We are trying to get agreement on a plan which will be submitted to improve maintenance on this packages.

Definitely do NOT orphan these packages. They are used for important work in many areas, such that orphaning them will cause major disruption. We need to determine a more sustainable path to keeping the mingw packages fully up2date with native packages.

Opps, I should clarify, I'm talking about the mingw packages in current support Fedora branches. I personally have any need / care for the EPEL branches, and honestly don't even see much benefit to having these in EPEL in the first place.

Note that orphan doesn't mean remove. What I simply worry about is that the "main admin" of those packages is clearly not interested in them yet remains listed as "main admin". That creates false assumptions (such as when the nonresponsive bugzilla was opened) and needs to be fixed. A better long term plan to handle those packages is indeed nice to have, yet the people who care should be able to determine the actual point of contact here.

I would like to remind everyone that we have a proposal to remove such vulnerable packages at: https://pagure.io/fesco/issue/1935 , but no one is willing to implement it ;(

@huzaifas Thanks. This one is on my radar, but I was busy adapting and implementing the FTBFS policy properly first :(

I had discussions with the people using these packages within Red Hat, and this was what we think could be done:

  • Drop EPEL 7 branches (unless someone is willing to maintain them). We only want these packages in Fedora. (Some of these packages are in RHEL 8, so EPEL 8 doesn't apply)

  • There is a script already existing which compares the base package versions with the mingw-* package versions. I volunteer to run this script once a month to identify outdated packages, posting the results on Fedora devel list.

  • Discuss with Fedora upstream the longer term plan of moving many mingw-* packages to subpackages (eg. mingw-glib2 becomes a subpackage of glib2). This solves the problem once and for all since you'll only need to solve CVEs in one place, and the packages will always be up to date.

We actually have a couple of candidate scripts for doing the comparisons. This is one which Daniel Berrange wrote back in the day: https://berrange.fedorapeople.org/mingw-compare/ and there is one which Sandro Mani has written more recently: https://bugzilla.redhat.com/show_bug.cgi?id=1795721#c6

To supplement @berrange's data, I did some digging into overall security bugs in the distro, which I wrote up at https://mivehind.net/2020/01/28/Fedora-has-too-many-security-bugs/

Something like @huzaifas's proposal is very attractive, but to my mind it doesn't help with EPEL because there's no release point for those - which means it won't affect half the bugs.

So if we agree in principle with the general intent to remove all the EPEL mingw packages, given lack of any interested maintainer, how should we move forward ?

If this was Fedora branch I'd say we orphan them all, and perhaps mail Fedora devel to alert other interested parties, who them have 6 weeks to step forward before the auto-retirement script purges them. I'm not clear if the auto-retirement of orphans happens in EPEL branches though ?

I'm fine with removing/orphaning/whatever all el7/ branches.

I'm fine with removing/orphaning/whatever all el7/ branches.

Me too.

I suggest coordinating any mass retirement of packages in EPEL via epel-devel mailing lists or https://pagure.io/epel/issues

Note that orphaned packages are usually kept in EPEL.

@rjones Anything new to add here? The EPEL thread didn't appear to go very far.

Looks like no one is stepping up to maintain those epel7 branches, so I will orphan all of them right now and close any bugs which are only against EPEL 7.

I sent out a final warning message to all the package owners and CC'd to epel-devel.

Looks like no one is stepping up to maintain those epel7 branches, so I will orphan all of them right now and close any bugs which are only against EPEL 7.

@rjones By orphan, you mean retire, right? (Because you can't orphan a package in only one branch)

I've written a new python script for comparing Fedora native and mingw RPMs. It reports on any differences in the version, license, url, source and patch tags in the spec files.

validate.py

Out of 244 mingw RPMs in Fedora, 26 pass this validation, 218 have some difference.

Some of the differences are really tiny, but others are quite significant. The CVE differences are clearly important, but I've also found 80 cases where the license declared for mingw is different from that declare for native. Here are full results:

check.log

Open on terminal to benefit from the ANSI coloured text.

To me this re-inforces my view that we should prioritize the merging of mingw build into the native spec, as sub-RPMs, such that we're guaranteed they'll always be in sync.

Oh, and I should point out that not all the errors reported by that script are bugs in the mingw packages. In some cases the mingw is correct and the native package is actually what has broken info. For example GConf has broken URL, where as mingw-GConf is correct.

Note that the "merge into the main specs" implies the maintainers of the main specs are willing to share the load. For example the idea that we would add mingw build to the python package terrifies me:

  • the spec is overly complicated already
  • the build time is already quite significant
  • every FTBFS is blocking a lot of unrelated things
  • the mingw Python has a large amount of patches

In the vast majority of cases I would expect the extra maintainer burden to be negligible, as the mingw packages don't carry custom non-upstream patches, and are not significantly more likely to break during build than native already are. We picked separate mingw src.rpms originally because we were indeed afraid that their would be many mingw specific maint problems needing custom patching, or frequently breaking builds. In practice that turned out to not be the case, and the maint problems we face are actually caused by the use of separate src.rpms, not solve by it.

Of course there can be exceptions to the general rule. So if python is crazy enough to justify a separate mingw RPM that's ok to stay that way. It shouldn't block merging of mingw & native packages in the common, easy, case.

That totally makes sense. I just wanted to say that maintainers of the packages should be on board before we make such decision.

mingw-* EPEL 7 packages (107) have been retired. All related bugs (307) have been
mass closed (CLOSED/EOL).

The unofficial leader of MinGW these days is @smani and I don't see him being consulted. I think the trigger was pulled too quickly on this change, but I also do not have any hard feelings against it.

FWW, I also do not have the capacity to deal with EPEL7, so I'm ok with them being retired.

Open Build Service (used by openSUSE) has multibuild feature. So you can have multiple spec files in one repo sharing common parts build different packages.

I've opened Koji RFE for something very similar: https://pagure.io/koji/issue/1905

Once we have this functionality, we can try to improve situation without having to add bloat into all those packages and having to deal with bootstraps.


That said, I'm not sure what to do now with those packages.

Metadata Update from @dcantrel:
- Issue tagged with: meeting

8 months ago

With the bulk of the security issues (but not all!) closed with the epel7 retirement, fesco would like to ask maintainers to try and solve/clear/fix the remaining cve's if possible. If there's a desire to move mingw builds into their base package builds, please write that up as a system wide change and move it through the process for the release you want to target.

Thanks!

Metadata Update from @kevin:
- Issue close_status updated to: Accepted
- Issue status updated to: Closed (was: Open)

8 months ago

This is unfortunate, as a subset of the packages was used by silent parties for cross-compiling their Windows programs on CentOS 7.x.

As a public service, I've rebuilt the subset that was used for cross-compiling the packages from source and posted an RPM repository for it in Copr:

https://copr.fedorainfracloud.org/coprs/alonid/mingw-epel7/

I hope it serves useful to whoever winds up here.

@alonid you don't have to keep them in a COPR. If you're a packager you are welcome to apply as co-maintainer and push EPEL updates yourself. I will add you to any package you want.

Absolutely what the previous comment says. This isn't a ban on mingw packages in EPEL 7. It's a realisation that we didn't have the time or people committed to maintain then in EPEL 7, so many hadn't been updated since c.2014.

I wish I could do that, but I'm not sure that I have the time resources to sort out and patch for the security issues of all of these packages in EPEL, even when limiting this task solely to the cross-compiler.

The task is usually following the native package so it would be taking the SRPM from RHEL or CentOS and shipping it for MinGW EPEL. Those upstreams take care of the CVE patching for you. We (Sandro, Richard, myself) can't commit the time to keep the EPEL branches in sync with their latest updates.

MinGW may require shipping the latest gcc to keep up with any MinGW / win32 specific issues though so you could follow the latest Rawhide packages for that.

I'm not sure that CentOS/RHEL patch enough packages for security fixes. I've done a few samples. For example, libxslt in an updated CentOS 7.7 is from a 2014 version (libxslt-1.1.28-5), which does not includes CVE fixes that came in later in the corresponding Fedora package (for example CVE-2019-11068, which RH flagged as "affected" for RHEL 7). This still looks like a lot of work, looking at each package to see if it was properly patched. Now, if the EPEL mingw-libxslt was simply synced to the Fedora version of mingw-libxslt then it would have been easier, but I'm not sure it's wanted.

I did manage however to build the newer mingw compiler toolchain for EPEL (i.e. with newer gcc and binutils) - result. It adds isl and cloog as new dependencies. If it's sufficient to only have these packages I may consider.

Login to comment on this ticket.

Metadata
Attachments 2
Attached 8 months ago View Comment
Attached 8 months ago View Comment