#1877 large number of packages FTBFS in F28
Closed 5 years ago Opened 5 years ago by zbyszek.

https://bugzilla.redhat.com/show_bug.cgi?id=1555378

Bug 1555378 depends on 874 open bugs

That's scary. I don't know what FESCo can do, but we should at least make some call for action. Would another "mini mass rebuild" where packages are just trivially rebuilt, perhaps help? If we cannot fix builds of those packages, should be retire uninstallable packages? Do we have a way to query which packages are not installable?


Metadata Update from @zbyszek:
- Issue tagged with: meeting

5 years ago

There used to be times when FTBFS packages were removed after some period of time [1]. Monitoring Ruby ecosystem, it would help to get rid of all the unmaintained packages. OTOH, there are also packages which failed rebuild just because their test suite fails randomly (and with infrastructure load during mass rebuild, there is probably higher chance to trigger such errors).

The main thing we can do is see if @till is still running the process to remove old packages or if we need to get someone else running it or figure out how to automate it. makinga call to arms to have the packages fixed is the only other thing we can do.

I ran the process of removing packages with broken deps for F21, F22 and F23 and then I think @till did it for two releases after that, but then neither of us had time I think and it stopped happening (and also IIRC something went wrong with too aggressive removing in F25, maybe? that broke one of the default deliverables and caused a backslash).

I could do it again for F28 if FESCo wants that.

I think it needs a bit of careful approach though as we haven't run it in a few releases and we're a bit short on time before the final freeze; it should be at least possible to identify packages that look like safe candidates to remove and remove those, and maybe resume the process in full for F29.

F23: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/WXEQ3Z4PWZVHKDEDRPRVTUJ6I5FIIWML/
F22: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/3KFWFS33CBGSEFYFN4AINNEDIGFT3RYJ/

I do not run the process atm. There are some issues since the scripts broke after the deprecation of pkgdb or would use outdated data. There is also currently no retiring of long-time orphaned pkgs happening, which used the same script. The problem is that there is afaik no nice way to track if just certain branches are orphaned and the required information is spread across three services.

The bug reports are missing build logs, they had them in the past. Just for this it makes sense to do a rebuilds for just these packages and update the bug reports. Btw. when cleaning up the packages that needed to drop tcp_wrappers it appeared to me that some packages where not properly rebuilt during the mass-rebuild (last commit was from releng for mass rebuild and just submitting them to koji worked and I believe I did not see a failed older build in koji).

@kalev

I ran the process of removing packages with broken deps for F21, F22 and F23

I think broken dependencies is a bit different topic, although useful as well.

@till

some packages where not properly rebuilt during the mass-rebuild (last commit was from releng for mass rebuild and just submitting them to koji worked and I believe I did not see a failed older build in koji).

I definitely met package, where the dependencies were not installed due to some infrastructure hiccup.

FESCo Meeting (2018-04-06):

Deferred to next week due to lack of quorum. Please discuss in the ticket.

All the R packages (~50) probably failed due to the broken texlive at the time, and likely will rebuild just fine. Don't know what happened to their maintainers though.

As I see it, there are several high-level points:

1) We need scripts to identify the packages that are FTBFS, and someone to run and maintain said scripts. I understand that with many recent rel-eng changes, the current scripts might need some love and attention.

2) It would be ideal to be able to tell which packages fail in and of themselves, and which ones fail due to problems with dependencies. In an ideal world, it would be nice to see "Package Y is causing 53 other packages to fail building", and focus our efforts on the packages that are required by the highest number of other packages.

3) For packages in certain languages, work with the SIGs (Python SIG, Perl SIG, Dot Net SIG, NodeJS SIG, etc.) to help with those packages.

This was discussed in FESCo meeting today:
AGREED: till will create a list of FTBFS packages, post it to devel and announce that they will be retired in 6 weeks if not being worked on (+6, 0, -0) (tyll, 15:31:57)
ACTION: till will send a list of FTBFS packages to devel and retire them in 6 weeks (tyll, 15:32:13)

Nevertheless in further discussion till noted that there's 2944 packages that didn't get rebuilt, even though we have only 817 FTBFS bugs in NEW state. till will file a releng ticket about this.

4 packages were listed erroneously, because either they were freshly added and do not have an F28 branch (rust-hyper), or they were unretired during the move to pagure (https://pagure.io/fedora-infrastructure/issue/6859).

I also closed bugs for those packages where a successful build for F28 was done and an update exists in bodhi, but the maintainer forgot to add the bug number to update description. I didn't differentiate between updates there were already pushed to stable and not, because that'd have required additional scripting and didn't seem important enough.

There's a large group of packages for which there was a successful build for F28, but no update was ever submitted. After some consideration, I think that we should just leave those packages alone and close the FTBFS-in-F28 bugs for them. My reasoning is that if we were to automatically create bodhi updates for all those packages, without maintainer oversight, we risk regressions. There were good reasons to have the mass rebuild and it certainly would be better to use the rebuilt packages, but we are almost at final freeze and it's better to avoid churn. (That applies only to any kind of automated updates. If maintainers of individual packages want to create updates, that's of course OK.)

I fired off koji builds for all packages on the list in #1555378. Some are still ongoing, but so far the tally is 79 successful, 565 failed. For those that succeeded, I think we should follow the course described in the previous paragraph and close bugs.

Finally, we have a large group of packages which still fail to build. We voted to retire them after 6 weeks, but I now think this would be premature. I would like to propose a more nuanced approach:

Retire packages if
- they FTBFS in two mass rebuilds (i.e. a few weeks after the F29 mass rebuild we would retire all packages that have .fc27-or-earlier as the lass successful build)
- they FTBFS in a mass rebuild and are are uninstallable because of missing dependencies

In both cases bugs MUST be opened, and we should wait 4 weeks, and retire packages automatically if those bugs are still in NEW state. We should to this after the mass rebuild in every release cycle. It's too late to retire packages for F28 right now. We could do this in master now, i.e. retire packages which have a last successful build with .fc26.

Till noticed that there's a large number of packages which failed to build in the F28 mass rebuild but no bug was opened for them. We have to assume that maintainers might even not be aware of this failure in those cases. We shouldn't retire them without opening bugs first.

Packages that built successfully:

$ koji list-builds --after=2018-04-12 --owner=zbyszek|grep COMPLETE
GLC_Player-2.3.0-17.fc28                                 zbyszek           COMPLETE
GLC_lib-2.2.0-18.fc28                                    zbyszek           COMPLETE
R-ALL-1.6.0-6.fc28                                       zbyszek           COMPLETE
R-Biobase-2.36.2-4.fc28                                  zbyszek           COMPLETE
R-Biostrings-2.44.0-4.fc28                               zbyszek           COMPLETE
R-BufferedMatrix-1.40.0-4.fc28                           zbyszek           COMPLETE
R-BufferedMatrixMethods-1.40.0-4.fc28                    zbyszek           COMPLETE
R-ROC-1.46.0-6.fc28                                      zbyszek           COMPLETE
R-RODBC-1.3.15-4.fc28                                    zbyszek           COMPLETE
R-RUnit-0.4.26-14.fc28                                   zbyszek           COMPLETE
R-Rsolid-0.9.31-23.fc28                                  zbyszek           COMPLETE
R-XML-3.98.1.7-4.fc28                                    zbyszek           COMPLETE
R-affyio-1.46.0-4.fc28                                   zbyszek           COMPLETE
R-bitops-1.0.6-8.fc28                                    zbyszek           COMPLETE
R-caTools-1.17.1-7.fc28                                  zbyszek           COMPLETE
R-combinat-0.0.8-14.fc28                                 zbyszek           COMPLETE
R-fibroEset-1.6.0-6.fc28                                 zbyszek           COMPLETE
R-lmtest-0.9.35-4.fc28                                   zbyszek           COMPLETE
R-mAr-1.1.2-15.fc28                                      zbyszek           COMPLETE
R-maanova-1.46.1-4.fc28                                  zbyszek           COMPLETE
R-multtest-2.32.0-4.fc28                                 zbyszek           COMPLETE
R-mvtnorm-1.0.6-4.fc28                                   zbyszek           COMPLETE
R-pls-2.4.3-8.fc28                                       zbyszek           COMPLETE
R-preprocessCore-1.38.1-4.fc28                           zbyszek           COMPLETE
R-qcc-2.2-13.fc28                                        zbyszek           COMPLETE
R-qvalue-1.38.0-6.fc28                                   zbyszek           COMPLETE
R-rlecuyer-0.3.4-4.fc28                                  zbyszek           COMPLETE
R-sciplot-1.1.0-11.fc28                                  zbyszek           COMPLETE
R-statmod-1.4.20-8.fc28                                  zbyszek           COMPLETE
R-timeDate-3010.98-8.fc28                                zbyszek           COMPLETE
R-tkWidgets-1.42.0-6.fc28                                zbyszek           COMPLETE
R-waveslim-1.7.5-7.fc28                                  zbyszek           COMPLETE
R-wavethresh-4.6.8-4.fc28                                zbyszek           COMPLETE
R-widgetTools-1.42.0-6.fc28                              zbyszek           COMPLETE
R-zoo-1.7.13-7.fc28                                      zbyszek           COMPLETE
aircrack-ng-1.2-0.16rc4.fc28                             zbyszek           COMPLETE
archaius-0.7.3-7.fc28                                    zbyszek           COMPLETE
atomix-3.22.0-5.fc28                                     zbyszek           COMPLETE
augeas-vala-0.10.0-12.fc28                               zbyszek           COMPLETE
bluecurve-gtk-themes-1.0.0-19.fc28                       zbyszek           COMPLETE
bluecurve-kde-theme-1.0.0-20.fc28                        zbyszek           COMPLETE
bmpanel2-2.1-0.23.pre1.fc28                              zbyszek           COMPLETE
bwrap-oci-0.1.2-10.fc28                                  zbyszek           COMPLETE
c++-gtk-utils-2.0.16-12.fc28                             zbyszek           COMPLETE
cairo-clock-0.3.4-21.fc28                                zbyszek           COMPLETE
connman-1.34-5.fc28                                      zbyszek           COMPLETE
ecc-25519-java-1.0.3-4.fc28                              zbyszek           COMPLETE
frepple-3.1-8.fc28                                       zbyszek           COMPLETE
gfal2-util-1.5.1-2.fc28                                  zbyszek           COMPLETE
glaxium-0.5-28.fc28                                      zbyszek           COMPLETE
golang-github-eapache-go-resiliency-0-0.7.gitb86b1ec.fc28  zbyszek           COMPLETE
golang-github-hashicorp-go-retryablehttp-0-0.4.git6e85be8.fc28  zbyszek           COMPLETE
golint-0-0.6.gitc7bacac.fc28                             zbyszek           COMPLETE
gprolog-1.4.4-12.fc28                                    zbyszek           COMPLETE
java-gnome-4.1.3-15.fc28                                 zbyszek           COMPLETE
libminc-2.3.00-11.fc28                                   zbyszek           COMPLETE
log4c-1.2.4-14.fc28                                      zbyszek           COMPLETE
ltrace-0.7.91-26.fc28                                    zbyszek           COMPLETE
php-react-event-loop-0.4.3-3.fc28                        zbyszek           COMPLETE
python-h5io-0.1-0.9.gita73bba4.fc28                      zbyszek           COMPLETE
python-locket-0.2.0-7.fc28                               zbyszek           COMPLETE
python-netdiff-0.4.7-8.fc28                              zbyszek           COMPLETE
python-optcomplete-1.2-0.15.20130428hg9583af7.fc28       zbyszek           COMPLETE
python-pycha-0.7.0-6.fc28                                zbyszek           COMPLETE
python-pywt-0.5.2-6.fc28                                 zbyszek           COMPLETE
python-tenacity-4.8.0-1.fc28                             zbyszek           COMPLETE
python-wheel-0.30.0a0-9.fc28                             zbyszek           COMPLETE
qmforge-2.3.2-8.fc28                                     zbyszek           COMPLETE
reactive-streams-1.0.0-4.fc28                            zbyszek           COMPLETE
rsyntaxtextarea-2.6.1-3.fc28                             zbyszek           COMPLETE
rubygem-awesome_spawn-1.3.0-6.fc28                       zbyszek           COMPLETE
rubygem-websocket-1.2.2-8.fc28                           zbyszek           COMPLETE
shibboleth-java-support-7.1.1-6.fc28                     zbyszek           COMPLETE
spock-0.7-0.18.groovy.2.0.fc28                           zbyszek           COMPLETE
sshj-0.13.0-7.fc28                                       zbyszek           COMPLETE
ssm-1.4-8.fc28                                           zbyszek           COMPLETE
tcpreen-1.4.4-18.fc28                                    zbyszek           COMPLETE
tempus-fugit-1.2-0.4.SNAPSHOT.fc28                       zbyszek           COMPLETE
zkclient-0.10-5.fc28                                     zbyszek           COMPLETE

Failed:

$ koji list-builds --after=2018-04-12 --owner=zbyszek|grep FAILED
389-ds-1.2.2-11.fc28                                     zbyszek           FAILED
InsightToolkit-4.9.1-8.fc28                              zbyszek           FAILED
Java-WebSocket-1.3.1-0.5.git58d1778.fc28                 zbyszek           FAILED
OpenTK-1.1-11.4c.fc28                                    zbyszek           FAILED
Pound-2.7-7.fc28                                         zbyszek           FAILED
PyMca-4.7.3-12.fc28                                      zbyszek           FAILED
PyX-0.14.1-7.fc28                                        zbyszek           FAILED
R-AnnotationDbi-1.30.1-5.fc28                            zbyszek           FAILED
R-BSgenome-1.38.0-4.fc28                                 zbyszek           FAILED
R-BSgenome.Celegans.UCSC.ce2-1.4.0-6.fc28                zbyszek           FAILED
R-DynDoc-1.52.0-4.fc28                                   zbyszek           FAILED
R-GeneR-2.24.0-17.fc28                                   zbyszek           FAILED
R-GenomicFeatures-1.20.1-5.fc28                          zbyszek           FAILED
R-RM2-0.0-19.fc28                                        zbyszek           FAILED
R-TH-data-1.0.6-5.fc28                                   zbyszek           FAILED
R-affy-1.54.0-4.fc28                                     zbyszek           FAILED
R-affydata-1.16.0-5.fc28                                 zbyszek           FAILED
R-car-2.0.25-5.fc28                                      zbyszek           FAILED
R-hgu133acdf-2.16.0-5.fc28                               zbyszek           FAILED
R-hgu95av2cdf-2.16.0-5.fc28                              zbyszek           FAILED
R-hgu95av2probe-2.16.0-5.fc28                            zbyszek           FAILED
R-msm-1.6.4-4.fc28                                       zbyszek           FAILED
R-multcomp-1.4.5-5.fc28                                  zbyszek           FAILED
R-sandwich-2.3.0-7.fc28                                  zbyszek           FAILED
R-systemfit-1.1.15-11.fc28                               zbyszek           FAILED
Ray-2.3.1-18.fc28                                        zbyszek           FAILED
SocketW-031026-13.fc28                                   zbyszek           FAILED
activeio-3.1.4-17.fc28                                   zbyszek           FAILED
activemq-5.6.0-20.fc28                                   zbyszek           FAILED
activemq-cpp-3.8.4-8.fc28                                zbyszek           FAILED
activemq-protobuf-1.1-16.fc28                            zbyszek           FAILED
adplay-1.7-5.fc28                                        zbyszek           FAILED
ambari-1.5.1-14.fc28                                     zbyszek           FAILED
apache-commons-primitives-1.1-0.6.678495.fc28            zbyszek           FAILED
apigen-4.1.2-4.fc28.2                                    zbyszek           FAILED
appmenu-qt5-0.3.0+16.10.20160628.1-6.fc28                zbyszek           FAILED
apvlv-0.1.4-15.fc28                                      zbyszek           FAILED
ara-0.14.0-2.fc28                                        zbyszek           FAILED
arm-none-eabi-gcc-cs-7.1.0-6.fc28                        zbyszek           FAILED
arm-none-eabi-gdb-7.6.2-9.fc28                           zbyszek           FAILED
atlascpp-0.6.4-3.fc28                                    zbyszek           FAILED
avr-gdb-7.1-17.fc28                                      zbyszek           FAILED
ayttm-0.6.3-19.fc28                                      zbyszek           FAILED
belle-sip-1.4.2-6.fc28                                   zbyszek           FAILED
biber-2.6-7.fc28                                         zbyszek           FAILED
bibtex2html-1.97-15.fc28                                 zbyszek           FAILED
blitz-0.10-14.fc28                                       zbyszek           FAILED
blobwars-1.19-18.fc28                                    zbyszek           FAILED
boost-gdb-printers-6-3.fc28                              zbyszek           FAILED
booth-1.0-2.570876d.git.fc28.5                           zbyszek           FAILED
btanks-0.9.8083-17.fc28                                  zbyszek           FAILED
btest-0.54-10.fc28                                       zbyszek           FAILED
burp-1.4.40-8.fc28                                       zbyszek           FAILED
busybox-1.26.2-4.fc28                                    zbyszek           FAILED
cdrdao-1.2.3-32.fc28                                     zbyszek           FAILED
ceres-solver-1.13.0-3.fc28                               zbyszek           FAILED
chck-0-5.20161208git0036426.fc28                         zbyszek           FAILED
choqok-1.6.0-5.fc28                                      zbyszek           FAILED
clamsmtp-1.10-23.fc28                                    zbyszek           FAILED
claws-mail-plugins-3.14.0-5.fc28                         zbyszek           FAILED
coin-or-FlopC++-1.2.4-6.fc28                             zbyszek           FAILED
compat-SFML16-1.6-16.fc28                                zbyszek           FAILED
compat-gcc-296-2.96-147.4                                zbyszek           FAILED
cookcc-0.3.3-17.fc28                                     zbyszek           FAILED
cross-gcc-7.2.1-2.fc28.1                                 zbyszek           FAILED
cube-4.3.5-4.fc28                                        zbyszek           FAILED
cwiid-0.6.00-32.20100505gitfadf11e.fc28                  zbyszek           FAILED
dnssec-check-2.1-11.fc28                                 zbyszek           FAILED
dnssec-nodes-2.1-10.fc28                                 zbyszek           FAILED
dopewars-1.5.12-23.1033svn.fc28                          zbyszek           FAILED
dtdparser-1.21-20.fc28                                   zbyszek           FAILED
dumbster-1.6-23.fc28                                     zbyszek           FAILED
dvb-apps-1.1.2-15.1505.3d43b280298c.fc28                 zbyszek           FAILED
e2tools-0.0.16-30.fc28                                   zbyszek           FAILED
eclipse-m2e-antlr-0.15.0-0.11.gitb2d587b.fc28            zbyszek           FAILED
eclipse-m2e-cxf-0.14.0-0.10.gitf22f9ea.fc28              zbyszek           FAILED
eclipse-m2e-egit-0.15.0-0.3.gitdef7cfb.fc28              zbyszek           FAILED
eclipse-m2e-mavendev-0.3.0.201506181201-6.fc28           zbyszek           FAILED
eclipse-m2e-modello-0.16.0-0.11.git5d1062a.fc28          zbyszek           FAILED
eclipse-m2e-plexus-0.15.0-0.11.git8baf151.fc28           zbyszek           FAILED
eclipse-m2e-sisu-0.15.0-0.11.git81d3bc5.fc28             zbyszek           FAILED
eclipse-m2e-sourcelookup-1.1.0.201506181114-6.fc28       zbyszek           FAILED
ehcache2-2.10.2.2.21-4.fc28                              zbyszek           FAILED
elemental-0.87.7-8.fc28                                  zbyszek           FAILED
elfelli-0.3.1-21.fc28                                    zbyszek           FAILED
emacs-evil-1.2.9-4.fc28                                  zbyszek           FAILED
emacs-pymacs-0.25-11.fc28                                zbyszek           FAILED
ergo-3.6-1.fc28                                          zbyszek           FAILED
esc-1.1.1-3.fc28                                         zbyszek           FAILED
escape-200912250-18.fc28                                 zbyszek           FAILED
eucalyptus-3.3.0-0.12.20130408git32052445.fc28           zbyszek           FAILED
eurephia-1.1.0-16.fc28                                   zbyszek           FAILED
fedora-screensaver-theme-1.0.0-16.fc28                   zbyszek           FAILED
fedorainfinity-screensaver-theme-1.0.0-14.fc28           zbyszek           FAILED
feedreader-2.0.2-5.fc28                                  zbyszek           FAILED
felix-fileinstall-3.5.0-5.fc28                           zbyszek           FAILED
fes-0.1-14.fc28                                          zbyszek           FAILED
firmware-addon-dell-2.2.9-16.fc28                        zbyszek           FAILED
font-manager-0.7.2-8.fc28                                zbyszek           FAILED
fontmatrix-0.9.99-35.r1218.fc28                          zbyszek           FAILED
forbidden-apis-1.7-6.fc28                                zbyszek           FAILED
free42-1.4.77-6.fc28                                     zbyszek           FAILED
frogr-1.3-6.fc28                                         zbyszek           FAILED
ftnchek-3.3.1-26.fc28                                    zbyszek           FAILED
generic-jms-ra-1.0.7-4.fc28                              zbyszek           FAILED
gettext-commons-0.9.6-17.fc28                            zbyszek           FAILED
ghasher-1.2.1-24.fc28                                    zbyszek           FAILED
glmark2-2014.03-10.fc28                                  zbyszek           FAILED
gnome-colors-icon-theme-5.5.1-12.fc28                    zbyszek           FAILED
gnome-sharp-2.24.2-17.fc28                               zbyszek           FAILED
gnue-common-0.6.9-22.fc28                                zbyszek           FAILED
gogoc-1.2-53.fc28                                        zbyszek           FAILED
gogui-1.4.9-10.fc28                                      zbyszek           FAILED
golang-github-Shopify-sarama-1.7.0-0.6.git87ec8d7.fc28   zbyszek           FAILED
golang-github-bmizerany-perks-0-0.11.gitd9a9656.fc28     zbyszek           FAILED
golang-github-coreos-go-oidc-0-0.8.gitee7cb1f.fc28       zbyszek           FAILED
golang-github-digitalocean-godo-0-0.10.git8dc1f54.fc28   zbyszek           FAILED
golang-github-docker-go-1.5.1-0.7.gitd30aec9.fc28        zbyszek           FAILED
golang-github-docker-go-connections-0.1.2-0.7.git6e4c13d.fc28  zbyszek           FAILED
golang-github-docker-spdystream-0-0.15.gitb2c3287.fc28   zbyszek           FAILED
golang-github-endophage-gotuf-0-0.8.git9bcdad0.fc28      zbyszek           FAILED
golang-github-fatih-pool-0-0.8.gitcba550e.fc28           zbyszek           FAILED
golang-github-gengo-grpc-gateway-0-0.5.gitdcb8443.fc28   zbyszek           FAILED
golang-github-ghodss-yaml-0-0.18.git0ca9ea5.fc28         zbyszek           FAILED
golang-github-go-sql-driver-mysql-1.2-8.fc28             zbyszek           FAILED
golang-github-go-tomb-tomb-0-0.12.gitd5d1b58.fc28        zbyszek           FAILED
golang-github-gonum-blas-0-0.9.git80dca99.fc28           zbyszek           FAILED
golang-github-gonum-graph-0.9-9.fc28                     zbyszek           FAILED
golang-github-hashicorp-go-immutable-radix-0-0.8.gitaca1bd0.fc28  zbyszek           FAILED
golang-github-hashicorp-hcl-0-0.14.gitef8133d.fc28       zbyszek           FAILED
golang-github-hashicorp-raft-mdb-0-0.10.git4ec3694.fc28  zbyszek           FAILED
golang-github-inconshreveable-muxado-0-0.10.gitf693c7e.fc28  zbyszek           FAILED
golang-github-influxdb-gomdb-0-0.8.git29fe330.fc28       zbyszek           FAILED
golang-github-jfrazelle-go-0-0.8.git6e461eb.fc28         zbyszek           FAILED
golang-github-mesos-mesos-go-0-0.10.git83b52d7.fc28      zbyszek           FAILED
golang-github-mitchellh-cli-0-0.4.git8102d0e.fc28        zbyszek           FAILED
golang-github-pkg-sftp-0-0.6.git4d0e916.fc28             zbyszek           FAILED
golang-github-racker-perigee-0-0.10.git0c00cb0.fc28      zbyszek           FAILED
golang-github-skynetservices-skydns-2.5.3-0.8.a.git8688008.fc28  zbyszek           FAILED
golang-github-smartystreets-go-aws-auth-0-0.9.git1f0db8c.fc28  zbyszek           FAILED
golang-github-tchap-go-patricia-1.0.1-12.fc28            zbyszek           FAILED
golang-github-vbatts-tar-split-0.9.13-7.fc28             zbyszek           FAILED
golang-github-xeipuuv-gojsonreference-0-0.5.gite02fc20.fc28  zbyszek           FAILED
golang-googlecode-go-decimal-inf-0.9.0-0.5.git3887ee9.fc28  zbyszek           FAILED
golang-googlecode-log4go-0-0.10.hgc3294304d93f.fc28      zbyszek           FAILED
google-oauth-java-client-1.22.0-4.fc28                   zbyszek           FAILED
gplugin-0.27.0-8.fc28                                    zbyszek           FAILED
grads-2.0.2-25.fc28                                      zbyszek           FAILED
grhino-0.16.1-5.fc28                                     zbyszek           FAILED
gsf-sharp-0.8.1-31.fc28                                  zbyszek           FAILED
gsignond-1.0.6-5.fc28                                    zbyszek           FAILED
gsl-2.4-4.fc28                                           zbyszek           FAILED
gspiceui-0.9.98-21.fc28                                  zbyszek           FAILED
gst-entrans-1.2.1-4.fc28                                 zbyszek           FAILED
gstreamer-0.10.36-20.fc28                                zbyszek           FAILED
gstreamer-plugins-base-0.10.36-20.fc28                   zbyszek           FAILED
gstreamer-plugins-good-0.10.31-21.fc28                   zbyszek           FAILED
gtk+extra-2.1.2-22.fc28                                  zbyszek           FAILED
gtkmathview-0.8.0-24.fc28                                zbyszek           FAILED
guilt-0.36-6.fc28                                        zbyszek           FAILED
gyachi-1.2.11-18.fc28                                    zbyszek           FAILED
hackedbox-0.8.5-21.fc28                                  zbyszek           FAILED
hail-0.8-0.22.gf9c5b967.fc28                             zbyszek           FAILED
hgsubversion-1.8.7-3.fc28                                zbyszek           FAILED
hibernate-hql-1.3.0-0.4.Alpha2.fc28                      zbyszek           FAILED
hibernate-search-5.5.4-4.fc28                            zbyszek           FAILED
hibernate3-3.6.10-23.fc28                                zbyszek           FAILED
hid4java-0.4.0-6.fc28                                    zbyszek           FAILED
hive-0.12.0-11.fc28                                      zbyszek           FAILED
horst-4.2-6.fc28                                         zbyszek           FAILED
hystrix-1.4.21-8.fc28                                    zbyszek           FAILED
ibus-unikey-0.6.1-14.fc28                                zbyszek           FAILED
idlj-maven-plugin-1.2.1-10.fc28                          zbyszek           FAILED
igraph-0.7.1-7.fc28                                      zbyszek           FAILED
ikarus-0.0.3-18.fc28                                     zbyszek           FAILED
ike-2.2.1-12.fc28                                        zbyszek           FAILED
iksemel-1.5-0.6.git978b733.fc28                          zbyszek           FAILED
ircd-ratbox-2.2.9-10.fc28                                zbyszek           FAILED
ircp-tray-0.7.6-14.fc28                                  zbyszek           FAILED
jackson-dataformat-cbor-2.7.6-4.fc28                     zbyszek           FAILED
jackson-dataformat-csv-2.7.6-4.fc28                      zbyszek           FAILED
jackson-dataformat-smile-2.7.6-4.fc28                    zbyszek           FAILED
jackson-dataformat-yaml-2.7.6-4.fc28                     zbyszek           FAILED
jackson-datatype-jdk8-2.7.6-5.fc28                       zbyszek           FAILED
jackson-datatype-jsr310-2.7.6-5.fc28                     zbyszek           FAILED
jacorb-2.3.2-4.jbossorg.5.fc28                           zbyszek           FAILED
jarbundler-2.2.0-14.fc28                                 zbyszek           FAILED
jasperreports-6.2.2-5.fc28                               zbyszek           FAILED
jboss-dmr-1.3.0-4.fc28                                   zbyszek           FAILED
jboss-jaxb-intros-1.0.2-13.fc28                          zbyszek           FAILED
jboss-web-8.0.0-0.9.Alpha1.fc28                          zbyszek           FAILED
jboss-web-native-2.0.10-13.fc28                          zbyszek           FAILED
jclouds-1.9.2-5.fc28                                     zbyszek           FAILED
jemmy-2.3.0.0-16.fc28                                    zbyszek           FAILED
jenkins-ant-plugin-1.2-9.fc28                            zbyszek           FAILED
jenkins-antisamy-markup-formatter-plugin-1.3-5.fc28      zbyszek           FAILED
jenkins-commons-jelly-1.1.20120928-13.fc28               zbyszek           FAILED
jenkins-credentials-plugin-1.27-4.fc28                   zbyszek           FAILED
jenkins-crypto-util-1.4-9.fc28                           zbyszek           FAILED
jenkins-external-monitor-job-plugin-1.4-7.fc28           zbyszek           FAILED
jenkins-extras-memory-monitor-1.9-6.fc28                 zbyszek           FAILED
jenkins-icon-shim-1.0.4-7.fc28                           zbyszek           FAILED
jenkins-instance-identity-1.4-8.fc28                     zbyszek           FAILED
jenkins-javadoc-plugin-1.3-7.fc28                        zbyszek           FAILED
jenkins-jexl-1.1-8.20111212.fc28                         zbyszek           FAILED
jenkins-junit-plugin-1.12-4.fc28                         zbyszek           FAILED
jenkins-ldap-plugin-1.11-6.fc28                          zbyszek           FAILED
jenkins-mailer-plugin-1.17-4.fc28                        zbyszek           FAILED
jenkins-matrix-auth-plugin-1.2-6.fc28                    zbyszek           FAILED
jenkins-matrix-project-plugin-1.6-5.fc28                 zbyszek           FAILED
jenkins-pam-auth-plugin-1.2-6.fc28                       zbyszek           FAILED
jenkins-remoting-2.62.3-4.fc28                           zbyszek           FAILED
jenkins-script-security-plugin-1.18.1-4.fc28             zbyszek           FAILED
jenkins-ssh-cli-auth-1.2-11.fc28                         zbyszek           FAILED
jenkins-ssh-credentials-plugin-1.11-7.fc28               zbyszek           FAILED
jenkins-ssh-slaves-plugin-1.10-6.fc28                    zbyszek           FAILED
jenkins-sshd-1.6-10.fc28                                 zbyszek           FAILED
jeromq-0.3.6-4.fc28                                      zbyszek           FAILED
jeuclid-3.1.3-26.fc28                                    zbyszek           FAILED
jrosetta-1.0.4-16.fc28                                   zbyszek           FAILED
js-yui2-2.9.0-13.fc28                                    zbyszek           FAILED
jsilver-1.0.0-14.fc28                                    zbyszek           FAILED
json4s-3.2.7-8.fc28                                      zbyszek           FAILED
jtds-1.3.1-9.fc28                                        zbyszek           FAILED
jutils-1.0.1-16.20110719svn.fc28                         zbyszek           FAILED
kawa-2.0-9.fc28                                          zbyszek           FAILED
kcemirror-0.1.5-21.fc28                                  zbyszek           FAILED
kde-plasma-activitymanager-0.5-14.fc28                   zbyszek           FAILED
kdesrc-build-1.15.1-15.fc28                              zbyszek           FAILED
kio-gdrive-1.2.1-3.fc28                                  zbyszek           FAILED
kmplayer-0.11.3c-16.fc28                                 zbyszek           FAILED
kpilot-5.3.0-26.fc28                                     zbyszek           FAILED
labrea-2.5.1-18.fc28                                     zbyszek           FAILED
lastfmlib-0.4.0-17.fc28                                  zbyszek           FAILED
latrace-0.5.11-14.fc28                                   zbyszek           FAILED
leafpad-0.8.18.1-19.fc28                                 zbyszek           FAILED
lekhonee-gnome-0.12-13.fc28                              zbyszek           FAILED
leveldbjni-1.8-18.fc28                                   zbyszek           FAILED
libXevie-1.0.3-15.fc28                                   zbyszek           FAILED
libappindicator-12.10.0-19.fc28                          zbyszek           FAILED
libclc-0.2.0-10.git1cb3fbf.fc28                          zbyszek           FAILED
libdap-3.18.3-5.fc28                                     zbyszek           FAILED
libexplain-1.4-9.fc28                                    zbyszek           FAILED
libfaketime-0.9.6-7.fc28                                 zbyszek           FAILED
libgda-5.2.2-16.fc28                                     zbyszek           FAILED
libgringotts-1.2.1-23.fc28                               zbyszek           FAILED
libhfi1-0.5-27.fc28                                      zbyszek           FAILED
libhttpserver-0.9.0-8.fc28                               zbyszek           FAILED
libkdtree++-0.7.0-18.fc28                                zbyszek           FAILED
libkolab-1.0.2-12.fc28                                   zbyszek           FAILED
libmimedir-0.4-19.fc28                                   zbyszek           FAILED
libnasl-2.2.11-22.fc28                                   zbyszek           FAILED
libnetdude-0.11-17.fc28                                  zbyszek           FAILED
libocrdma-1.0.8-7.fc28                                   zbyszek           FAILED
libodb-mysql-2.4.0-9.fc28                                zbyszek           FAILED
libomxil-bellagio-0.9.3-16.fc28                          zbyszek           FAILED
libopensync-plugin-palm-0.22-16.fc28                     zbyszek           FAILED
liboping-1.10.0-4.fc28                                   zbyszek           FAILED
libpwiz-3.0.11133-4.fc28                                 zbyszek           FAILED
libqtelegram-ae-10.0.0-6.fc28                            zbyszek           FAILED
libraqm-0.1.1-5.fc28                                     zbyszek           FAILED
libratbag-0.9-5.fc28                                     zbyszek           FAILED
libsvm-3.21-7.fc28                                       zbyszek           FAILED
libtaginfo-0.2.0-10.fc28                                 zbyszek           FAILED
llvm34-3.4.2-14.fc28                                     zbyszek           FAILED
llvm35-3.5.2-8.fc28                                      zbyszek           FAILED
loki-lib-0.1.7-18.fc28                                   zbyszek           FAILED
lookup-2.1-10.fc28                                       zbyszek           FAILED
lostirc-0.4.6-26.fc28                                    zbyszek           FAILED
lsyncd-2.2.2-4.fc28                                      zbyszek           FAILED
manaplus-1.3.10.27.2-14.fc28                             zbyszek           FAILED
mdadm-4.0-6.fc28                                         zbyszek           FAILED
monotone-viz-1.0.2-21.fc28                               zbyszek           FAILED
ncid-1.6-6.fc28                                          zbyszek           FAILED
nodejs-dependency-lister-1.0.3-9.fc28                    zbyszek           FAILED
nodejs-grunt-contrib-nodeunit-0.4.1-6.fc28               zbyszek           FAILED
nodejs-grunt-html-validation-0.1.18-10.fc28              zbyszek           FAILED
nodejs-monocle-1.1.51-12.fc28                            zbyszek           FAILED
nodejs-npm-registry-client-7.1.0-4.fc28                  zbyszek           FAILED
openprops-0.7.1-4.fc28                                   zbyszek           FAILED
openpts-0.2.6-17.fc28                                    zbyszek           FAILED
openrdf-sesame-2.8.10-4.fc28                             zbyszek           FAILED
opensaml-java-xmltooling-1.3.4-15.fc28                   zbyszek           FAILED
openvas-scanner-5.1.1-5.fc28                             zbyszek           FAILED
openvibe-1.1.0-8.fc28                                    zbyszek           FAILED
openwebbeans-1.2.8-5.fc28                                zbyszek           FAILED
orocos-bfl-0.8.99-12.20160503gitc1b18e3.fc28             zbyszek           FAILED
orpie-1.5.2-9.fc28                                       zbyszek           FAILED
ortp-0.23.0-7.fc28                                       zbyszek           FAILED
oscap-anaconda-addon-0.7-7.fc28                          zbyszek           FAILED
osslsigncode-1.7.1-6.fc28                                zbyszek           FAILED
ovirt-engine-sdk-java-3.6.10.0-4.fc28                    zbyszek           FAILED
owasp-java-html-sanitizer-20160422.1-4.fc28              zbyszek           FAILED
papaki-1.0.0-0.12.Beta3.fc28                             zbyszek           FAILED
patricia-trie-0.2-14.20091116svn.fc28                    zbyszek           FAILED
pdf-stapler-0.3.3-11.fc28                                zbyszek           FAILED
peervpn-0.044-5.fc28                                     zbyszek           FAILED
perdition-2.1-11.fc28                                    zbyszek           FAILED
perl-OpenOffice-UNO-0.07-25.fc28                         zbyszek           FAILED
pesign-test-app-5-8.fc28                                 zbyszek           FAILED
pgadmin3-1.22.2-5.fc28                                   zbyszek           FAILED
pgmodeler-0.9.0-0.6.alpha1.fc28                          zbyszek           FAILED
php-Slim-2.6.2-7.fc28                                    zbyszek           FAILED
php-nette-database-2.4.4-1.fc28.1                        zbyszek           FAILED
php-password-compat-1.0.4-6.fc28                         zbyszek           FAILED
php-pear-Auth_HTTP-2.1.8-7.fc28                          zbyszek           FAILED
php-pear-Text-Diff-1.2.2-3.fc28                          zbyszek           FAILED
pikdev-0.9.2-26.fc28                                     zbyszek           FAILED
piklab-0.16.1-19.fc28                                    zbyszek           FAILED
planets-0.1.13-22.fc28                                   zbyszek           FAILED
plasma-mediacenter-5.7.5-6.fc28                          zbyszek           FAILED
port-allocator-maven-plugin-1.2-13.fc28                  zbyszek           FAILED
primer3-2.3.6-10.fc28                                    zbyszek           FAILED
prover9-200911a-15.fc28                                  zbyszek           FAILED
prozilla-2.0.4-24.fc28                                   zbyszek           FAILED
ptlib-2.10.10-21.fc28                                    zbyszek           FAILED
purple-telegram-1.3.1-6.fc28                             zbyszek           FAILED
pyee-5.0.0-2.fc28                                        zbyszek           FAILED
pyqt-mail-checker-2.1.15-7.fc28                          zbyszek           FAILED
pyrasite-2.0-13.fc28                                     zbyszek           FAILED
pyrrd-0.1.0-14.fc28                                      zbyszek           FAILED
python-MDAnalysis-0.16.2-4.fc28                          zbyszek           FAILED
python-bashate-0.5.1-6.fc28                              zbyszek           FAILED
python-binaryornot-0.4.0-7.fc28                          zbyszek           FAILED
python-catkin_tools-0.4.4-3.fc28                         zbyszek           FAILED
python-envisage-4.5.0-6.fc28                             zbyszek           FAILED
python-flask-restless-0.17.0-10.fc28                     zbyszek           FAILED
python-flask-silk-0.1.2-11.fc28                          zbyszek           FAILED
python-flower-0.9.2-1.fc28                               zbyszek           FAILED
python-gabbi-1.33.0-3.fc28                               zbyszek           FAILED
python-hidapi-0.7.99.post20-7.fc28                       zbyszek           FAILED
python-jabberbot-0.16-6.fc28                             zbyszek           FAILED
python-joblib-0.11-3.fc28                                zbyszek           FAILED
python-keystonemiddleware-4.14.0-3.fc28                  zbyszek           FAILED
python-libasyncns-0.7.1-19.fc28                          zbyszek           FAILED
python-magnumclient-2.3.0-5.fc28                         zbyszek           FAILED
python-nibabel-2.2.0-2.fc28                              zbyszek           FAILED
python-nipy-0.4.1-4.fc28                                 zbyszek           FAILED
python-nose2-0.6.5-6.fc28                                zbyszek           FAILED
python-oslo-db-4.17.0-3.fc28                             zbyszek           FAILED
python-oslo-messaging-5.17.1-4.fc28                      zbyszek           FAILED
python-oslo-policy-1.14.0-5.fc28                         zbyszek           FAILED
python-oslo-vmware-2.14.0-5.fc28                         zbyszek           FAILED
python-osrf-pycommon-0.1.2-6.fc28                        zbyszek           FAILED
python-pacpy-1.0.2-6.fc28                                zbyszek           FAILED
python-pbkdf2-1.3-9.fc28                                 zbyszek           FAILED
python-pecan-notario-0.0.3-9.fc28                        zbyszek           FAILED
python-portalocker-1.0.1-4.fc28                          zbyszek           FAILED
python-py2pack-0.6.3-6.fc28                              zbyszek           FAILED
python-pyblock-0.53-16.fc28                              zbyszek           FAILED
python-pylons-1.0.1-10.fc28                              zbyszek           FAILED
python-pyrad-2.0-12.git1ca2da3.fc28                      zbyszek           FAILED
python-pyvmomi-6.5-6.fc28                                zbyszek           FAILED
python-rhev-1.0-15.rc1.fc28.12                           zbyszek           FAILED
python-ripe-atlas-cousteau-1.3-5.fc28                    zbyszek           FAILED
python-ryu-4.15-2.fc28                                   zbyszek           FAILED
python-s3transfer-0.1.10-4.fc28                          zbyszek           FAILED
python-semantic_version-2.6.0-4.fc28                     zbyszek           FAILED
python-shade-1.18.1-3.fc28                               zbyszek           FAILED
python-sqlacodegen-1.1.6-7.fc28                          zbyszek           FAILED
python-statsmodels-0.8.0-4.fc28                          zbyszek           FAILED
python-thriftpy-0.3.9-6.fc28                             zbyszek           FAILED
python-vcstools-0.1.39-5.fc28                            zbyszek           FAILED
python-yamllint-1.6.0-6.fc28                             zbyszek           FAILED
python26-psutil-0.6.1-11.fc28                            zbyszek           FAILED
python3-openid-3.0.10-4.fc28                             zbyszek           FAILED
python3-pyudev-0.20.0-5.fc28                             zbyszek           FAILED
pywebkitgtk-1.1.8-16.fc28                                zbyszek           FAILED
pyxmlsec-0.3.1-10.fc28                                   zbyszek           FAILED
qbrew-0.4.1-21.fc28                                      zbyszek           FAILED
qca-pkcs11-2.0.0-0.14.fc28.beta2                         zbyszek           FAILED
qdevelop-0.29-10.fc28                                    zbyszek           FAILED
qpid-java-6.0.4-8.fc28                                   zbyszek           FAILED
qtcurve-1.8.18-11.g330bfa5.fc28                          zbyszek           FAILED
qtoctave-0.10.1-25.fc28                                  zbyszek           FAILED
qtscriptgenerator-0.2.0-15.fc28                          zbyszek           FAILED
quake3-1.36-31.svn2102.fc28                              zbyszek           FAILED
querydsl-4.0.4-8.fc28                                    zbyszek           FAILED
qutim-0.4-0.12.git.468bede.fc28                          zbyszek           FAILED
racoon2-20100526a-36.fc28                                zbyszek           FAILED
rapidsvn-0.12.1-13.fc28                                  zbyszek           FAILED
ratbox-services-1.2.1-19.fc28                            zbyszek           FAILED
rdesktop-1.8.3-7.fc28                                    zbyszek           FAILED
rdkit-2017.09.1-2.fc28                                   zbyszek           FAILED
re2c-0.16-5.fc28                                         zbyszek           FAILED
reactor-2.0.8-5.fc28                                     zbyszek           FAILED
reflections-0.9.10-5.fc28                                zbyszek           FAILED
reg-0.4.1-8.fc28                                         zbyszek           FAILED
rescu-1.8.2-0.5.gitbeb9897.fc28                          zbyszek           FAILED
resiprocate-1.10.2-30.fc28                               zbyszek           FAILED
rogue-5.4.5-24.fc28                                      zbyszek           FAILED
rospack-2.0.14-19.fc28                                   zbyszek           FAILED
rubygem-activerecord-nulldb-adapter-0.3.1-8.fc28         zbyszek           FAILED
rubygem-arrayfields-4.7.4-15.fc28                        zbyszek           FAILED
rubygem-asciidoctor-pdf-1.5.0-0.5.alpha.13.fc28          zbyszek           FAILED
rubygem-attributes-5.0.1-19.fc28                         zbyszek           FAILED
rubygem-authlogic-3.4.2-6.fc28                           zbyszek           FAILED
rubygem-axiom-types-0.1.1-8.fc28                         zbyszek           FAILED
rubygem-bogus-0.1.6-6.fc28                               zbyszek           FAILED
rubygem-bourne-1.6.0-6.fc28                              zbyszek           FAILED
rubygem-bunny-0.7.9-11.fc28                              zbyszek           FAILED
rubygem-chunky_png-1.2.7-8.fc28                          zbyszek           FAILED
rubygem-climate_control-0.0.3-10.fc28                    zbyszek           FAILED
rubygem-closure-compiler-1.1.12-4.fc28                   zbyszek           FAILED
rubygem-cocaine-0.5.7-6.fc28                             zbyszek           FAILED
rubygem-codemirror-rails-5.16.0-4.fc28                   zbyszek           FAILED
rubygem-coercible-1.0.0-8.fc28                           zbyszek           FAILED
rubygem-compass-1.0.1-7.fc28                             zbyszek           FAILED
rubygem-cookiejar-0.3.2-10.fc28                          zbyszek           FAILED
rubygem-dependor-1.0.1-8.fc28                            zbyszek           FAILED
rubygem-descendants_tracker-0.0.4-7.fc28                 zbyszek           FAILED
rubygem-dotenv-0.8.0-8.fc28                              zbyszek           FAILED
rubygem-dynect_rest-0.4.3-13.fc28                        zbyszek           FAILED
rubygem-equalizer-0.0.11-6.fc28                          zbyszek           FAILED
rubygem-escape_utils-1.1.0-10.fc28                       zbyszek           FAILED
rubygem-facade-1.0.5-8.fc28                              zbyszek           FAILED
rubygem-facets-2.8.0-14.fc28                             zbyszek           FAILED
rubygem-fog-1.38.0-5.fc28                                zbyszek           FAILED
rubygem-gemnasium-parser-0.1.9-8.fc28                    zbyszek           FAILED
rubygem-gettext-setup-0.25-4.fc28                        zbyszek           FAILED
rubygem-gruff-0.3.6-13.fc28                              zbyszek           FAILED
rubygem-haml-5.0.1-4.fc28                                zbyszek           FAILED
rubygem-hawler-0.3-18.fc28                               zbyszek           FAILED
rubygem-hipchat-1.4.0-6.fc28                             zbyszek           FAILED
rubygem-hydra-0.24.0-11.fc28                             zbyszek           FAILED
rubygem-ice_nine-0.11.1-6.fc28                           zbyszek           FAILED
rubygem-inflecto-0.0.2-7.fc28                            zbyszek           FAILED
rubygem-innertube-1.1.0-9.fc28                           zbyszek           FAILED
rubygem-joiner-0.3.4-6.fc28                              zbyszek           FAILED
rubygem-jruby-openssl-0.9.7-7.fc28                       zbyszek           FAILED
rubygem-just_paginate-0.2.2-9.fc28                       zbyszek           FAILED
rubygem-kwalify-0.7.2-15.fc28                            zbyszek           FAILED
rubygem-literati-0.0.4-8.fc28                            zbyszek           FAILED
rubygem-marc-1.0.2-2.fc28                                zbyszek           FAILED
rubygem-markaby-0.5-17.fc28                              zbyszek           FAILED
rubygem-middleware-0.1.0-7.fc28                          zbyszek           FAILED
rubygem-mongoid-6.1.0-3.fc28                             zbyszek           FAILED
rubygem-mono_logger-1.1.0-8.fc28                         zbyszek           FAILED
rubygem-net-http-persistent-2.9.4-9.fc28                 zbyszek           FAILED
rubygem-openscap-0.4.7-4.fc28                            zbyszek           FAILED
rubygem-openstack-quantum-client-0.1.5-13.fc28           zbyszek           FAILED
rubygem-orm_adapter-0.5.0-7.fc28                         zbyszek           FAILED
rubygem-pathname2-1.7.3-7.fc28                           zbyszek           FAILED
rubygem-pervasives-1.1.0-18.fc28                         zbyszek           FAILED
rubygem-picnic-0.8.1-15.fc28                             zbyszek           FAILED
rubygem-pkgwat-0.1.4-12.fc28                             zbyszek           FAILED
rubygem-puppet-lint-1.1.0-6.fc28                         zbyszek           FAILED
rubygem-rails-api-0.2.1-8.fc28                           zbyszek           FAILED
rubygem-rails_autolink-1.1.6-6.fc28                      zbyszek           FAILED
rubygem-redis-namespace-1.5.2-6.fc28                     zbyszek           FAILED
rubygem-resque-cleaner-0.3.0-8.fc28                      zbyszek           FAILED
rubygem-restr-0.5.0-15.fc28                              zbyszek           FAILED
rubygem-reststop-0.4.0-15.fc28                           zbyszek           FAILED
rubygem-rgen-0.7.0-5.fc28                                zbyszek           FAILED
rubygem-riddle-1.5.11-7.fc28                             zbyszek           FAILED
rubygem-riot-0.12.7-6.fc28                               zbyszek           FAILED
rubygem-rmail-1.0.0-16.fc28                              zbyszek           FAILED
rubygem-rots-0.2.2-4.20160208git8c278ef.fc28             zbyszek           FAILED
rubygem-rspec-longrun-0.1.2-9.fc28                       zbyszek           FAILED
rubygem-rspec2-core-2.14.8-9.fc28.2                      zbyszek           FAILED
rubygem-rufus-scheduler-2.0.4-14.fc28                    zbyszek           FAILED
rubygem-sanitize-2.1.0-9.fc28                            zbyszek           FAILED
rubygem-shoulda-3.5.0-6.fc28                             zbyszek           FAILED
rubygem-shoulda-matchers-2.8.0-5.fc28                    zbyszek           FAILED
rubygem-sprite-factory-1.7-5.fc28                        zbyszek           FAILED
rubygem-taskjuggler-3.5.0-6.fc28                         zbyszek           FAILED
rubygem-text-format-1.0.0-18.fc28                        zbyszek           FAILED
rubygem-thinking-sphinx-3.1.3-6.fc28                     zbyszek           FAILED
rubygem-tiltout-1.4.0-8.fc28                             zbyszek           FAILED
rubygem-virtus-1.0.5-6.fc28                              zbyszek           FAILED
rubygem-xmpp4r-0.5-16.fc28                               zbyszek           FAILED
rubygem-xmpp4r-simple-0.8.8-16.fc28                      zbyszek           FAILED
rust-addr2line-0.5.0-3.fc28                              zbyszek           FAILED
rust-seahash-3.0.5-3.fc28                                zbyszek           FAILED
scala-stm-0.7-9.fc28                                     zbyszek           FAILED
scalaz-7.0.0-9.fc28                                      zbyszek           FAILED
sfntly-0-0.14.r214.fc28                                  zbyszek           FAILED
shogun-6.0.0-10.fc28                                     zbyszek           FAILED
sigar-1.6.5-0.21.git58097d9.fc28                         zbyszek           FAILED
signon-glib-1.9-9.fc28                                   zbyszek           FAILED
silvia-0.2.2-0.15.9324db2git.fc28                        zbyszek           FAILED
sim-0.9.5-0.50.20091129svn3078rev.fc28                   zbyszek           FAILED
simplevalidation-1.0-0.8.SNAPSHOT.fc28                   zbyszek           FAILED
sinfo-0.0.48-15.fc28                                     zbyszek           FAILED
sipp-3.5.1-4.fc28                                        zbyszek           FAILED
sks-1.1.6-6.fc28                                         zbyszek           FAILED
snoopy-2.2.6-7.fc28                                      zbyszek           FAILED
snownews-1.5.12-18.fc28                                  zbyszek           FAILED
soletta-1-7.fc28                                         zbyszek           FAILED
soundkonverter-2.2.2-5.fc28                              zbyszek           FAILED
soundmodem-0.20-9.fc28                                   zbyszek           FAILED
spice-xpi-2.8.90-14.fc28                                 zbyszek           FAILED
sqlite2-2.8.17-24.fc28                                   zbyszek           FAILED
sslogger-0.96-19.fc28                                    zbyszek           FAILED
stapler-1.242-4.fc28                                     zbyszek           FAILED
statsd-0.7.2-11.fc28                                     zbyszek           FAILED
stax-utils-0-0.13.20110309svn238.fc28                    zbyszek           FAILED
steadyflow-0.2.0-11.fc28                                 zbyszek           FAILED
steghide-0.5.1-33.fc28                                   zbyszek           FAILED
stratagus-2.2.7-12.fc28                                  zbyszek           FAILED
subsurface-4.6.2-7.fc28                                  zbyszek           FAILED
sugar-colordeducto-7-10.fc28                             zbyszek           FAILED
sugar-pippy-71-2.fc28                                    zbyszek           FAILED
sugar-story-15-5.fc28                                    zbyszek           FAILED
sugar-visualmatch-49-8.fc28                              zbyszek           FAILED
sugar-xoeditor-13-7.fc28                                 zbyszek           FAILED
sugar-yupana-17-6.fc28                                   zbyszek           FAILED
svxlink-15.11-5.fc28                                     zbyszek           FAILED
swift-3.0-0.14.rc2.fc28                                  zbyszek           FAILED
swing-layout-1.0.4-15.fc28                               zbyszek           FAILED
synapse-0.2.99.2-6.fc28                                  zbyszek           FAILED
tagainijisho-1.0.3-8.fc28                                zbyszek           FAILED
takari-smart-builder-0.6.0-3.fc28                        zbyszek           FAILED
target-isns-0.6.2-6.fc28                                 zbyszek           FAILED
taskjuggler-2.4.3-27.fc28                                zbyszek           FAILED
tcl-tclxml-3.2-21.fc28                                   zbyszek           FAILED
tclabc-1.3.7-6.fc28                                      zbyszek           FAILED
tcmu-runner-1.1.3-5.fc28                                 zbyszek           FAILED
tcpcrypt-0.4-5.fc28                                      zbyszek           FAILED
teal-1_40b-19.fc28                                       zbyszek           FAILED
telepathy-haze-0.8.0-10.fc28                             zbyszek           FAILED
termit-2.9.6-12.fc28                                     zbyszek           FAILED
tinyproxy-1.8.4-4.fc28                                   zbyszek           FAILED
tinyxpath-1.3.1-8.fc28                                   zbyszek           FAILED
tipcutils-2.0.6-7.fc28                                   zbyszek           FAILED
tlsdate-0.0.13-8.fc28                                    zbyszek           FAILED
torcs-1.3.7-5.fc28                                       zbyszek           FAILED
tqsllib-2.2-16.fc28                                      zbyszek           FAILED
trafficserver-5.3.0-10.fc28                              zbyszek           FAILED
transgui-5.0.1-11.fc28                                   zbyszek           FAILED
treecc-0.3.8-21.fc28                                     zbyszek           FAILED
trickle-1.07-25.fc28                                     zbyszek           FAILED
tritonus-0.3.7-0.32.20101108cvs.fc28                     zbyszek           FAILED
truth-0.23-7.fc28                                        zbyszek           FAILED
ttname-1-13.fc28                                         zbyszek           FAILED
tunneler-1.1.1-21.fc28                                   zbyszek           FAILED
tuxcmd-0.6.70-23.fc28                                    zbyszek           FAILED
ucblogo-6.0-24.fc28                                      zbyszek           FAILED
udev-browse-0.3-11.fc28                                  zbyszek           FAILED
ufw-kde-0.5.0-0.8.20161006git.fc28                       zbyszek           FAILED
unison213-2.13.16-34.fc28                                zbyszek           FAILED
unison227-2.27.157-10.fc28                               zbyszek           FAILED
unison240-2.40.128-10.fc28                               zbyszek           FAILED
usnic-tools-1.1.1.0-5.fc28                               zbyszek           FAILED
ustl-2.5-5.fc28                                          zbyszek           FAILED
v8-314-3.14.5.10-9.fc28                                  zbyszek           FAILED
valabind-1.4.0-2.fc28                                    zbyszek           FAILED
w3c-libwww-5.4.1-0.31.20060206cvs.fc28                   zbyszek           FAILED
webkit-sharp-0.3-22.fc28                                 zbyszek           FAILED
weld-core-2.3.5-4.fc28                                   zbyszek           FAILED
weplab-0.1.5-22.fc28                                     zbyszek           FAILED
whatsup-1.14-22.fc28                                     zbyszek           FAILED
wine-docs-1.4-10.fc28                                    zbyszek           FAILED
wmudmount-2.2-8.fc28                                     zbyszek           FAILED
woden-1.0-0.16.M9.fc28                                   zbyszek           FAILED
wyrd-1.4.5-17.fc28                                       zbyszek           FAILED
xar-1.5.2-19.fc28                                        zbyszek           FAILED
xfce-theme-manager-0.3.6-6.fc28                          zbyszek           FAILED
xmltool-3.3-20.fc28                                      zbyszek           FAILED
xmount-0.7.3-5.fc28                                      zbyszek           FAILED
xournal-0.4.8-10.fc28                                    zbyszek           FAILED
xpa-2.1.18-4.fc28                                        zbyszek           FAILED
yubikey-val-2.39-2.fc28                                  zbyszek           FAILED
zfs-fuse-0.7.2.2-8.fc28                                  zbyszek           FAILED
zhu3d-4.2.6-12.fc28                                      zbyszek           FAILED

Here is the rel-eng bug report on missing build logs for the mass rebuild FTBFS bugs: https://pagure.io/releng/issue/7468

Rel-Eng Bug Report for packages that were not rebuilt completely at all:
https://pagure.io/releng/issue/7469

BTW

$ dnf repoquery --disablerepo='*' --enablerepo='rawhide-source' -a | tee sources
$ grep fc29 sources | wc -l
5928
$ grep fc28 sources | wc -l
13413
$ grep fc27 sources | wc -l
1621
$ grep fc26 sources | wc -l
185
$ grep fc25 sources | wc -l
27
$ grep fc24 sources | wc -l
141
$ grep fc23 sources | wc -l
42
$ grep fc22 sources | wc -l
25
$ grep fc21 sources | wc -l
34
$ grep fc20 sources | wc -l
2
$ grep fc19 sources | wc -l
0
$ grep fc18 sources | wc -l
0

This is on the agenda for Friday's FESCo meeting.

AGREED: let's defer this ticket for 2 weeks to give releng/tyll/zbyszek time to sort things out some more. let's also file a new ticket to revisit FTBFS policy

https://meetbot.fedoraproject.org/teams/fesco/fesco.2018-05-04-15.00.html

Metadata Update from @bowlofeggs:
- Issue untagged with: meeting

5 years ago

F28 is out. I don't think there's much we can do for F28 at this point. My proposal is to:
1. close this bug now
2. handle the issues with releng scripts in https://pagure.io/releng/issue/7469
3. update the policy in https://pagure.io/fesco/issue/1890 and commit to doing things "right" in F29

In particular, for the 770 packages which have open FTBFS-F28 bugs, I think we should consider the clock already running, and retire them before F29 (according to the refreshed policy, if it gets approved). But let's discuss that as part of the policy ticket too.

Metadata Update from @zbyszek:
- Issue tagged with: meeting

5 years ago

This was discussed in the FESCo meeting today:
AGREED: Close this bug (+7, 0, 0)
releng ticket #7469 and fesco ticket 1890 remain open to track related issues

Metadata Update from @zbyszek:
- Issue untagged with: meeting
- Issue status updated to: Closed (was: Open)

5 years ago

Login to comment on this ticket.

Metadata