Looking quickly at the logs, the koji_import task is failing. I ll look at it in more details tomorrow.

Thanks, @cverna!

A simple container builds works (https://koji.fedoraproject.org/koji/taskinfo?taskID=41405711), so it seems that this is an flatpak specific issue. Did anything change in the metadata sent to koji recently ?

@otaylor would you know ?

It also worked for approx 73/75 Flatpaks Kalev built. I think a first step
would be looking at the kojihub logs corresponding to the rpc request where
the response couldn't be parsed to see if there is anything there.

It also worked for approx 73/75 Flatpaks Kalev built.
Yes that's why I asked if something changed recently.
I think a first step
would be looking at the kojihub logs corresponding to the rpc request where
the response couldn't be parsed to see if there is anything there.
It is not that easy to find a request in koji logs there is a loooot of output. I ll try to replicate in staging that might be easier to debug.

I think the difference is that we're now including Flatpaks metadata -
which includes icons, translated descriptions, etc. as labels, and (if
I remember correctly) the labels are included in the Koji metadata, and
either the size, or some escaping problem, or something with encoding is
causing a problem with the Koji import.

Thanks.

Just to be sure, I did another built attempt for redeclipse just now and it's still failing the same way (https://koji.fedoraproject.org/koji/taskinfo?taskID=41406689). I've built several other flatpaks in the mean time and no problems there, it's only redeclipse and hydrogen that are failing.

'fedpkg clone flatpaks/redeclipse' and 'fedpkg flatpak-build' reproduce this for me.

I took a look at the logged metadata from a failed build, and there was nothing suspicious there - the redeclipse appstream data is pretty minimal, and parses correctly at XML (hopefully that wouldn't matter!)

Could still patch the package to strip it down even further or remove the appstream data entirely (create a private- branch for the RPM, change the module to point at that) - but I think that's only if getting logs from koji or staging koji fails.

So I have been trying to build hydrogen in staging and I currently have the following error during the container build.

2020-02-20 12:18:51,539 - atomic_reactor.inner - INFO - Dockerfile used for build:
FROM registry.fedoraproject.org/fedora:latest

LABEL name="hydrogen"
LABEL com.redhat.component="hydrogen"
LABEL version="master"
LABEL release="3020191105065943"

ADD atomic-reactor-includepkgs /tmp/

RUN mkdir -p /var/tmp/flatpak-build/dev &&     for i in null zero random urandom ; do cp -a /dev/$i /var/tmp/flatpak-build/dev ; done

RUN cat /tmp/atomic-reactor-includepkgs >> /etc/dnf/dnf.conf && \
    INSTALLDIR=/var/tmp/flatpak-build && \
    DNF='\
    dnf -y --nogpgcheck \
    --disablerepo=* \
    --enablerepo=atomic-reactor-koji-plugin-* \
    --enablerepo=atomic-reactor-module-* \
    ' && \
    $DNF --installroot=$INSTALLDIR-init install system-release && \
    mkdir -p $INSTALLDIR/etc/ && \
    cp $INSTALLDIR-init/etc/os-release $INSTALLDIR/etc/os-release && \
    $DNF --installroot=$INSTALLDIR module enable flatpak-common:f30 flatpak-runtime:f30 hydrogen:master && \
    $DNF --installroot=$INSTALLDIR install hydrogen flatpak-runtime-config
RUN rpm --root=/var/tmp/flatpak-build -qa --qf '%{NAME};%{VERSION};%{RELEASE};%{ARCH};%{EPOCH};%{SIZE};%{SIGMD5};%{BUILDTIME};%{SIGPGP:pgpsig};%{SIGGPG:pgpsig}\n' > /var/tmp/flatpak-build.rpm_qf
COPY cleanup.sh /var/tmp/flatpak-build/tmp/
RUN chroot /var/tmp/flatpak-build/ /bin/sh /tmp/cleanup.sh

LABEL "release"="3020191105065943.2" "authoritative-source-url"="registry.fedoraproject.org" "distribution-scope"="public" "vendor"="Fedora Project" "build-date"="2020-02-20T12:16:04.040963" "architecture"="x86_64" "vcs-type"="git" "vcs-ref"="5877ddfc6bc95cf6703870e7d51d250a0064bf17" "com.redhat.build-host"="osbs-node01.stg.phx2.fedoraproject.org"

2020-02-20 12:18:51,539 - atomic_reactor.inner - ERROR - image build failed: The command '/bin/sh -c cat /tmp/atomic-reactor-includepkgs >> /etc/dnf/dnf.conf &&     INSTALLDIR=/var/tmp/flatpak-build &&     DNF='    dnf -y --nogpgcheck     --disablerepo=*     --enablerepo=atomic-reactor-koji-plugin-*     --enablerepo=atomic-reactor-module-*     ' &&     $DNF --installroot=$INSTALLDIR-init install system-release &&     mkdir -p $INSTALLDIR/etc/ &&     cp $INSTALLDIR-init/etc/os-release $INSTALLDIR/etc/os-release &&     $DNF --installroot=$INSTALLDIR module enable flatpak-common:f30 flatpak-runtime:f30 hydrogen:master &&     $DNF --installroot=$INSTALLDIR install hydrogen flatpak-runtime-config' returned a non-zero code: 1
Traceback (most recent call last):
  File "/usr/bin/atomic-reactor", line 11, in <module>
    load_entry_point('atomic-reactor==1.6.47', 'console_scripts', 'atomic-reactor')()
  File "/usr/lib/python3.7/site-packages/atomic_reactor/cli/main.py", line 318, in run
    cli.run()
  File "/usr/lib/python3.7/site-packages/atomic_reactor/cli/main.py", line 300, in run
    args.func(args)
  File "/usr/lib/python3.7/site-packages/atomic_reactor/cli/main.py", line 99, in cli_inside_build
    substitutions=args.substitute)
  File "/usr/lib/python3.7/site-packages/atomic_reactor/inner.py", line 615, in build_inside
    build_result = dbw.build_docker_image()
  File "/usr/lib/python3.7/site-packages/atomic_reactor/inner.py", line 512, in build_docker_image
    raise PluginFailedException(self.build_result.fail_reason)
atomic_reactor.plugin.PluginFailedException: The command '/bin/sh -c cat /tmp/atomic-reactor-includepkgs >> /etc/dnf/dnf.conf &&     INSTALLDIR=/var/tmp/flatpak-build &&     DNF='    dnf -y --nogpgcheck     --disablerepo=*     --enablerepo=atomic-reactor-koji-plugin-*     --enablerepo=atomic-reactor-module-*     ' &&     $DNF --installroot=$INSTALLDIR-init install system-release &&     mkdir -p $INSTALLDIR/etc/ &&     cp $INSTALLDIR-init/etc/os-release $INSTALLDIR/etc/os-release &&     $DNF --installroot=$INSTALLDIR module enable flatpak-common:f30 flatpak-runtime:f30 hydrogen:master &&     $DNF --installroot=$INSTALLDIR install hydrogen flatpak-runtime-config' returned a non-zero code: 1

Do you have an idea on how I could try to debug this ? or maybe you have an idea of what could have gone wrong ?

The koji task is here --> https://koji.stg.fedoraproject.org/koji/taskinfo?taskID=90009547

Ok so odcs was not working properly after running the following commands it is now fixed and we are able to reproduce the error in stg.

on odcs-frontend01.stg.phx2.fedoraproject.org I ran :

$ umount /mnt/koji_fedora && mount /mnt/koji_fedora

Good :) Thank you for looking into this!

I have added a few debug logs on the koji-hub in stg and I was able to get the following output

https://cverna.fedorapeople.org/error_log

And I have extracted the xml from that log

https://cverna.fedorapeople.org/xml_data.txt

According to the exception

[Mon Feb 24 08:57:07.992134 2020] [:error] [pid 39644] ExpatError: not well-formed (invalid token): line 41231, column 35

the following line is the problem

    <id>org.hydrogenmusic.Hydrogen</id>

But I am not 100% sure what is wrong there :pensive:

I'm thinking that the problem is at character 35 on the next line:

<summary xml:lang="fr">Bo\xeete \xe0 rythme Hydrogen</summary>

Character 35 is the \xee - it looks like what was declared had ISO-8859-1 in it - \xee is î and \xe0 is à

if this is right, the immediate fix is to patch the Hydrogen package to fix the apstream data (and contribute the fix back upstream) - but we also have other issues to file:

flatpak should have validated the XML of the appstream data before embedding it as a label
OSBS shouldn't fail uploading to Koji if a label has arbitrary binary data in it.

Turns out to be a problem with the Python 3 port of the koji client library:

https://pagure.io/koji/pull-request/2040

The thing that makes those two Flatpaks special is that they have appdata that has translations only to Latin-1 languages and no other languages.

Oh, awesome! Thanks for tracking that down, @otaylor and @cverna!

We are in freeze now, but should we backport that patch after freeze?

Or should we ask for a freeze break?

It would be good to get upstream mainainers acking the fix too.

After freeze should be fine. No particular rush here.

@otaylor @kalev do we want to wait for upstream on this ticket ? or should we try to carry the patch for now ?

I think we can wait a bit for upstream to comment - but if it looks like a while until there will be a release with this fix, we should carry this patch in Fedora or an infra build. (My confidence in the patch is pretty high.)

The Upstream PR was merged today, do we want to close that ticket or wait for a new version of koji to be released and deployed ?

Nice! Either way works for me.

Ok closing then. I ll keep an eye on the next koji release and update that ticket when that happens.

OK, I see new koji is finally deployed and I can confirm this is fixed. Thanks, @otaylor and @cverna both!