#1703 Fedora-30-20190319.0 DOOMED
Opened a year ago by dustymabe. Modified a year ago

pungi.global.log

[LIVE_IMAGES     ] [ERROR   ] [FAIL] Live (variant Server, arch armhfp, subvariant Server) failed, but going on anyway.
[LIVE_IMAGES     ] [ERROR   ] LiveImage task failed: 33627373. See /mnt/koji/compose/30/Fedora-30-20190319.0/logs/armhfp/liveimage-Fedora-Server-armhfp-30_Beta-1.1.armhfp.log for more details.
[LIVE_IMAGES     ] [ERROR   ] [FAIL] Live (variant Workstation, arch armhfp, subvariant Workstation) failed, but going on anyway.
[LIVE_IMAGES     ] [ERROR   ] LiveImage task failed: 33627427. See /mnt/koji/compose/30/Fedora-30-20190319.0/logs/armhfp/liveimage-Fedora-Workstation-armhfp-30_Beta-1.1.armhfp.log for more details.
[LIVE_IMAGES     ] [ERROR   ] [FAIL] Live (variant Spins, arch armhfp, subvariant LXDE) failed, but going on anyway.
[LIVE_IMAGES     ] [ERROR   ] LiveImage task failed: 33627379. See /mnt/koji/compose/30/Fedora-30-20190319.0/logs/armhfp/liveimage-Fedora-LXDE-armhfp-30_Beta-1.1.armhfp.log for more details.
[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Astronomy_KDE) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 33627377. See /mnt/koji/compose/30/Fedora-30-20190319.0/logs/i386-x86_64/livemedia-Labs-Astronomy_KDE.i386-x86_64.log for more details.
[LIVE_IMAGES     ] [ERROR   ] [FAIL] Live (variant Labs, arch armhfp, subvariant Python_Classroom) failed, but going on anyway.
[LIVE_IMAGES     ] [ERROR   ] LiveImage task failed: 33627372. See /mnt/koji/compose/30/Fedora-30-20190319.0/logs/armhfp/liveimage-Fedora-Python-Classroom-armhfp-30_Beta-1.1.armhfp.log for more details.
[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Scientific_KDE) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 33627393. See /mnt/koji/compose/30/Fedora-30-20190319.0/logs/i386-x86_64/livemedia-Labs-Scientific_KDE.i386-x86_64.log for more details.
[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Design_suite) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 33627380. See /mnt/koji/compose/30/Fedora-30-20190319.0/logs/i386-x86_64/livemedia-Labs-Design_suite.i386-x86_64.log for more details.
[LIVE_IMAGES     ] [ERROR   ] [FAIL] Live (variant Spins, arch armhfp, subvariant SoaS) failed, but going on anyway.
[LIVE_IMAGES     ] [ERROR   ] LiveImage task failed: 33627410. See /mnt/koji/compose/30/Fedora-30-20190319.0/logs/armhfp/liveimage-Fedora-SoaS-armhfp-30_Beta-1.1.armhfp.log for more details.
[LIVE_IMAGES     ] [ERROR   ] [FAIL] Live (variant Spins, arch armhfp, subvariant KDE) failed, but going on anyway.
[LIVE_IMAGES     ] [ERROR   ] LiveImage task failed: 33627375. See /mnt/koji/compose/30/Fedora-30-20190319.0/logs/armhfp/liveimage-Fedora-KDE-armhfp-30_Beta-1.1.armhfp.log for more details.
[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Robotics) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 33627425. See /mnt/koji/compose/30/Fedora-30-20190319.0/logs/i386-x86_64/livemedia-Labs-Robotics.i386-x86_64.log for more details.
[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Games) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 33627400. See /mnt/koji/compose/30/Fedora-30-20190319.0/logs/i386-x86_64/livemedia-Labs-Games.i386-x86_64.log for more details.
[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Jam_KDE) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 33627420. See /mnt/koji/compose/30/Fedora-30-20190319.0/logs/i386-x86_64/livemedia-Labs-Jam_KDE.i386-x86_64.log for more details.
[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Security) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 33627413. See /mnt/koji/compose/30/Fedora-30-20190319.0/logs/i386-x86_64/livemedia-Labs-Security.i386-x86_64.log for more details.
[LIVE_IMAGES     ] [ERROR   ] [FAIL] Live (variant Spins, arch armhfp, subvariant LXQt) failed, but going on anyway.
[LIVE_IMAGES     ] [ERROR   ] LiveImage task failed: 33627424. See /mnt/koji/compose/30/Fedora-30-20190319.0/logs/armhfp/liveimage-Fedora-LXQt-armhfp-30_Beta-1.1.armhfp.log for more details.
[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Python_Classroom) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 33627431. See /mnt/koji/compose/30/Fedora-30-20190319.0/logs/i386-x86_64/livemedia-Labs-Python_Classroom.i386-x86_64.log for more details.
[LIVE_IMAGES     ] [ERROR   ] [FAIL] Live (variant Spins, arch armhfp, subvariant Mate) failed, but going on anyway.
[LIVE_IMAGES     ] [ERROR   ] LiveImage task failed: 33627385. See /mnt/koji/compose/30/Fedora-30-20190319.0/logs/armhfp/liveimage-Fedora-Mate-armhfp-30_Beta-1.1.armhfp.log for more details.
[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Spins, arch *, subvariant Xfce) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 33627436. See /mnt/koji/compose/30/Fedora-30-20190319.0/logs/i386-x86_64/livemedia-Spins-Xfce.i386-x86_64.log for more details.
[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Spins, arch *, subvariant SoaS) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 33627439. See /mnt/koji/compose/30/Fedora-30-20190319.0/logs/i386-x86_64/livemedia-Spins-SoaS.i386-x86_64.log for more details.
[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Spins, arch *, subvariant LXDE) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 33627446. See /mnt/koji/compose/30/Fedora-30-20190319.0/logs/i386-x86_64/livemedia-Spins-LXDE.i386-x86_64.log for more details.
[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Spins, arch *, subvariant Cinnamon) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 33627445. See /mnt/koji/compose/30/Fedora-30-20190319.0/logs/i386-x86_64/livemedia-Spins-Cinnamon.i386-x86_64.log for more details.
[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Spins, arch *, subvariant Mate) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 33627451. See /mnt/koji/compose/30/Fedora-30-20190319.0/logs/i386-x86_64/livemedia-Spins-Mate.i386-x86_64.log for more details.
[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Spins, arch *, subvariant LXQt) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 33627456. See /mnt/koji/compose/30/Fedora-30-20190319.0/logs/i386-x86_64/livemedia-Spins-LXQt.i386-x86_64.log for more details.
  • No Task ID, look at log statement
[ERROR   ] [FAIL] Image build (variant Cloud, arch s390x, subvariant Cloud_Base) failed, but going on anyway.
[ERROR   ] [FAIL] Image build (variant Cloud, arch ppc64le, subvariant Cloud_Base) failed, but going on anyway.
  • No Task ID, look at log statement
[ERROR   ] [FAIL] Image build (variant Cloud, arch ppc64le, subvariant Cloud_Base) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] Hardlinking /mnt/koji/packages/Fedora-Cloud-Base/30_Beta/1.1/images/Fedora-Cloud-Base-30_Beta-1.1.aarch64.qcow2 to /mnt/koji/compose/30/Fedora-30-20190319.0/compose/Cloud/aarch64/images/Fedora-Cloud-Base-30_Beta-1.1.aarch64.qcow2
  • No Task ID, look at log statement
[ERROR   ] [FAIL] Image build (variant Container, arch armhfp, subvariant Container_Base) failed, but going on anyway.
[ERROR   ] [FAIL] Image build (variant Container, arch s390x, subvariant Container_Base) failed, but going on anyway.
  • No Task ID, look at log statement
[ERROR   ] [FAIL] Image build (variant Container, arch s390x, subvariant Container_Base) failed, but going on anyway.
[ERROR   ] [FAIL] Image build (variant Container, arch ppc64le, subvariant Container_Base) failed, but going on anyway.
  • No Task ID, look at log statement
[ERROR   ] [FAIL] Image build (variant Container, arch ppc64le, subvariant Container_Base) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] Hardlinking /mnt/koji/packages/Fedora-Container-Base/30_Beta/1.1/images/Fedora-Container-Base-30_Beta-1.1.aarch64.tar.xz to /mnt/koji/compose/30/Fedora-30-20190319.0/compose/Container/aarch64/images/Fedora-Container-Base-30_Beta-1.1.aarch64.tar.xz
  • No Task ID, look at log statement
[ERROR   ] [FAIL] Image build (variant Container, arch armhfp, subvariant Container_Minimal_Base) failed, but going on anyway.
[ERROR   ] [FAIL] Image build (variant Container, arch ppc64le, subvariant Container_Minimal_Base) failed, but going on anyway.
  • No Task ID, look at log statement
[ERROR   ] [FAIL] Image build (variant Container, arch ppc64le, subvariant Container_Minimal_Base) failed, but going on anyway.
[ERROR   ] [FAIL] Image build (variant Container, arch s390x, subvariant Container_Minimal_Base) failed, but going on anyway.
  • No Task ID, look at log statement
[ERROR   ] [FAIL] Image build (variant Container, arch s390x, subvariant Container_Minimal_Base) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] Hardlinking /mnt/koji/packages/Fedora-Container-Minimal-Base/30_Beta/1.1/images/Fedora-Container-Minimal-Base-30_Beta-1.1.aarch64.tar.xz to /mnt/koji/compose/30/Fedora-30-20190319.0/compose/Container/aarch64/images/Fedora-Container-Minimal-Base-30_Beta-1.1.aarch64.tar.xz
[IMAGE_BUILD     ] [ERROR   ] [FAIL] Image build (variant Labs, arch *, subvariant Scientific) failed, but going on anyway.
[IMAGE_BUILD     ] [ERROR   ] ImageBuild task failed: 33627415. See /mnt/koji/compose/30/Fedora-30-20190319.0/logs/i386-x86_64/imagebuild-Labs-Scientific-vagrant-libvirt-vagrant-virtualbox.i386-x86_64.log for more details.
[IMAGE_BUILD     ] [ERROR   ] [FAIL] Image build (variant Server, arch *, subvariant Server) failed, but going on anyway.
[IMAGE_BUILD     ] [ERROR   ] ImageBuild task failed: 33627421. See /mnt/koji/compose/30/Fedora-30-20190319.0/logs/aarch64-armhfp/imagebuild-Server-Server-raw-xz.aarch64-armhfp.log for more details.
  • No Task ID, look at log statement
[ERROR   ] [FAIL] Image build (variant Spins, arch armhfp, subvariant Minimal) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] Hardlinking /mnt/koji/packages/Fedora-Minimal/30_Beta/1.1/images/Fedora-Minimal-30_Beta-1.1.aarch64.raw.xz to /mnt/koji/compose/30/Fedora-30-20190319.0/compose/Spins/aarch64/images/Fedora-Minimal-30_Beta-1.1.aarch64.raw.xz
  • No Task ID, look at log statement
[ERROR   ] [FAIL] Image build (variant Workstation, arch armhfp, subvariant Workstation) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] Hardlinking /mnt/koji/packages/Fedora-Workstation/30_Beta/1.1/images/Fedora-Workstation-30_Beta-1.1.aarch64.raw.xz to /mnt/koji/compose/30/Fedora-30-20190319.0/compose/Workstation/aarch64/images/Fedora-Workstation-30_Beta-1.1.aarch64.raw.xz
  • Compose run failed because: - 33627398
[ERROR   ] Compose run failed: LiveImage task failed: 33627398. See /mnt/koji/compose/30/Fedora-30-20190319.0/logs/armhfp/liveimage-Fedora-Minimal-armhfp-30_Beta-1.1.armhfp.log for more details.

Not sure what happened, so I am running a scratch appliance build again.

https://koji.fedoraproject.org/koji/taskinfo?taskID=33630750

Not sure what happened, so I am running a scratch appliance build again.
https://koji.fedoraproject.org/koji/taskinfo?taskID=33630750

This is just great, now it hit Error in the HTTP2 framing layer :sob:

Refiring again: https://koji.fedoraproject.org/koji/taskinfo?taskID=33630985

Weird and weirder errors showing up:

DEBUG util.py:439:  repo: downloading from remote: koji-override-0
DEBUG util.py:439:  error: Curl error (28): Timeout was reached for https://kojipkgs.fedoraproject.org/compose/30/Fedora-30-20190319.0/compose/Everything/armhfp/os/repodata/b2be61abcb4da43e69f09ae32fd49c12866d04a2d3e2c42f4df806706177f9e6-comps-Everything.armhfp.xml.zck [SSL connection timeout] (https://kojipkgs.fedoraproject.org/compose/30/Fedora-30-20190319.0/compose/Everything/armhfp/os/repodata/b2be61abcb4da43e69f09ae32fd49c12866d04a2d3e2c42f4df806706177f9e6-comps-Everything.armhfp.xml.zck).
DEBUG util.py:439:  error: Zchunk header checksum didn't match expected checksum (https://kojipkgs.fedoraproject.org/compose/30/Fedora-30-20190319.0/compose/Everything/armhfp/os/repodata/b2be61abcb4da43e69f09ae32fd49c12866d04a2d3e2c42f4df806706177f9e6-comps-Everything.armhfp.xml.zck)

Hoping 3rd time is the charm: https://koji.fedoraproject.org/koji/taskinfo?taskID=33631045

Did the original appearance of these http/2 framing errors coincide with the enabling of zchunk support?

I'm starting to suspect this is related to the partial range request thing that's part of the whole zchunk setup.

I'm starting to suspect this is related to the partial range request thing that's part of the whole zchunk setup.

IOW not a bug in zchunk, but exposing a code path in http request software that is probably not exercised often and probably a bug there? cc @jdieter

Ok, the initial error is because appliance-creator doesn't understand zchunked comps.xml, while the metadata was generated by createrepo_c-0.12.2, which is once again generating zchunked comps.xml.

createrepo_c-0.12.2 is part of an update that includes an updated dnf that's capable of reading zchunked comps.xml, so I'm not sure why appliance-creator isn't working. @ngompa, does appliance-creator call dnf directly or is it trying to open comps.xml directly?

We don't care an awful lot about the initial error here, we know what that is. What we care about is the http/2 framing errors which we are now running into constantly - they have failed nearly every Branched compose in the last two weeks - but which I think did not happen at all until the zchunk stuff landed.

edit: OK, let me qualify that a bit. Obviously we care about the initial issue - I suspect there's a problem going on there where we're not getting the packages from the override repo in some environment where they're needed. But aside from that problem, when we run a compose that doesn't have that problem (e.g. a compose without an override repo), it just about always fails on an http/2 framing error when trying to download the zchunk metadata. That's what happened to most of the test builds @mohanboddu ran in this issue. It's what happened to 20190318.n.0. And 20190317.n.0. And 20190313.n.0. And 20190311.n.0...

It's always the zchunk metadata it blows up on. (And it always seems to be the Xfce armhfp image...)

Yeah, sorry, I should have made it clear that I am looking into that too.

A couple of days ago, I pushed out a zchunk build that fixes a bug in how zchunk handles multipart boundaries. Basically, if the web server offered a non-hex boundary (something that I only ran across when testing with a lighttpd server), zchunk wouldn't be able to read the data.

The normal message in dnf.librepo.log that indicates this error is:
Curl error (56): Failure when receiving data from the peer

I haven't come across any HTTP/2 framing error messages when trying to reproduce the error, but I did hit this error one of the ten to fifteen times I hit the kojipkgs server.

Is there any chance we can tag that build into the scratch appliance build?

Is there any way we can get the dnf.librepo.log from one of the systems hitting the HTTP/2 framing error?

I am seeing that error in some of the failure cases, yes, but it definitely seems to be a different thing from the HTTP/2 framing error.

You can look through failed createImage tasks here. Disregard the 30-Beta-1.1 tasks as they definitely had some issue with mismatched dnf package versions (the versions from the override repo weren't pulled into the mock environments). If you look at the Rawhide and Branched nightlies, aside from ones that fail on package deps or buildroot size issues and whatknot, you'll see multiple cases of the HTTP/2 framing error and multiple cases of the '56' error.

So if you have a fix for the '56' error that's great and we should probably send it to F30 stable, but I don't think it's the same as the HTTP/2 framing error.

Note the most recent Rawhide compose, Fedora-Rawhide-201903019.n.0, also failed on the HTTP/2 framing error. That would have had the latest dnf, libdnf, librepo and zchunk - everything currently tagged f31.

Yeah, you're right. That last Rawhide compose had zchunk-1.0.4, which should have fixed the '56' error, but obviously didn't fix the HTTP/2 error. Is there any way we can get the dnf.librepo.log from it? If not, I've got a Raspberry Pi that I can install armhfp Rawhide on and try to debug it from there.

@mohanboddu might be able to get a hold of the log from one of his scratch tasks, maybe. I don't think we get at it from the 'outside'.

@jdieter since we are very much up against the F30 Beta deadline here, is there some relatively simple way we can just turn off zchunk support, if that's what we decide we want to do to try and get Beta done? At this point if we do not get a Beta compose done today, the first Beta date is basically gone.

@adamwill, you should be able to just set zchunk=False in /etc/dnf/dnf.conf

what "should be handled" there?

The comps groups handing logic.

To be clear, I don't handle metadata directly at all.

@ngompa, no problem, it looks like it was a mismatch between dnf versions in the override repo and mock. That's been sorted (I think), but we've got a larger problem with the HTTP/2 framing error.

@jdieter no, it should be handled by DNF through dnfinst in imgcreate: https://github.com/livecd-tools/livecd-tools/blob/master/imgcreate/dnfinst.py

@adamwill, just looking at this code in the appliance-creator, it looks like it generates dnf.conf manually, so you might need a patch to add zchunk=False there.

well, I think we'd be on a bit of a fool's errand if we tried to leave the zchunk metadata intact but ensure everything that does anything in the compose process doesn't use it. I was more thinking of an easy way we can just not have the zchunk metadata generated at all, so nothing can try to use it and blow up.

@adamwill, the easiest way to do that is update fedora.conf in pungi-fedora to drop the createrepo_extra_args.

@adamwill, I've done a https://pagure.io/pungi-fedora/pull-request/706 to stop zchunk metadata generation. You'll need to find someone to merge it.

@adamwill, I've done a https://pagure.io/pungi-fedora/pull-request/706 to stop zchunk metadata generation. You'll need to find someone to merge it.

@jdieter Please fix the fedora-beta.conf file as well. Thanks

Done. Thanks for merging!

@mohanboddu, I realize you're probably really busy at the moment, but if I could get /var/log/dnf.librepo.log from one of your scratch builds that failed with the HTTP/2 framing error, I'd greatly appreciate it.

I got it from buildvm-armv7-04.arm.fedoraproject.org which is used in task: 33631110

@mohanboddu, thanks so much for getting that for me, but it looks like it's the logs from the builder's local updates rather than the task updates. If the task is called in a chroot, possibly the logs are in a different location (or may just be sent to /dev/null).

It looks like I might have to manually test using one of the test arm servers and see if I can reproduce there.

@mohanboddu, @adamwill Ok, some progress. I've managed to reproduce on one of the Fedora test systems (specifically arm03-packager00.cloud.fedoraproject.org).

I installed a minimal system into /home/fedora/jdieter/install_root, and then changed fedora.repo in the install_root to point to https://kojipkgs.fedoraproject.org/compose/$releasever/Fedora-30-20190319.0/compose/Everything/$basearch/os.

When clearing the cache and doing a simple dnf --disablerepo=* --enablerepo=fedora list available, it fails roughly 1/2 the time. /var/log/dnf.librepo.log contains either of the following:

2019-03-20T20:15:34Z DEBUG check_transfer_statuses: Transfer finished: repodata/161c873a9a780caa43c51465415f6863abd59593df9a9ceaa6232621eced317f-primary.xml.zck (Effective url: https://kojipkgs.fedoraproject.org/compose/30/Fedora-30-20190319.0/compose/Everything/armhfp/os/repodata/161c873a9a780caa43c51465415f6863abd59593df9a9ceaa6232621eced317f-primary.xml.zck)
2019-03-20T20:15:34Z DEBUG check_transfer_statuses: Error during transfer: Curl error (16): Error in the HTTP2 framing layer for https://kojipkgs.fedoraproject.org/compose/30/Fedora-30-20190319.0/compose/Everything/armhfp/os/repodata/161c873a9a780caa43c51465415f6863abd59593df9a9ceaa6232621eced317f-primary.xml.zck []

OR

2019-03-20T20:15:34Z DEBUG check_transfer_statuses: Transfer finished: repodata/102ec5f532a57deab24ff8fa0a1d031570a8fc2c8491f476279baa56a9a31c08-filelists.xml.zck (Effective url: https://kojipkgs.fedoraproject.org/compose/30/Fedora-30-20190319.0/compose/Everything/armhfp/os/repodata/102ec5f532a57deab24ff8fa0a1d031570a8fc2c8491f476279baa56a9a31c08-filelists.xml.zck)
2019-03-20T20:15:34Z DEBUG check_transfer_statuses: Error during transfer: Curl error (16): Error in the HTTP2 framing layer for https://kojipkgs.fedoraproject.org/compose/30/Fedora-30-20190319.0/compose/Everything/armhfp/os/repodata/102ec5f532a57deab24ff8fa0a1d031570a8fc2c8491f476279baa56a9a31c08-filelists.xml.zck [Unable to resolve kojipkgs.fedoraproject.org]

I have no idea why sometimes the supplementary error is blank and other times it says it's unable to resolve kojipkgs.fedoraproject.org after it's downloaded the file. I'm going to see if I can get better error messages from librepo.

Might just try a bare 'curl --verbose' on that url in the chroot?

I had just changed librepo to add CURLOPT_VERBOSE to the download. I'm attaching its output here.
dnf-stdout.log

And, while I'm at it, here's dnf.librepo.log:
dnf.librepo.log

I'll go ahead and try running curl --verbose without any ranges and see if I hit the errors. They seem to come in clumps. There was one point where I tried maybe five or six times over a half hour and didn't hit a single HTTP2 framing error, and then hit two in a row.

I've done some more probing and this is definitely zchunk-related. It seems to be an issue with reusing curl handles when downloading the second part of the file (the data), which should work, but seems to crap out occasionally. There are also some (possibly related) speed issues, so I'll see if I can get a patch ready for librepo within the next day or two.

Thanks a lot for working on that, @jdieter .

Ok, I've tracked it down. Full details at https://bugzilla.redhat.com/show_bug.cgi?id=1690971#c6, but the framing errors are being triggered by a bug in libcurl and another in zchunk. The libcurl fix is already in Rawhide, and I've just pushed out a fix for zchunk. Any chance we could re-enable zchunk metadata generation for Rawhide only?

@jdieter Thanks for working on this. I will enable this tomorrow (if we get a rawhide compose) after zchunk is in rawhide repos.

@mohanboddu, thanks so much! That would be perfect!

Login to comment on this ticket.

Metadata
Attachments 3
Attached a year ago View Comment
Attached a year ago View Comment
Attached a year ago View Comment