#333 improve --libvirt default experience
Closed: Fixed None Opened 8 years ago by kparal.

I already opened an audit, but with @jskladan we found out that Phab does not really notify anyone about it nor display it on the homepage, so I'm creating a ticket. The default --libvirt experience is now subpar, the details are provided here https://phab.qadevel.cloud.fedoraproject.org/rLTRN3ba67bd98cec29303bb10d2ea873ce044318f962 .

Please improve all of that or at least some of that, but let's do it quickly, so that people can have improved experience soon.

I talked to @tflink and he uploaded a fresh image to https://tflink.fedorapeople.org/taskotron/taskotron-cloud/images/ (I also asked him now to delete the old images), so at least the hosting part should be "good enough" right now.


This ticket had assigned some Differential requests:
D775

The only objection I have to making the raw images that we build available publicly is the filesize - they're 3G of mostly empty data right now.

One approach would be to sync compressed versions with qadevel (soon to be qa.fp.o) and host gzipped images that way.

or we could convert to qcow2, and strip the empty space using something like virt-sparsify or just the plain qemu img-convert could do it IIRC

I did a small test with the image Tim uploaded here:
https://tflink.fedorapeople.org/taskotron/taskotron-cloud/images/

I used these commands:

$ virt-sparsify --format raw --convert raw 160301_0000-fedora-23-taskotron_cloud-x86_64.img sparsified.img
$ virt-sparsify --format raw --convert qcow2 160301_0000-fedora-23-taskotron_cloud-x86_64.img sparsified.qcow2
$ qemu-img convert -f raw -O qcow2 160301_0000-fedora-23-taskotron_cloud-x86_64.img 160301_0000-fedora-23-taskotron_cloud-x86_64.qcow2
$ gzip -k <files>

Here's how the images look on disk:

$ du -h *
3,1G    160301_0000-fedora-23-taskotron_cloud-x86_64.img
341M    160301_0000-fedora-23-taskotron_cloud-x86_64.img.gz
797M    160301_0000-fedora-23-taskotron_cloud-x86_64.qcow2
339M    160301_0000-fedora-23-taskotron_cloud-x86_64.qcow2.gz
770M    sparsified.img
337M    sparsified.img.gz
785M    sparsified.qcow2
334M    sparsified.qcow2.gz

But since http can't transfer sparse files, these are the real transfer sizes:

$ ls -lh
total 6,7G
-rw-rw-r--. 1 kparal kparal 3.0G Mar 11 17:14 160301_0000-fedora-23-taskotron_cloud-x86_64.img
-rw-rw-r--. 1 kparal kparal 341M Mar 11 17:14 160301_0000-fedora-23-taskotron_cloud-x86_64.img.gz
-rw-r--r--. 1 kparal kparal 797M Mar 11 17:20 160301_0000-fedora-23-taskotron_cloud-x86_64.qcow2
-rw-r--r--. 1 kparal kparal 339M Mar 11 17:20 160301_0000-fedora-23-taskotron_cloud-x86_64.qcow2.gz
-rw-r--r--. 1 kparal kparal 3.0G Mar 11 17:17 sparsified.img
-rw-r--r--. 1 kparal kparal 337M Mar 11 17:17 sparsified.img.gz
-rw-r--r--. 1 kparal kparal 785M Mar 11 17:17 sparsified.qcow2
-rw-r--r--. 1 kparal kparal 334M Mar 11 17:17 sparsified.qcow2.gz

As you can see, for bandwidth purposes, it doesn't really matter which format we use, as long as we compress it. For disk storage purposes, the default raw image is the worst choice, and both sparsified raw image and any qcow2 are substantially better. The remaining question is performance of the client, that I haven't tested and would be definitely interesting to know.

The only "problem" there is, that we'd need to add "uncompress the gz file you just got from the image_url" functionality...

! In #739#10117, @jskladan wrote:
The only "problem" there is, that we'd need to add "uncompress the gz file you just got from the image_url" functionality...

To be clear, I want to support just the manual downloading approach in the beginning. So:
1. ImageFactory creates images every day
2. Some script distributes these images to our build slaves
3. The same script compresses the images and distributes them to our image hosting machine (qa.fp.o?)
4. Users look at that directory from time to time, download latest image(s), puts them to /var/lib/libtaskotron/images
5. libtaskotron automatically finds them (if they have configures force_imageurl=False) and uses the latest

I now realize what you meant, you wanted to use our hosted image urls directly in the imageurl, which means testcloud would need to learn uncompressing images if needed. Well, that's a valid use case, and a reasonable one. I think we can start by not supporting it in the beginning. Once we have other pieces of the puzzle in place, we can then decide whether we're fine with just manual downloading for the moment, or whether we want to teach testcloud to uncompress images or write our custom, more intelligent downloader code (which could recognize and download the most recent image automatically, for example, without specifying imageurl at all).

I'm not touching testcloud with a ten-feet-long stick, if it's just about the use-case described, then there are no objections from me at all

Login to comment on this ticket.

Metadata