#11735 imagefactory built container images lack hardlinks
Closed: Fixed 7 months ago by petersen. Opened 7 months ago by petersen.

  • Describe the issue

https://github.com/redhat-imaging/imagefactory/issues/412 (from 2017)

This is actually an old issue, but it seems the impact on the base image is small...
but for fedora-toolbox it is huge (the image is more than 3 times bigger now).

  • When do you need this? (YYYY/MM/DD)

Soon would be wonderful

  • If we cannot complete your request, what is the impact?

fedora-toolbox:39+ images are 5.5GB large.

@otaylor has a proposed one-line patch/workaround in the upstream issue.
Could that be tested for Rawhide?


We'd just need the package updated in Fedora with that patch and it deployed to infra (which would need a freeze break request for release).

We'd just need the package updated in Fedora with that patch and it deployed to infra (which would need a freeze break request for release).

Yes - unlike livemedia-creator, imagefactory runs on the builder node, not inside a buildroot, so it would require an infrastructure FBR, a F38 build tagged into the infrastructure tag, and a restart of all the builders.

While I failed to actually get imagefactory working locally after a few more hours of trying, I did some alternate testing and turned my pseudo-patch into a PR.

Metadata Update from @phsmoura:
- Issue tagged with: medium-gain, medium-trouble, ops

7 months ago

@otaylor has a proposed one-line patch/workaround in the upstream issue.
Could that be tested for Rawhide?

Built imagefactory-1.1.16-7.fc40 for Rawhide with the patch from @otaylor :
https://src.fedoraproject.org/rpms/imagefactory/c/28c6a561b78dacdc6dc9e00240ea4a
https://koji.fedoraproject.org/koji/taskinfo?taskID=108105756

We'd just need the package updated in Fedora with that patch and it deployed to infra (which would need a freeze break request for release).

Built imagefactory-1.1.16-7.fc39 for F39 with the patch from @otaylor :
https://bodhi.fedoraproject.org/updates/FEDORA-2023-a8871574f4
https://src.fedoraproject.org/rpms/imagefactory/c/28c6a561b78dacdc6dc9e00240ea4a
https://koji.fedoraproject.org/koji/taskinfo?taskID=108105779

We already have a Final Freeze exception but the bug was filed against fedora-toolbox for the lack of a better alternative.

What's the next step?

We actually need a f38 build. The builders are fedora 38. :)

I can just build it in infra tags. We will need a infra freeze exception.

@adamwill @pbrobinson can you +1?

Got a +1 from pbrobinson on chat.

Builders updated now. Fingers all crossed. ;)

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

7 months ago

We actually need a f38 build. The builders are fedora 38. :)

I can just build it in infra tags. We will need a infra freeze exception.

Thanks for taking care of the build!

OK, so that didn't work - the image build this morning still has broken hardlinks. The problem is that the code that needs to be changed here is in imagefactory-plugins-Docker, and there are two separate dist-git repos / SRPMs 'imagefactory' and 'imagefactory-plugins' that use the same upstream source code. So while the patch applied to the imagefactory distgit, it didn't actually do anything.

re-opening for re-updating the builders.

Metadata Update from @adamwill:
- Issue status updated to: Open (was: Closed)

7 months ago

ok. Updated builders and restarted kojid. :)

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

7 months ago

I am afraid the size still seems unchanged today

Metadata Update from @petersen:
- Issue status updated to: Open (was: Closed)

7 months ago

Turned out that in the last attempt, we didn't upgrade imagefactory-plugins-Docker just imagefactory-plugins. @kevin updated the latter yester,day, and the build last night looks good:

$ du -hs Fedora-Container-Toolbox-39-202310*
595M    Fedora-Container-Toolbox-39-20231019.n.0.x86_64.tar.xz
187M    Fedora-Container-Toolbox-39-20231028.n.0.x86_64.tar.xz

$ tar tvf 2ae30563affb13258b6b99a81a7f93884ac036a0e03afaf3abaefbbd2c939ec5/layer.tar | grep i915_dri.so
hrwxr-xr-x root/root         0 2023-10-04 20:00 ./usr/lib64/dri/i915_dri.so link to ./usr/lib64/dri/imx-drm_dri.so

Wonderful, looks good here too, thank you everyone!

$ podman pull fedora-toolbox:39
Trying to pull registry.fedoraproject.org/fedora-toolbox:39...
Getting image source signatures
Copying blob 20112aa9dde1 [===============>----------------------] 142.2MiB / 342.8MiB | 361.1 MiB/s
^C
$ podman pull fedora-toolbox:40
Trying to pull registry.fedoraproject.org/fedora-toolbox:40...
Getting image source signatures
Copying blob 77522bf8d121 [>-------------------------------------] 9.5MiB / 345.3MiB | 24.3 MiB/s
^C

Metadata Update from @petersen:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

7 months ago

The fedora base images also seem about 9MB smaller now on disk, yay :-) (192M -> 183M)

I was doing some scratch builds of the fedora-toolbox:40 OCI image recently, and found that in some builds the hard links are absent from %{_libdir}/dri and %{_prefix}/lib/locale/locale-archive*, and hence those images are a lot bigger than expected. eg., compare these two:
https://koji.fedoraproject.org/koji/taskinfo?taskID=108340937
https://koji.fedoraproject.org/koji/taskinfo?taskID=108344786

The first tarball is a lot bigger than the second.

I was doing some scratch builds of the fedora-toolbox:40 OCI image recently, and found that in some builds the hard links are absent from %{_libdir}/dri and %{_prefix}/lib/locale/locale-archive*, and hence those images are a lot bigger than expected. eg., compare these two:
https://koji.fedoraproject.org/koji/taskinfo?taskID=108340937
https://koji.fedoraproject.org/koji/taskinfo?taskID=108344786

The first tarball is a lot bigger than the second.

Kevin confirmed that he found 3 x86_64 builders that still had a buggy Image Factory.

Login to comment on this ticket.

Metadata
Boards 1
Ops Status: Backlog