#9532 tar errors out when creating hard links (aarch64 flatpak issue)
Closed: Fixed 3 years ago by kevin. Opened 3 years ago by kalev.

I ran into an issue with aarch64 flatpaks where tar is erroring out when creating hard links. The koji task is https://koji.fedoraproject.org/koji/taskinfo?taskID=57503246 and the matching x86_64 build went fine. aarch64 reproducibly errors out with:

tar: files/lib/python3.9/site-packages/hydrapaper/__pycache__/__main__.cpython-39.pyc: Cannot hard link to ‘var/tmp/flatpak-build/app/lib/python3.9/site-packages/hydrapaper/__pycache__/__main__.cpython-39.opt-1.pyc’: No such file or directory
tar: var/tmp/flatpak-build/app/lib/python3.9/site-packages/hydrapaper/__pycache__/app_window.cpython-39.pyc: Cannot hard link to ‘var/tmp/flatpak-build/app/lib/python3.9/site-packages/hydrapaper/__pycache__/app_window.cpython-39.opt-1.pyc’: No such file or directory
tar: var/tmp/flatpak-build/app/lib/python3.9/site-packages/hydrapaper/__pycache__/apply_wallpapers.cpython-39.pyc: Cannot hard link to ‘var/tmp/flatpak-build/app/lib/python3.9/site-packages/hydrapaper/__pycache__/apply_wallpapers.cpython-39.opt-1.pyc’: No such file or directory
tar: var/tmp/flatpak-build/app/lib/python3.9/site-packages/hydrapaper/__pycache__/confManager.cpython-39.pyc: Cannot hard link to ‘var/tmp/flatpak-build/app/lib/python3.9/site-packages/hydrapaper/__pycache__/confManager.cpython-39.opt-1.pyc’: No such file or directory
tar: var/tmp/flatpak-build/app/lib/python3.9/site-packages/hydrapaper/__pycache__/get_desktop_environment.cpython-39.pyc: Cannot hard link to ‘var/tmp/flatpak-build/app/lib/python3.9/site-packages/hydrapaper/__pycache__/get_desktop_environment.cpython-39.opt-1.pyc’: No such file or directory
tar: var/tmp/flatpak-build/app/lib/python3.9/site-packages/hydrapaper/__pycache__/headerbar.cpython-39.pyc: Cannot hard link to ‘var/tmp/flatpak-build/app/lib/python3.9/site-packages/hydrapaper/__pycache__/headerbar.cpython-39.opt-1.pyc’: No such file or directory
tar: files/lib/python3.9/site-packages/hydrapaper/__pycache__/is_image.cpython-39.pyc: Cannot hard link to ‘var/tmp/flatpak-build/app/lib/python3.9/site-packages/hydrapaper/__pycache__/is_image.cpython-39.opt-1.pyc’: No such file or directory
tar: var/tmp/flatpak-build/app/lib/python3.9/site-packages/hydrapaper/__pycache__/listbox_helper.cpython-39.pyc: Cannot hard link to ‘var/tmp/flatpak-build/app/lib/python3.9/site-packages/hydrapaper/__pycache__/listbox_helper.cpython-39.opt-1.pyc’: No such file or directory
tar: var/tmp/flatpak-build/app/lib/python3.9/site-packages/hydrapaper/__pycache__/main_stack.cpython-39.pyc: Cannot hard link to ‘var/tmp/flatpak-build/app/lib/python3.9/site-packages/hydrapaper/__pycache__/main_stack.cpython-39.opt-1.pyc’: No such file or directory
tar: var/tmp/flatpak-build/app/lib/python3.9/site-packages/hydrapaper/__pycache__/monitor_parser.cpython-39.pyc: Cannot hard link to ‘var/tmp/flatpak-build/app/lib/python3.9/site-packages/hydrapaper/__pycache__/monitor_parser.cpython-39.opt-1.pyc’: No such file or directory
tar: var/tmp/flatpak-build/app/lib/python3.9/site-packages/hydrapaper/__pycache__/monitors_flowbox.cpython-39.pyc: Cannot hard link to ‘var/tmp/flatpak-build/app/lib/python3.9/site-packages/hydrapaper/__pycache__/monitors_flowbox.cpython-39.opt-1.pyc’: No such file or directory
tar: var/tmp/flatpak-build/app/lib/python3.9/site-packages/hydrapaper/__pycache__/settings_box.cpython-39.pyc: Cannot hard link to ‘var/tmp/flatpak-build/app/lib/python3.9/site-packages/hydrapaper/__pycache__/settings_box.cpython-39.opt-1.pyc’: No such file or directory
tar: var/tmp/flatpak-build/app/lib/python3.9/site-packages/hydrapaper/__pycache__/singleton.cpython-39.pyc: Cannot hard link to ‘var/tmp/flatpak-build/app/lib/python3.9/site-packages/hydrapaper/__pycache__/singleton.cpython-39.opt-1.pyc’: No such file or directory
tar: var/tmp/flatpak-build/app/lib/python3.9/site-packages/hydrapaper/__pycache__/threading_helper.cpython-39.pyc: Cannot hard link to ‘var/tmp/flatpak-build/app/lib/python3.9/site-packages/hydrapaper/__pycache__/threading_helper.cpython-39.opt-1.pyc’: No such file or directory
tar: var/tmp/flatpak-build/app/lib/python3.9/site-packages/hydrapaper/__pycache__/wallpaper_flowbox_item.cpython-39.pyc: Cannot hard link to ‘var/tmp/flatpak-build/app/lib/python3.9/site-packages/hydrapaper/__pycache__/wallpaper_flowbox_item.cpython-39.opt-1.pyc’: No such file or directory
tar: var/tmp/flatpak-build/app/lib/python3.9/site-packages/hydrapaper/__pycache__/wallpaper_merger.cpython-39.pyc: Cannot hard link to ‘var/tmp/flatpak-build/app/lib/python3.9/site-packages/hydrapaper/__pycache__/wallpaper_merger.cpython-39.opt-1.pyc’: No such file or directory
tar: var/tmp/flatpak-build/app/lib/python3.9/site-packages/hydrapaper/__pycache__/wallpapers_flowbox.cpython-39.pyc: Cannot hard link to ‘var/tmp/flatpak-build/app/lib/python3.9/site-packages/hydrapaper/__pycache__/wallpapers_flowbox.cpython-39.opt-1.pyc’: No such file or directory
tar: var/tmp/flatpak-build/app/lib/python3.9/site-packages/hydrapaper/__pycache__/wallpapers_folder_listbox_row.cpython-39.pyc: Cannot hard link to ‘var/tmp/flatpak-build/app/lib/python3.9/site-packages/hydrapaper/__pycache__/wallpapers_folder_listbox_row.cpython-39.opt-1.pyc’: No such file or directory
tar: var/tmp/flatpak-build/app/lib/python3.9/site-packages/hydrapaper/__pycache__/wallpapers_folders_view.cpython-39.pyc: Cannot hard link to ‘var/tmp/flatpak-build/app/lib/python3.9/site-packages/hydrapaper/__pycache__/wallpapers_folders_view.cpython-39.opt-1.pyc’: No such file or directory
tar: Exiting with failure status due to previous errors
2020-12-15 11:28:36,608 - atomic_reactor.plugin - DEBUG - Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/atomic_reactor/plugin.py", line 268, in run
    plugin_response = plugin_instance.run()
  File "/usr/lib/python3.9/site-packages/atomic_reactor/plugins/prepub_flatpak_create_oci.py", line 116, in run
    ref_name, outfile, tarred_outfile = self.builder.build_container(tarred_filesystem)
  File "/usr/lib/python3.9/site-packages/flatpak_module_tools/flatpak_builder.py", line 882, in build_container
    ref_name = self._create_app_oci(tarred_filesystem, outfile)
  File "/usr/lib/python3.9/site-packages/flatpak_module_tools/flatpak_builder.py", line 736, in _create_app_oci
    subprocess.check_call(['tar', 'xCfz', builddir, tarred_filesystem])
  File "/usr/lib64/python3.9/subprocess.py", line 373, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['tar', 'xCfz', '/tmp/tmpideyjdf6/build', '/tmp/tmpideyjdf6/filesystem.tar.gz']' returned non-zero exit status 2.

I am not sure what's going on: could it be something to do with how the file system is mounted that causes tar to error out? Is it a file system that doesn't support hard links (or doesn't support hard links that point to a different directory)?


Metadata Update from @mobrien:
- Issue assigned to mobrien

3 years ago

I don't know enough about how OSBS (Open Shift Build System) composes flatpacks for any architecture. @otaylor can you give some pointed on what to look at?

Metadata Update from @smooge:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: OSBS, medium-gain, medium-trouble, ops, releng

3 years ago

I'm just going to have a look at this now to see if it was caused by the installation but my knowledge of flatpaks is limited so any assistance is appreciated

Thanks! I'm not very knowledgeable how atomic reactor works, but when looking through the two logs (x86_64.log that succeeded and aarch64.log that failed) it stands out that the failed one says that docker uses:

"Driver": "overlay2",

and the one that succeeds says:

Driver": "devicemapper",

I don't know if that's what causing the issue, but might be related? Could it be that overlayfs doesn't support all the hard link features that Fedora rpm contents uses?

@kalev that is likely the issue, we came to a similar conclusion independently. Not sure if this is something we can fix easily. I'll do some investigation

There is a reasonably high amount of work involved in changing it so I wouldn't be able to promise the change anytime soon. Its something that would need to be discussed as it would probably need an outage as well

Fair enough, thanks. I'll do some tests in the mean time and try to figure out how many flatpaks are affected.

I've found two more flatpaks that fail the same way: gnome-music and gedit. That is out of ~30 I've tried to build so far. It should be easy enough to work this around in individual flatpak container.yaml by adding

platforms:
    only: x86_64

... so it's not super urgent to get the infra side fixed :)

Thanks for looking into this @kalev hopefully we will get it resolved in the new year

Talking about this with kalev, we figured out what is going wrong - it's a question of how flatpak-module-tools manipulates python tarfile.TarInfo headers when rewriting paths. If a path goes from > 100 to < 100 bytes, then the member.name field is changed, but member.pax_headers is left untouched and ends up overriding the name.

I'm not sure yet why we're seeing this:
- Only with python-3.9 ?
- In some cases only on aarch64 ?

But I do know why it it's not happening with local builds - the paths we rewrite in that case are different and shorter. In any case, I'll come up with a fixed version of flatpak-module-tools and that should resolve this - nothing to do with filesystem drivers :-)

Only with python-3.9

Probably this is actually "only with Python-3.8 and newer" - in Python-3.8, the default format for the tarfile module was changed from GNU_FORMAT to PAX_FORMAT

OK, fixed the bug upstream, built a new version for Fedora, submitted f32/f33 updates:

https://bodhi.fedoraproject.org/updates/FEDORA-2020-c4773405c8

When you have a chance, please update the buildroot to contain this python3-flatpak-module-tools - hopefully that will resolve this issue.

@otaylor I've updated the buildroot container on the aarch64 builders with the updated version of python3-flatpak-module-tools. Let me know if it solves the issue.

I've updated the buildroot container on the aarch64 builders with the updated version of python3-flatpak-module-tools.

https://koji.fedoraproject.org/koji/buildinfo?buildID=1660933 completed successfully on aarch64, so I think we can consider this fixed.

Could we get this updated for x86_64 as well? #9538 and #9542 are manifestations of the same issue on x86_64.

Ok great! I have added the change to the x86_64 nodes now too.

Metadata Update from @mobrien:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

3 years ago

Issue status updated to: Open (was: Closed)

3 years ago

Issue status updated to: Closed (was: Open)
Issue close_status updated to: Fixed

3 years ago

Login to comment on this ticket.

Metadata
Boards 1
ops Status: Done