PR#9: FlatpakBuilder: Add xa.metadata as ostree commit metadata for runtimes - flatpak-module-tools

flatpak-module-tools

#9 FlatpakBuilder: Add xa.metadata as ostree commit metadata for runtimes

Merged 4 years ago by otaylor. Opened 4 years ago by kalev.

kalev/flatpak-module-tools runtime-add-xa-metadata into master

FlatpakBuilder: Add xa.metadata as ostree commit metadata for runtimes

Kalev Lember • 4 years ago

1197302

file modified

		`@@ -678,6 +678,9 @@`
		`# Run flatpak-build-finish to add extra metadata, based on finish-args`
		`self._build_finish(builddir)`

		`+ with open(os.path.join(builddir, 'metadata'), 'r') as f:`
		`+ metadata = f.read()`
otaylor commented 4 years ago I checked the full set of metadata for all Flatpaks and runtimes on my system and they are all ascii-only (and almost everything in the metadata looks inherently ascii only) - so this is probably fine - for Python2, it will give a byte string, and for Python3, it will interpret the data according to the locale's encoding.
		`+`
		`runtime_ref = 'runtime/{id}/{arch}/{branch}'.format(**args)`

		`subprocess.check_call(['ostree', 'commit',`
		`@@ -685,6 +688,7 @@`
		`'--owner-gid=0', '--no-xattrs',`
		`'--canonical-permissions',`
		`'--branch', runtime_ref,`
		`+ '--add-metadata-string', 'xa.metadata=' + metadata,`
otaylor commented 4 years ago And in Python3, this is going to reencode according to the locale encoding - which is what the GLib command line processing expects. So, as long as metadata is UTF-8 and the locale encoding is UTF-8., it will even handle non-ASCII fine, though we only need to do with ASCII.
		`'-s', 'build of ' + runtime_ref,`
		`'--tree=tar=' + tarred_filesystem,`
		`'--tree=dir=' + builddir])`

kalev commented 4 years ago

Newer versions of flatpak (after https://github.com/flatpak/flatpak/commit/116a8b848788c582fb2c8577f1250438b100da25)
verify that the metadata matches ostree commit xa.metadata key. As we
use ostree directly for creating runtimes, we need to add the key
manually here.

Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1822753

kalev commented 4 years ago

Note that I haven't tested this. Pretty sure there's a syntax error or something hiding as I don't usually write python :) Let me figure out how to test this before merging.

alexl commented 4 years ago

The argument construction look ok to me, assuming nothing in python unescapes anything.
However, why read the file when we just wrote it?

alexl commented 4 years ago

Oh, i see, build-finish appends to it.

kalev commented 4 years ago

Yes, exactly, we need to call out to build-finish first to get the final metadata file.

rebased onto 1197302

4 years ago

kalev commented 4 years ago

OK, the patch did have a syntax error which I've now fixed. Seems to be working in my testing; at least I got the metadata back with:

` ``
$ ostree show --repo . --print-metadata-key=xa.metadata runtime/org.fedoraproject.Platform/x86_64/f32
'[Runtime]\nname=org.fedoraproject.Platform\nruntime=org.fedoraproject.Platform/x86_64/f32\nsdk=org.fedoraproject.Sdk/x86_64/f32\n\n[Environment]\nGI_TYPELIB_PATH=/app/lib64/girepository-1.0\nGST_PLUGIN_SYSTEM_PATH=/app/lib64/gstreamer-1.0:/usr/lib64/gstreamer-1.0\nLD_LIBRARY_PATH=/app/lib64\nXDG_DATA_DIRS=/app/share:/usr/share:/usr/share/runtime/share:/run/host/share\n'
```

... which seems right to me.

@jkonecny Can you test installing http://kalev.fedorapeople.org/flatpak-runtime-f32-3220200414150125.oci.tar.gz with rawhide flatpak and see if it fixes https://bugzilla.redhat.com/show_bug.cgi?id=1822753 ?

jkonecny commented 4 years ago

I've tested this on installed Rawhide machine and in a Rawhide mock.

Both I'm getting this result:

[anaconda@dhcp54 ~]$ flatpak-module install http://kalev.fedorapeople.org/flatpak-runtime-f32-3220200414150125.oci.tar.gz
2020-04-22 12:16:42,623 - MainThread - moksha.hub - WARNING - Cannot find qpid python module. Make sure you have python-qpid installed.
2020-04-22 12:16:42,807 - MainThread - MBS.utils.greenwave - WARNING - Greenwave is not configured or configured improperly
INSTALLING
==========
running: ostree init --mode=archive-z2 --repo /home/anaconda/.local/share/flatpak-module-tools/repo
running: flatpak build-update-repo /home/anaconda/.local/share/flatpak-module-tools/repo
Updating appstream branch
Updating summary
running: flatpak remote-add --user --no-gpg-verify flatpak-module-tools /home/anaconda/.local/share/flatpak-module-tools/repo

Note that the directories

'/var/lib/flatpak/exports/share'
'/home/anaconda/.local/share/flatpak/exports/share'

are not in the search path set by the XDG_DATA_DIRS environment variable, so
applications installed by Flatpak may not appear on your desktop until the
session is restarted.

Downloading image  [####################################]  100%
running: tar xfa /tmp/tmpn8nobjv3/flatpak-runtime-f32-3220200414150125.oci.tar.gz
running: flatpak build-import-bundle --update-appstream --oci /home/anaconda/.local/share/flatpak-module-tools/repo /tmp/tmpn8nobjv3/oci
error: Wrong type for string member release
error: flatpak failed (exit status=1)

Based on discussion with @kalev it seems to be problem with how the runtime is build not with the fix.

We've decided to test this in Rawhide directly because the runtime is already broken and it will be much, much easier to test it there.

otaylor commented on line 6 of flatpak_module_tools/flatpak_builder.py 4 years ago

I checked the full set of metadata for all Flatpaks and runtimes on my system and they are all ascii-only (and almost everything in the metadata looks inherently ascii only) - so this is probably fine - for Python2, it will give a byte string, and for Python3, it will interpret the data according to the locale's encoding.

otaylor commented on line 15 of flatpak_module_tools/flatpak_builder.py 4 years ago

And in Python3, this is going to reencode according to the locale encoding - which is what the GLib command line processing expects. So, as long as metadata is UTF-8 and the locale encoding is UTF-8., it will even handle non-ASCII fine, though we only need to do with ASCII.