Hi Owen,

I see that the proposal is to make f39-flatpak-app inherit from f39, but shouldn't it inherit from f39-flatpak-runtime instead so that it uses the packages that have been rebuilt specifically for the runtime?

koji add-tag-inheritance f39-flatpak-app-packages f39-app-runtime --priority=0

Typo: I think it should be f39-flatpak-runtime instead of f39-app-runtime

They don't include packages from f39-override, to avoid those showing up in a container.

How do we override things then to make things show up in the containers? Maybe it would make sense to have a flatpak-specific override tag that makes it possible to cherry-pick packages that haven't made it to stable in bodhi yet?

I see that the proposal is to make f39-flatpak-app inherit from f39, but shouldn't it inherit from f39-flatpak-runtime instead so that it uses the packages that have been rebuilt specifically for the runtime?

Yes, the build environment for Flatpaks obviously needs flatpak-runtime-macros in addition to any such rebuilt packages! Making f39-flatpak-app tag inherit from f39-flatpak-app won't do the right thing for package lists, because of the --intransitive inheritance of f39-flatpak-runtime from f39. Instead I've edited the above to put it into the inheritance of f39-flatpak-app-build.

koji add-tag f39-flatpak-app-build --arches x86_64,aarch64,ppc64le,s390x
koji add-tag-inheritance f39-flatpak-app-build f39-flatpak-app --priority=0
koji add-tag-inheritance f39-flatpak-app-build f39-flatpak-runtime --priority=10
koji add-tag-inheritance f39-flatpak-app-build f39-build --priority=20

koji add-tag-inheritance f39-flatpak-app-packages f39-app-runtime --priority=0

Typo: I think it should be f39-flatpak-runtime instead of f39-app-runtime

Yep, thanks!

They don't include packages from f39-override, to avoid those showing up in a container.

How do we override things then to make things show up in the containers? Maybe it would make sense to have a flatpak-specific override tag that makes it possible to cherry-pick packages that haven't made it to stable in bodhi yet?

This is something that will matter only for runtimes, since for applications the f39-flatpak-app target directly has a dest_tag of f39-flatpak-app - f39-flatpak-app is as current as f39-updates-candidate.

Having f39-override in the stack when installing the runtime doesn't make sense -- we don't want to include just a few newer packages that are needed as build dependencies. Even though that's the current situation - we currently build runtimes out of the packages in f39-build!

I see three possibilities

1. Build runtimes from packages in f39-updates-candidate
2. Have an f39-flatpak-runtime-override tag that we manually manage, as you suggest
3. Only have stable packages in the runtime

My feeling is that the last is fine - I don't see a lot of value in being able to build a runtime early with testing packages, and have it sit around in testing until those packages go stable. I'd rather see us frequently (in an automated fashion!) push a new runtime when included packages have stable updates.

Maybe we can try it that way and adjust based on experience? We can always add a override tag or switch to updates-candidate later.

Automated runtime updates sound like a good idea to me! I think it would be awesome if we could automatically publish a new runtime update every day based on packages that got pushed to stable. Or maybe weekly or bi-weekly and then go through testing? Not sure.

However, in some cases we'd still need manual overrides (or alternatively the possibility to build out of a side tag, or better even - both). Consider firefox updates for example. Firefox is a good example because it's security sensitive - almost every update comes with CVE issues attached that need to get pushed out quickly.

Firefox often depends on new nss, but nss is not part of the firefox flatpak but instead part of the runtime. When firefox starts requiring new nss, then the nss team builds new nss, adds it as a buildroot override and then the firefox update is built. Both are submitted to updates-testing, and after a day or two migrate to stable.

I've tried to be quick with firefox updates and get them built on the same day as the rpm builds to avoid lagging behind with security updates. Whenever firefox has started depending on new nss, I've tried to also put out a new flatpak runtime update to include the new nss. Most of the time it's been easy because the new nss build has already been tagged as -override so it's gotten automatically included in the runtime container builds.

If we don't allow building against overridden nss then the flatpak updates are inevitably going to fall behind rpm, which I think would be unfortunate.

I think I would prefer a flatpak specific override tag though. Including everything that's in f39-override is bound to lead us into issues at some point, and I think it would make sense to make overriding an explicit action.

Another case would be if something is broken in the flatpak runtime and we have a package that fixes it. If we don't have an override system then we'd have to wait until the corresponding rpm update is pushed to stable before we can even test a flatpak runtime update fix, again leading to delays with publishing updates.

Thanks for writing that up in detail - it's really helpful! I've added a f39-flatpak-runtime-override tag to the above.

Here's everything in 'koji list-tag-inheritance' format:

     # destination tag for runtime-specific RPMs
     f39-flatpak-runtime
..I.  └─f39

     # build a runtime-specific RPM
     f39-flatpak-runtime-build
....  ├─f39-flatpak-runtime
....  └─f39-build
....     └─f39-override
....         └─f39

     # Build a runtime container
     f39-flatpak-runtime-packages
....  ├─f39-flatpak-runtime
....  └─f39-flatpak-runtime-override
....     └─f39

     # destination tag for app-specific RPMs (prefix=/app rebuilds)
     f39-flatpak-app
..I.  └─f39

     # build an app-specific RPM
     f39-flatpak-app-build
....  ├─f39-flatpak-app
....  ├─f39-flatpak-runtime
....  └─f39-build
....     └─f39-override
....         └─f39

     # Build an application container
     f39-flatpak-runtime-packages
....  └─f39-flatpak-app

Thanks for the detailed writeup on this (and for working on it in the first place ;)

Some general comments:

• We will need to add these to https://pagure.io/releng/blob/main/f/scripts/branching/make-koji-release-tags (this is what we use on branching to make all the new tags)
  (of course we still need to make the initial f39 ones now too)

• Do we want to include all arches here? (or rather to be more clear, we are building packages for all arches, but only doing the containers for aarch64/x86_64, right?). Would be be helpfull to just do those arches for the runtime/apps?

• I'm all for automated runtime rebuilding, but we do need to be careful I think pushing those too fast. Users may become annoyed if they have to update all the time (even with deltas), but thats up to the flatpak sig I suppose and bodhi is always in there too.

Otherwise it all looks reasonable to me. :)

• We will need to add these to https://pagure.io/releng/blob/main/f/scripts/branching/make-koji-release-tags (this is what we use on branching to make all the new tags)
  (of course we still need to make the initial f39 ones now too)

OK, I can eventually work on a pull request for that.

• Do we want to include all arches here? (or rather to be more clear, we are building packages for all arches, but only doing the containers for aarch64/x86_64, right?). Would be be helpfull to just do those arches for the runtime/apps?

If we switch to building Flatpaks with my Flatpak Koji plugin, then what architectures we build Flatpaks for is up to us. We might still want to restrict to aarch64/x86_64, since the use case for Flatpaks for ppc64le/s390x is slim, and faster and less load on the s390x builders is better.

I agree that the set of arches for RPM builds should be restricted to what we are going to be Flatpaks for - other architectures are still getting exercised by the standard builds of the packages.

• I'm all for automated runtime rebuilding, but we do need to be careful I think pushing those too fast. Users may become annoyed if they have to update all the time (even with deltas), but thats up to the flatpak sig I suppose and bodhi is always in there too.

Yeah. There would have to be some sort of tempo for released updates. Haven't really worked out the idea :-)

I took an initial stab at porting this over to make-koji-rlease-tags:

https://pagure.io/releng/pull-request/11512

I took an initial stab at porting this over to make-koji-rlease-tags:

https://pagure.io/releng/pull-request/11512

Thank you! I've added some comments there.

I don't think that needs to block going ahead with this. I've edited the script to have --arches=x86_64,aarch64 as discussed above. Would someone be able to to try creating the tag structure for F39?

I'd like to discuss this at the releng meeting tomorrow and make sure @humaton is all happy with it, but if so, one of us can run it tomorrow as far as I am concerned.

Hi I tried to run the script on staging but it seems that target creations points to non existing tag

koji add-target f39-flatpak-candidate f39-flatpak-container-build f39-flatpak-updates-candidate

Hi I tried to run the script on staging but it seems that target creations points to non existing tag
koji add-target f39-flatpak-candidate f39-flatpak-container-build f39-flatpak-updates-candidate

The script above explictly doesn't create the tags that were previously created by make-koji-release-tags - but these haven't created yet, since the f39 tags would not be created by that script until the f39 branch point.

If I have it correct, the necessary commands to create them are:

[ Edited to fix f39-flatpak--updates-candidate]

koji add-tag f39-flatpak
koji add-tag --parent f39-flatpak f39-flatpak-updates
koji add-tag --parent f39-flatpak-updates f39-flatpak-updates-candidate
koji add-tag --parent f39-flatpak-updates f39-flatpak-updates-testing
koji add-tag --parent f39-flatpak-updates-testing f39-flatpak-updates-testing-pending
koji add-tag --parent f39-flatpak-updates f39-flatpak-updates-pending
koji add-tag --parent f39-flatpak-updates f39-flatpak-override  

flatpak_pkgs=$(koji list-pkgs --quiet --tag="f38-flatpak" | awk '{ print $1}')
koji add-pkg --owner=releng "f39-flatpak" $flatpak_pkgs

Really need to get this done in order to move forward on the associated change proposal. Is there a time we can meet on Matrix so I can provide support in resolving any issues?

I'll try and get this this week...

ok. I ran things in stg... can you look at it and see if that looks reasonably correct?

If so, I can run it in prod and you can start testing things....

Reviewed what's in the stage - the only problem I found was an extra - in f39-flatpak--updates-candidate (was in my script in https://pagure.io/releng/issue/11479#comment-865423 -e dited to fix).

Otherwise, looks like exactly what I had in mind - I wouldn't be too surprised if we need some changes later once we start trying to use this.

Oh man, I missed the -- there too. ;(

ok, run in prod. :)

I guess we are done here?

Please re-open or file a new ticket for any further needed changes.

Filed https://pagure.io/releng/issue/11767 to fix the inheritance of -flatpak-runtime-override.