FYI, we created a mock-1.4.4 in the infra tags repo with a Epoch and nspawn disabled. However due to a lorax issue with that version, we have downgraded back to 1.3.4 for now.

This should be fixed in mock-1.4.6, though you'll still need to patch to switch back to chroots by default...

Would it be possible to use --bind-ro= for /run/pesign? Just stick it in config_opts['nspawn_args'] on the builders?

And for loop devices, precreate the device nodes on the host, and add something like --bind=/dev/loop-control --bind=/dev/loop{0,1,2,3,4,5,6,7,8,9}? This seems to work well enough (mkfs, mount -o bind, etc.), as long as user-namespaces (-U) are not used.

Yes, the loop issue was fixed in 1.4.6.

I don't know if bind-ro will work for pesign.

I also know appliance image creation doesn't work in nspawn, and some of the runroot stuff I suspect also will not work. I suppose we need someone to go through all the various tasks that a compose fires off and confirm that they work or should work in nspawn. Sadly a pretty daunting task.

@kevin , are these issues still open and if so, do you still want them fixed? Please advise.

Yep. We will need to fix them as yum3 is going to go away in the f30 timeframe most likely, and that means we have to update mock to the latest where it no longer has a dependency on yum.

So yes, we need to start figuring out how we can get the latest mock working for our use cases.

Some of this we can try in staging, or some of it people can just try at home and give us more info on.
Perhaps we can make a list of the various jobs a compose fires off and try and do each of those using the latest mock and file bugs as we go?

@kevin I believe @davidlt has already been doing this for Fedora RISC-V.

This ticket should be moved to a different queue since we do not have knowledge to complete this ticket.

Assigning to @kevin .

I think it would be good to try upgrading mock in staging. All the issues with nspawn that I was aware of were fixed. runroot can be forced to run with old chroot.

Yep. I agree... if you have a chance to do this (perhaps after the next koji sync?) that would be great, or I can try and do so.

We will need to check kernel builds also (but that will only be possible in prod).

@mohanboddu reports that we are still using mock-1.3.4 in the buildroot.

Currently mock-1.4.13-1.fc28.noarch is installed on all staging builders. Both scratch build from SCM and simple runroot succeeded.

@mizdebsk, does mock-1.4.13 use chroot or systemd-nspawn?

It uses nspawn by default, but it is possible to configure it to use old chroot. Fedora Koji doesn't configure this, so nspawn is used.

@mizdebsk, thanks.

Once this deployed to prod and if everything works fine, then we will close this ticket.

BTW upgrading to new mock and using systemd-nspawn can be threated independently. You can configure mock to use old-chroot while using the new version and change the configuration only when all issues with nspawn are resolved.

Yeah, I think we should do them independently... I'm planning on upgrading prod this week and will keep them on old-chroot, then we can switch that and start fixing things as we hit them. Or revert back to old-chroot if needed.

Status update: All the builders are moved to the current mock (with old-chroot), except for all the armv7 builders (they are stalled due to https://bugzilla.redhat.com/show_bug.cgi?id=1576593 )

From our grooming discussion on #fedora-releng channel on Apr 12 2019

update that we will try and switch it after f30 when we have time to debug issues

@mohanboddu reports that we are still using chroot in mock. Will talk to @kevin to see when will make the change.

FYI, the old chroot will not go away. See https://github.com/rpm-software-management/mock/issues/331
So unless you need/want better isolation, you can stay where you are.

From RelEng meeting:

#info We will ask the side tag requestors if they are okay with having systemd-nspawn in the side tag which will help us in testing the switch

This is now live in rawhide.

We hit mock/pesign issues, but worked around that.

Then we hit ostree issues, still being worked on.

We need to also update docs to no longer set this on new releases.

Looks like we worked around the ostree-installer issues (made a mock config for the runroot builders that passed cap_mac_admin in for it to work).
(see https://bugzilla.redhat.com/show_bug.cgi?id=2123812)

So, we are down to just updating docs. ;)

The docs is now merged closing.