ok, this is NOT the network/caching issue as before. This is different.

mbs uses mock with systemd-nspawn. This creates a new systemd 'machine' container every build. On the other arches those are cleaned up when the build is finished, but for some reason on s390x, they are not. So, they get filled up and bad things happen. ;(

# ansible -T 600 -a 'machinectl list | wc -l' -o buildvm-s390x -m shell | sort
buildvm-s390x-01.s390.fedoraproject.org | CHANGED | rc=0 | (stdout) 1
buildvm-s390x-02.s390.fedoraproject.org | CHANGED | rc=0 | (stdout) 1
buildvm-s390x-03.s390.fedoraproject.org | CHANGED | rc=0 | (stdout) 6
buildvm-s390x-04.s390.fedoraproject.org | CHANGED | rc=0 | (stdout) 5
buildvm-s390x-05.s390.fedoraproject.org | CHANGED | rc=0 | (stdout) 1954
buildvm-s390x-06.s390.fedoraproject.org | CHANGED | rc=0 | (stdout) 1150
buildvm-s390x-07.s390.fedoraproject.org | CHANGED | rc=0 | (stdout) 1
buildvm-s390x-08.s390.fedoraproject.org | CHANGED | rc=0 | (stdout) 2252
buildvm-s390x-09.s390.fedoraproject.org | CHANGED | rc=0 | (stdout) 1
buildvm-s390x-10.s390.fedoraproject.org | CHANGED | rc=0 | (stdout) 1653
buildvm-s390x-11.s390.fedoraproject.org | CHANGED | rc=0 | (stdout) 1727
buildvm-s390x-12.s390.fedoraproject.org | CHANGED | rc=0 | (stdout) 791
buildvm-s390x-13.s390.fedoraproject.org | CHANGED | rc=0 | (stdout) 1313
buildvm-s390x-14.s390.fedoraproject.org | CHANGED | rc=0 | (stdout) 315

I can clean them out, but we need to figure out whats happening and fix it.

I am not sure if it is a mbs issue, mock issue, systemd issue or something else.

Perhaps @sharkcz could look into this? Or at least point the way to what component is responsible?

I will check if I can reproduce outside koji, but it looks strange.

Can't reproduce with a local mock build on an up-to-date F-29. BTW isn't there something useful in /var/lib/mock/*/result/build.log on the builders?

koji cleans those up from the builder, so not really. ;(

I'll try and gather more info the next time this happens. I think we need to get a bunch of mbs builds before it will.

Ok, let me know then. I wonder if these are failed builds residues, but the counts lead to all builds rather ...

Just hit it again -- https://koji.fedoraproject.org/koji/taskinfo?taskID=33448932

I'm not aware of any special MBS configuration which would choose systemd-nspawn. Builds submitted by MBS should not be anyhow special.

I'm not aware of any special MBS configuration which would choose systemd-nspawn. Builds submitted by MBS should not be anyhow special.

We set some options on our tags for normal builds:
➜ ~ koji taginfo f29
Tag: f29 [3418]
Arches: None
Groups: appliance-build, build, livecd-build, livemedia-build, srpm-build
LOCKED
Required permission: 'admin'
Tag options:
mock.new_chroot : 0
mock.package_manager : 'dnf'
Inheritance:

So, perhaps one change we could do would be to set those on the tags mbs uses?

@ignatenkobrain will check for leftover machines... update in a few.

<many hours later after being sucked into all manner of fires>

They should be cleaned up now.

@kevin, MBS creates its own tags. We can add mock.new_chroot: 0 to the options MBS uses int tags creation. Tell me if we need to set this or other options please.

Yeah, if you could set that I think it would be nice. We do plan to move to new_chroot someday, but haven't yet.

Do you want me to file an issue somewhere to track that?

@kevin, you can assign it in fedora-infrastructure and set labels to factory2 and MBS and CC me there. I will try to do that on Monday next week (I think I won't manage it earlier).

We have merged MBS PR which makes the hardcoded Koji tag extra options configurable in config file, so we can in the future just configure them instead of releasing new MBS.

I plan to release new MBS today and deploy this later this week, this will fix this ticket.

Great! thanks!

Has this been deployed then?

I'm going to assume it has been deployed as I haven't seen any issues in the last week.

Please reopen if there's anything left to do here.