#1988 kojid should also cleanup any 'bootstrap' repos
Closed: Fixed 2 years ago by mikem. Opened 2 years ago by jcpunk.

It appears that kojid is not cleaning up any of the bootstrap repos generated by mock's bootstrap chroot abilities.


How are you using bootstrap caps? We're not supporting it at all for now (it is more a future RFE).

Metadata Update from @tkopecek:
- Custom field Size adjusted to None

2 years ago

My koji builders are EL7, but I'm building Fedora and EPEL8 packages.

I'm just setting config_opts['use_bootstrap_container'] = True in site-defaults.

With Mock v2 using bootstrap containers by default this is probably a bit more interesting....

Metadata Update from @tkopecek:
- Issue set to the milestone: 1.22

2 years ago

Yep, looks we will not evade this bullet :-)

We are hitting this issue in production. Over the past week we've filled up our drive three times. Twice after growing it to 250G.

Is there any production workarounds until this issue is resolved?

@praiskup Is there some mock option, which deletes the bootstrap data?

I was able to work-around it by adding: config_opts['use_bootstrap'] = False
to my /etc/mock/site-defaults.cfg.

Only command-line option, mock --scrub bootstrap.

I mean, bootstrap is sort of cache - when cleanup is requested, it takes non-trivial
time to reinitialize it in the next mock command. So it should be cleaned only when
really needed. Since it wastes a lot of space for you, I suppose you are using
something like --uniqueext? There's an upstream question which I'm not sure about ATM.

@praiskup - separate bootstrap dir is created for every build in koji (as separete buildroot per task is created), so space is easily exhausted.

And one more question: Can I expect, that boostrap dir is always /mock_dir/buildroot-bootstrap if /mock_dir/buildroot is buildroot directory?

separete buildroot per task is created

How do you clean the normal buildroot then? Using --scrub=all would work for you
(cleans everything, including bootstrap, and dnf caches).

Can I expect, that boostrap dir is always /mock_dir/buildroot-bootstrap if /mock_dir/buildroot is buildroot directory?

yes

@praiskup There is more complicated policy for that. If build is successful, then after two minutes we delete contents of rootdir. After another 24 hours we delete rootdir itself + mock config. There are some issues with hanging processes and interaction with rpmlib. For failed buildroots there is more complicated policy to have them long enough for debugging. So, it is being done manually outside of mock.

Anyway, at least final step can be done via mock scrub=all + deleting config.

Related code starts here: https://pagure.io/koji/blob/master/f/koji/daemon.py#_727

There are some issues with hanging processes and interaction with rpmlib.

This will be fixed in mock v2.2, merged just
today
.
Well, hanging processes were killed by mock for some time - but only those
processes that were started by rpmbuild process (not those started by
packages' scriptlets from rpm --root -i process).

For failed buildroots there is more complicated policy

Ok, in this case you indeed want --scrub bootstrap.

Nevermind, if you miss some functionality/tweak/fix that would help brewkoji here, fill RFE.

Metadata Update from @jcupova:
- Issue tagged with: testing-ready

2 years ago

Metadata Update from @mfilip:
- Issue tagged with: testing-done

2 years ago

@tkopecek I'm curious if we can rely on mock's clean/scrub behavior totally in koji

Yes, but not for the version in epel6. When we drop rhel6 support we can drop also this complicated cleaning code.

Metadata Update from @julian8628:
- Issue status updated to: Open (was: Closed)

2 years ago

Metadata Update from @julian8628:
- Issue untagged with: testing-done

2 years ago

Metadata Update from @jcupova:
- Issue tagged with: testing-done

2 years ago

Login to comment on this ticket.

Metadata
Related Pull Requests
  • #2372 Merged 2 years ago
  • #2212 Merged 2 years ago