Openshift in staging is no longer building anything, this is the error it is returning:
error: build error: devmapper: Thin Pool has 11561 free data blocks which is less than minimum required 11944 free data blocks. Create more free space in thin pool or use dm.min_free_space option to change behavior
URL: https://os.stg.fedoraproject.org/console/project/monitor-gating/browse/builds/monitor-gating-build/monitor-gating-build-40?tab=logs
So, looking at space on nodes with:
os-node03.stg.phx2.fedoraproject.org | CHANGED | rc=0 >> LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert root GuestVolGroup00 -wi-ao---- <58.59g docker-pool vg-docker twi-a-t--- 58.32g 42.00 18.44
os-node02.stg.phx2.fedoraproject.org | CHANGED | rc=0 >> LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert root GuestVolGroup00 -wi-ao---- <58.59g docker-pool vg-docker twi-a-t--- <48.60g 32.37 14.81
os-node01.stg.phx2.fedoraproject.org | CHANGED | rc=0 >> LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert root GuestVolGroup00 -wi-ao---- <58.59g docker-pool vg-docker twi-a-t--- 58.32g 40.75 17.38
os-node04.stg.phx2.fedoraproject.org | CHANGED | rc=0 >> LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert root GuestVolGroup00 -wi-ao---- <58.59g docker-pool vg-docker twi-a-t--- 58.32g 90.32 28.35
You can see os-node04 is at 90%. ;(
We setup a weekly cron job to remove docker images that aren't needed anymore, but it seems that there was a ton of builds from compose-tracker and others. I ran the weekly cron manually:
[root@os-node04 ~][STG]# docker rmi $(docker images --filter dangling=true -q)
[root@os-node04 ~][STG]# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert root GuestVolGroup00 -wi-ao---- <58.59g docker-pool vg-docker twi-a-t--- 58.32g 55.77 21.17
So, possibly we should run the cron more often, or perhaps we should just increase space. I guess we can wait and see if it happens again...
Metadata Update from @kevin: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
we set up compose-tracker to be rebuilt based on webhooks from pagure. The problem though is that pagure sends a webhook for everything so a new build gets kicked off anytime anything happens on the repo (comment, closed issue, etc). See https://pagure.io/pagure/issue/4275#comment-573772
Wow. nice. :) ok, hopefully some solution will show up upstream.
Documented in: https://pagure.io/Fedora-Infra/howtos/blob/master/f/free_space_in_openshift.md
Thanks for the fix!
Log in to comment on this ticket.