#8643 openshift in staging no longer building
Closed: Fixed 4 years ago by kevin. Opened 4 years ago by pingou.

Openshift in staging is no longer building anything, this is the error it is returning:

error: build error: devmapper: Thin Pool has 11561 free data blocks which is less than minimum required 11944 free data blocks. Create more free space in thin pool or use dm.min_free_space option to change behavior

URL: https://os.stg.fedoraproject.org/console/project/monitor-gating/browse/builds/monitor-gating-build/monitor-gating-build-40?tab=logs


So, looking at space on nodes with:

ansible -a 'lvs' os_nodes_stg

os-node03.stg.phx2.fedoraproject.org | CHANGED | rc=0 >>
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
root GuestVolGroup00 -wi-ao---- <58.59g
docker-pool vg-docker twi-a-t--- 58.32g 42.00 18.44

os-node02.stg.phx2.fedoraproject.org | CHANGED | rc=0 >>
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
root GuestVolGroup00 -wi-ao---- <58.59g
docker-pool vg-docker twi-a-t--- <48.60g 32.37 14.81

os-node01.stg.phx2.fedoraproject.org | CHANGED | rc=0 >>
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
root GuestVolGroup00 -wi-ao---- <58.59g
docker-pool vg-docker twi-a-t--- 58.32g 40.75 17.38

os-node04.stg.phx2.fedoraproject.org | CHANGED | rc=0 >>
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
root GuestVolGroup00 -wi-ao---- <58.59g
docker-pool vg-docker twi-a-t--- 58.32g 90.32 28.35

You can see os-node04 is at 90%. ;(

We setup a weekly cron job to remove docker images that aren't needed anymore, but it seems that there was a ton of builds from compose-tracker and others.
I ran the weekly cron manually:

[root@os-node04 ~][STG]# docker rmi $(docker images --filter dangling=true -q)

[root@os-node04 ~][STG]# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
root GuestVolGroup00 -wi-ao---- <58.59g
docker-pool vg-docker twi-a-t--- 58.32g 55.77 21.17

So, possibly we should run the cron more often, or perhaps we should just increase space. I guess we can wait and see if it happens again...

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

4 years ago

we set up compose-tracker to be rebuilt based on webhooks from pagure. The problem though is that pagure sends a webhook for everything so a new build gets kicked off anytime anything happens on the repo (comment, closed issue, etc). See https://pagure.io/pagure/issue/4275#comment-573772

Wow. nice. :) ok, hopefully some solution will show up upstream.

Login to comment on this ticket.

Metadata