We keep getting alerts that log01 is running low on disk space, looking in /var/log we have:
2.0M ┃ ┗━ journal.log 1.2G ┣━ journal/793b18f48b094bc3a93979878eccc696 ┃ ┖┄ [97 file(s)] 12G ┣━ merged ┃ ┖┄ [23 file(s)] 438G ┣━ lastlog 1.5T ┗━ hosts ┖┄ [4,564,798 file(s)]
...so while it's happening in merged, it's not really the cause. Looking in hosts we get:
60G ┣━ proxy06.vpn.fedoraproject.org/2024 ┃ ┖┄ [159,882 file(s)] 60G ┣━ proxy03.vpn.fedoraproject.org/2024 ┃ ┖┄ [170,767 file(s)] 61G ┣━ proxy11.vpn.fedoraproject.org/2024 ┃ ┖┄ [160,165 file(s)] 64G ┣━ proxy14.vpn.fedoraproject.org/2024 ┃ ┖┄ [155,785 file(s)] 64G ┣━ proxy09.vpn.fedoraproject.org/2024 ┃ ┖┄ [155,420 file(s)] 71G ┣━ proxy12.vpn.fedoraproject.org/2024 ┃ ┖┄ [170,668 file(s)] 85G ┣━ proxy101.iad2.fedoraproject.org/2024 ┃ ┖┄ [165,787 file(s)] 85G ┣━ proxy04.vpn.fedoraproject.org/2024 ┃ ┖┄ [153,719 file(s)] 88G ┣━ proxy110.iad2.fedoraproject.org/2024 ┃ ┖┄ [165,647 file(s)] 164G ┣━ proxy10.iad2.fedoraproject.org/2024 ┃ ┖┄ [169,637 file(s)] 167G ┗━ proxy01.iad2.fedoraproject.org/2024 ┖┄ [169,660 file(s)]
Even going lower it's mostly still proxy until you get to less than 10GB, with the exceptions being: dl0N, worker0N, oci-registry0N, people0N
So we can probably solve the problem with just proxyNN, but it might still be easier to do all of them.
Also worth noting that the logfile size changes significantly by month:
167G /var/log/hosts/proxy01.iad2.fedoraproject.org/2024 12G ┣━ 10 ┃ ┖┄ [12,438 file(s)] 15G ┣━ 02 ┃ ┖┄ [16,370 file(s)] 15G ┣━ 01 ┃ ┖┄ [17,515 file(s)] 16G ┣━ 07 ┃ ┖┄ [18,126 file(s)] 17G ┣━ 08 ┃ ┖┄ [18,122 file(s)] 17G ┣━ 03 ┃ ┖┄ [17,515 file(s)] 17G ┣━ 06 ┃ ┖┄ [17,394 file(s)] 17G ┣━ 09 ┃ ┖┄ [17,651 file(s)] 20G ┣━ 04 ┃ ┖┄ [16,947 file(s)] 21G ┗━ 05 ┖┄ [17,572 file(s)] 896 directories, 168,764 files, 167G size
...mostly due to distro/ostree releases:
38M ┣━ fedoraproject.org-access.log.xz 49M ┣━ src.fedoraproject.org-access.log.xz 78M ┣━ kojipkgs.fedoraproject.org-access.log.xz 125M ┗━ mirrors.fedoraproject.org-access.log.xz
...so it will likely go up a bunch as F41 is released (maybe next week).
The log data has grown tremendously on the proxies in the last couple of years. We used to archive once every couple of years and then it was every year. I agree with James, it is time to archive stuff off quarterly in an automated fashion. Is it possible to send it to an encrypted arctic storage somewhere?
Metadata Update from @phsmoura: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: ops
would it be worth it to logship to S3 and ultimately Glacier in the sponsored account?
I do have plans to leverage s3 for backups, but I don't want to rush that right now... we have backup space locally available.
Kevin and I moved Jan-June off that partition. There's a simple script in /var/log to help if we want to move more in 3-4 months, but it should be ignorable until next year atm.
Metadata Update from @james: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Log in to comment on this ticket.