Learn more about these different git repos.
Other Git URLs
[root@copr-be ~][PROD]# dmesg | grep Killed [12837497.767211] Out of memory: Killed process 22945 (Worker-1407-PPC) total-vm:205220kB, anon-rss:93816kB, file-rss:0kB, shmem-rss:0kB [12837501.826511] Out of memory: Killed process 2822 (createrepo_c) total-vm:150364kB, anon-rss:112728kB, file-rss:0kB, shmem-rss:0kB [12898787.329368] Out of memory: Killed process 25381 (Worker-2469-X86) total-vm:211948kB, anon-rss:100448kB, file-rss:80kB, shmem-rss:0kB [12898824.183340] Out of memory: Killed process 9989 (Worker-2512-X86) total-vm:212364kB, anon-rss:101068kB, file-rss:392kB, shmem-rss:0kB [12898830.698770] Out of memory: Killed process 10395 (Worker-2513-X86) total-vm:212364kB, anon-rss:101064kB, file-rss:20kB, shmem-rss:0kB [12898839.531983] Out of memory: Killed process 10935 (Worker-2514-AAR) total-vm:138376kB, anon-rss:100824kB, file-rss:0kB, shmem-rss:0kB [12898848.567010] Out of memory: Killed process 10941 (Worker-2515-X86) total-vm:212364kB, anon-rss:101080kB, file-rss:400kB, shmem-rss:0kB [12898900.382831] Out of memory: Killed process 10950 (Worker-2518-X86) total-vm:212364kB, anon-rss:101120kB, file-rss:404kB, shmem-rss:0kB [12898927.068564] Out of memory: Killed process 10944 (Worker-2516-X86) total-vm:212364kB, anon-rss:101108kB, file-rss:400kB, shmem-rss:0kB
Likely related #987
I'm experimenting with SWAP on, I remember that we discussed that having it ON caused performance issues.. so let's see the problem still exist and perhaps fix the real issue. https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=5243384032303deb0c61c5a95f066e12fa9c093f
Each Worker process eats 101M, it would be good to have a look why.
Watching /var/log/copr-backend/cleanup-redis-vms.log (see crontab -l), it is empty now at the time of re-enabling swap.
Metadata Update from @praiskup: - Issue assigned to praiskup
Everything is OK after several days. I'd configure swap in ansible.git but turns out the copr-be-dev instance doesn't have allocated /dev/vdb disk for swap, yet. So I'd wait for next re-spawn from scratch, for now I applied this: https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=a67f653aa3ab0e05266845827cbcedf5fb4c3d1c
We have 32G ram and 16G swap on backend now.
Metadata Update from @praiskup: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.