Precondition: I run FreeIPA integration tests in Docker environment on the local machine and on Azure Pipelines. The topology ( a combination of IPA's master/replica/client) may be different. Several topologies may be tested simultaneously.
Issue: I see that slapd consumes a lot of RAM (relatively). Of course, I make a memory limit (via cgroups) for containers. But all in vain.
There are interesting auto-tunning messages, for example:
- ldbm_back_start - found 8162012k physical memory - ldbm_back_start - found 5745452k available - ldbm_back_start - cache autosizing: db cache: 204050k - ldbm_back_start - cache autosizing: userRoot entry cache (2 total): 327680k - ldbm_back_start - cache autosizing: userRoot dn cache (2 total): 65536k - ldbm_back_start - cache autosizing: ipaca entry cache (2 total): 327680k - ldbm_back_start - cache autosizing: ipaca dn cache (2 total): 65536k - ldbm_back_start - total cache size: 972464373 B;
Hm, 8162012k is not a container's RAM size, but the host's one.
8162012k
Current algorithm of detecting RAM size (from code): 1) RLIMIT_AS (unlimited) 2) /proc/meminfo - is a host's one 3) /sys/fs/cgroup/memory/memory.limit_in_bytes - is a host's one because container's cgroup lives in another group, for example,
cat /sys/fs/cgroup/memory/system.slice/docker-b149aa6a6ad6c73635fc6f1525215ceecc3da631b7e5505fe1820d6ebdba1f16.scope/memory.limit_in_bytes 1572864000
Therefore, current implementation doesn't support such a configuration. I don't know whether it's by design or not.
After reading autotuning documentation [0] I see that it is possible to disable this feature but then requires manual intervention/tuning. This is not something that I want for CI.
My questions: 1) Do you know any good workaround? 2) Is it by design?
[0] https://access.redhat.com/documentation/en-us/red_hat_directory_server/10/html-single/performance_tuning_guide/index#enable_db_and_entry_cache_auto-sizing
This indeed sounds like an issue. I'll investigate soon.
Metadata Update from @firstyear: - Custom field origin adjusted to None - Custom field reviewstatus adjusted to None
Hi there,
I think this bug may be invalid? Looking at the docker documentation I see:
https://docs.docker.com/engine/reference/commandline/run/
QUOTE:
Specify hard limits on memory available to containers (-m, --memory)
These parameters always set an upper limit on the memory available to the container. On Linux, this is set on the cgroup and applications in a container can query it at /sys/fs/cgroup/memory/memory.limit_in_bytes.
END QUOTE.
Testing this with a container I can confirm it works as intended:
Container with no limit:
cat /sys/fs/cgroup/memory/memory.limit_in_bytes 9223372036854771712
Container started with -m 2GB
cat /sys/fs/cgroup/memory/memory.limit_in_bytes 2147483648
As a result, I'm going to have to ask for more information about your environment. Perhaps this is an issue with podman vs docker? It works on docker. Maybe it's cgroup v1 vs v2?
Can you tell me more about the docker host machine, and what container engine you are using?
Thanks,
@slev Post-holidays reminder that I'd like some more info to help you here :)
Can we also see "cat /proc/$PID/cgroup" for the ns-slapd process too?
@firstyear, hi. Today is the first working day in Russia. Please, give me some time. Thank you.
Host: Container engine: Docker cgroup: hybrid
ns-slapd cgroup from within host:
cat /proc/85408/cgroup 11:blkio:/system.slice/docker-effc7012609ac57c7f86c88c5df650bac0ec66fad228168f2f9b343d189b10e5.scope 10:freezer:/system.slice/docker-effc7012609ac57c7f86c88c5df650bac0ec66fad228168f2f9b343d189b10e5.scope 9:hugetlb:/system.slice/docker-effc7012609ac57c7f86c88c5df650bac0ec66fad228168f2f9b343d189b10e5.scope 8:cpuset:/system.slice/docker-effc7012609ac57c7f86c88c5df650bac0ec66fad228168f2f9b343d189b10e5.scope 7:perf_event:/system.slice/docker-effc7012609ac57c7f86c88c5df650bac0ec66fad228168f2f9b343d189b10e5.scope 6:devices:/system.slice/docker-effc7012609ac57c7f86c88c5df650bac0ec66fad228168f2f9b343d189b10e5.scope/system.slice/system-dirsrv.slice 5:cpu,cpuacct:/system.slice/docker-effc7012609ac57c7f86c88c5df650bac0ec66fad228168f2f9b343d189b10e5.scope 4:memory:/system.slice/docker-effc7012609ac57c7f86c88c5df650bac0ec66fad228168f2f9b343d189b10e5.scope/system.slice/system-dirsrv.slice/dirsrv@IPA-TEST.service 3:pids:/system.slice/docker-effc7012609ac57c7f86c88c5df650bac0ec66fad228168f2f9b343d189b10e5.scope/system.slice/system-dirsrv.slice/dirsrv@IPA-TEST.service 2:net_cls,net_prio:/system.slice/docker-effc7012609ac57c7f86c88c5df650bac0ec66fad228168f2f9b343d189b10e5.scope 1:name=systemd:/system.slice/docker-effc7012609ac57c7f86c88c5df650bac0ec66fad228168f2f9b343d189b10e5.scope/system.slice/system-dirsrv.slice/dirsrv@IPA-TEST.service 0::/system.slice/docker-effc7012609ac57c7f86c88c5df650bac0ec66fad228168f2f9b343d189b10e5.scope/system.slice/system-dirsrv.slice/dirsrv@IPA-TEST.service
ns-slapd cgroup from within container:
cat /proc/4405/cgroup 11:blkio:/system.slice/docker-effc7012609ac57c7f86c88c5df650bac0ec66fad228168f2f9b343d189b10e5.scope 10:freezer:/system.slice/docker-effc7012609ac57c7f86c88c5df650bac0ec66fad228168f2f9b343d189b10e5.scope 9:hugetlb:/system.slice/docker-effc7012609ac57c7f86c88c5df650bac0ec66fad228168f2f9b343d189b10e5.scope 8:cpuset:/system.slice/docker-effc7012609ac57c7f86c88c5df650bac0ec66fad228168f2f9b343d189b10e5.scope 7:perf_event:/system.slice/docker-effc7012609ac57c7f86c88c5df650bac0ec66fad228168f2f9b343d189b10e5.scope 6:devices:/system.slice/docker-effc7012609ac57c7f86c88c5df650bac0ec66fad228168f2f9b343d189b10e5.scope/system.slice/system-dirsrv.slice 5:cpu,cpuacct:/system.slice/docker-effc7012609ac57c7f86c88c5df650bac0ec66fad228168f2f9b343d189b10e5.scope 4:memory:/system.slice/docker-effc7012609ac57c7f86c88c5df650bac0ec66fad228168f2f9b343d189b10e5.scope/system.slice/system-dirsrv.slice/dirsrv@IPA-TEST.service 3:pids:/system.slice/docker-effc7012609ac57c7f86c88c5df650bac0ec66fad228168f2f9b343d189b10e5.scope/system.slice/system-dirsrv.slice/dirsrv@IPA-TEST.service 2:net_cls,net_prio:/system.slice/docker-effc7012609ac57c7f86c88c5df650bac0ec66fad228168f2f9b343d189b10e5.scope 1:name=systemd:/system.slice/docker-effc7012609ac57c7f86c88c5df650bac0ec66fad228168f2f9b343d189b10e5.scope/system.slice/system-dirsrv.slice/dirsrv@IPA-TEST.service 0::/system.slice/docker-effc7012609ac57c7f86c88c5df650bac0ec66fad228168f2f9b343d189b10e5.scope/system.slice/system-dirsrv.slice/dirsrv@IPA-TEST.service
Container:
cat /proc/self/cgroup 11:blkio:/system.slice/docker-effc7012609ac57c7f86c88c5df650bac0ec66fad228168f2f9b343d189b10e5.scope 10:freezer:/system.slice/docker-effc7012609ac57c7f86c88c5df650bac0ec66fad228168f2f9b343d189b10e5.scope 9:hugetlb:/system.slice/docker-effc7012609ac57c7f86c88c5df650bac0ec66fad228168f2f9b343d189b10e5.scope 8:cpuset:/system.slice/docker-effc7012609ac57c7f86c88c5df650bac0ec66fad228168f2f9b343d189b10e5.scope 7:perf_event:/system.slice/docker-effc7012609ac57c7f86c88c5df650bac0ec66fad228168f2f9b343d189b10e5.scope 6:devices:/system.slice/docker-effc7012609ac57c7f86c88c5df650bac0ec66fad228168f2f9b343d189b10e5.scope 5:cpu,cpuacct:/system.slice/docker-effc7012609ac57c7f86c88c5df650bac0ec66fad228168f2f9b343d189b10e5.scope 4:memory:/system.slice/docker-effc7012609ac57c7f86c88c5df650bac0ec66fad228168f2f9b343d189b10e5.scope 3:pids:/system.slice/docker-effc7012609ac57c7f86c88c5df650bac0ec66fad228168f2f9b343d189b10e5.scope 2:net_cls,net_prio:/system.slice/docker-effc7012609ac57c7f86c88c5df650bac0ec66fad228168f2f9b343d189b10e5.scope 1:name=systemd:/system.slice/docker-effc7012609ac57c7f86c88c5df650bac0ec66fad228168f2f9b343d189b10e5.scope 0::/system.slice/containerd.service
Note: the container is unprivileged and was created as:
docker create -v /sys/fs/cgroup:/sys/fs/cgroup:ro -m 2GB --cap-add=all IMAGE
Full reproducer: 1) Dockerfile
FROM fedora:31 RUN \ dnf install systemd ENTRYPOINT [ "/usr/sbin/init" ]
2) docker build -t fedora-test .
3) docker create --name fed_test -v /sys/fs/cgroup:/sys/fs/cgroup:ro -m 2GB --cap-add=all fedora-test
4) docker start fed_test
5) docker exec -ti fed_test /bin/bash -c 'cat /sys/fs/cgroup/memory/memory.limit_in_bytes'
9223372036854771712
Metadata Update from @mreynolds: - Issue set to the milestone: 1.4.1
-v /sys/fs/cgroup:/sys/fs/cgroup:ro
You're mounting the hosts cgroups over the docker ones. You shouldn't be doing this, and it's the cause of the issues you are seeing.
Metadata Update from @firstyear: - Issue close_status updated to: worksforme - Issue status updated to: Closed (was: Open)
-v /sys/fs/cgroup:/sys/fs/cgroup:ro You're mounting the hosts cgroups over the docker ones. You shouldn't be doing this, and it's the cause of the issues you are seeing.
I didn't agree. You might want to see 3 nice articles by Daniel Walsh:
https://developers.redhat.com/blog/2014/05/05/running-systemd-within-docker-container/ https://developers.redhat.com/blog/2016/09/13/running-systemd-in-a-non-privileged-container/ https://developers.redhat.com/blog/2019/04/24/how-to-run-systemd-in-a-container/
And another systemd's https://systemd.io/CGROUP_DELEGATION/
Finally, it's not a 389-ds issue, but my configuration.
For those who will face the same problem: if systemd at cgroups legacy mode (cgroup v1 only) it's enough to mount /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd (at least for now)
/sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd
With this change, the memory limitation works as expected and ns-slapd autotunes well. Thank you.
389-ds-base is moving from Pagure to Github. This means that new issues and pull requests will be accepted only in 389-ds-base's github repository.
This issue has been cloned to Github and is available here: - https://github.com/389ds/389-ds-base/issues/3853
If you want to receive further updates on the issue, please navigate to the github issue and click on subscribe button.
subscribe
Thank you for understanding. We apologize for all inconvenience.
Metadata Update from @spichugi: - Issue close_status updated to: wontfix (was: worksforme)
Login to comment on this ticket.