#49178 389 (setup-ds.pl) wants to allocate 40GiB of RAM
Closed: wontfix 11 months ago by mreynolds. Opened 3 years ago by ttorcz.

Issue Description

I'm trying to create FreeIPA replica using ipa-replica-install. The commands fails, and log file shows:
2017-03-21T14:19:47Z DEBUG calling setup-ds.pl
2017-03-21T14:19:47Z DEBUG Starting external process
2017-03-21T14:19:47Z DEBUG args=/usr/sbin/setup-ds.pl --silent --logfile - -f /tmp/tmp1bwAYm
2017-03-21T14:20:02Z DEBUG Process finished, return code=1
2017-03-21T14:20:02Z DEBUG stdout=[17/03/21:15:20:02] - [Setup] Info Could not import LDIF file '/var/lib/dirsrv/boot.ldif'. Error: 3072. Output: importing data ...
[21/Mar/2017:15:20:01.646134308 +0100] WARNING: Import is running with nsslapd-db-private-import-mem on; No other process is allowed to access the database
[21/Mar/2017:15:20:01.927035173 +0100] check_and_set_import_cache: pagesize: 4096, pages: -1, procpages: 4593
[21/Mar/2017:15:20:02.129829116 +0100] Import allocates -859122368KB import cache.
[21/Mar/2017:15:20:02.203908584 +0100] Setting ncache to: 858993460 to keep each chunk below 4Gbytes
[21/Mar/2017:15:20:02.348597443 +0100] libdb: BDB0147 malloc: 41231686328: Cannot allocate memory
[21/Mar/2017:15:20:02.452740622 +0100] libdb: BDB1543 unable to create new master region array: Cannot allocate memory
[21/Mar/2017:15:20:02.526455796 +0100] Unable to open new DB_ENV for import/export! 12
[21/Mar/2017:15:20:02.613398853 +0100] dblayer_instance_start fail: Cannot allocate memory (12)

Could not import LDIF file '/var/lib/dirsrv/boot.ldif'. Error: 3072. Output: importing data ...
[21/Mar/2017:15:20:01.646134308 +0100] WARNING: Import is running with nsslapd-db-private-import-mem on; No other process is allowed to access the database
[21/Mar/2017:15:20:01.927035173 +0100] check_and_set_import_cache: pagesize: 4096, pages: -1, procpages: 4593
[21/Mar/2017:15:20:02.129829116 +0100] Import allocates -859122368KB import cache.
[21/Mar/2017:15:20:02.203908584 +0100] Setting ncache to: 858993460 to keep each chunk below 4Gbytes
[21/Mar/2017:15:20:02.348597443 +0100] libdb: BDB0147 malloc: 41231686328: Cannot allocate memory
[21/Mar/2017:15:20:02.452740622 +0100] libdb: BDB1543 unable to create new master region array: Cannot allocate memory
[21/Mar/2017:15:20:02.526455796 +0100] Unable to open new DB_ENV for import/export! 12
[21/Mar/2017:15:20:02.613398853 +0100] dblayer_instance_start fail: Cannot allocate memory (12)

Package Version and Platform

389-ds-base-1.3.5.15-1.fc25.x86_64

```# free -h
total used free shared buff/cache available

Mem: 8388607T 223M 8388607T 307M 59M 8388607T

Swap: 15G 7.4G 8.6G```

Steps to reproduce

  1. Create libvirt-lxc container with 2GB of RAM.
  2. Install base fedora 25
  3. ipa-replica-install --mkhomedir --setup-ca --setup-dns --auto-forwarders

Actual results

2017-03-21T14:20:02Z DEBUG The ipa-replica-install command failed, exception: RuntimeError: failed to create ds instance Command '/usr/sbin/setup-ds.pl --silent --logfile - -f /tmp/tmp1bwAYm' returned non-zero exit status 1 2017-03-21T14:20:03Z ERROR failed to create ds instance Command '/usr/sbin/setup-ds.pl --silent --logfile - -f /tmp/tmp1bwAYm' returned non-zero exit status 1
2017-03-21T14:20:03Z ERROR The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information

Expected results

Working replica.


Likely caused by https://pagure.io/389-ds-base/issue/48864 which affects the import autotuning during setup-ds.pl import.

Metadata Update from @firstyear:
- Custom field reviewstatus adjusted to new
- Custom field type adjusted to defect

3 years ago

Metadata Update from @firstyear:
- Issue assigned to firstyear
- Issue set to the milestone: 1.3.7 backlog

3 years ago

Metadata Update from @firstyear:
- Issue tagged with: Complex, Performance

3 years ago

Metadata Update from @firstyear:
- Custom field reviewstatus reset (from new)
- Issue close_status updated to: fixed
- Issue status updated to: Closed (was: Open)

3 years ago

Metadata Update from @ttorcz:
- Issue status updated to: Open (was: Closed)

2 years ago

Reopening, as it wasn't fixed. I've tried to create replica in libvirt-lxc container today again, with 389-ds-base-1.3.7.7-1.fc27.x86_64.
The result is

[26/Nov/2017:13:21:30.424392946 +0000] - INFO - check_and_set_import_cache - pagesize: 4096, available bytes 9223372021559820288, process usage 24150016 
[26/Nov/2017:13:21:30.463022012 +0000] - INFO - check_and_set_import_cache - Import allocates 144115182101263KB import cache.
[26/Nov/2017:13:21:30.502810880 +0000] - NOTICE - dblayer_select_ncache - Setting ncache to: 34359737 to keep each chunk below 4Gbytes
tcmalloc: large alloc 1649270784 bytes == 0x555949b6e000 @  0x7fc1658719ad 0x7fc165891413 0x7fc15cb9d485 0x7fc15cb67f90 0x7fc15cb713bd 0x7fc15cb6b610 0x7fc15cb6bf32 0x7fc1580f7b77 0x7fc15813f9aa 0x555947949e8b 0x7fc16375703a 0x55594794b00a
tcmalloc: large alloc 584613150720 bytes == (nil) @  0x7fc1658719ad 0x7fc165891413 0x7fc15cb9d485 0x7fc15cb67f90 0x7fc15cb29149 0x7fc15cb6b6a8 0x7fc15cb6bf32 0x7fc1580f7b77 0x7fc15813f9aa 0x555947949e8b 0x7fc16375703a 0x55594794b00a
[26/Nov/2017:13:21:31.548291806 +0000] - ERR - libdb - BDB0147 malloc: 584613149444: Cannot allocate memory
[26/Nov/2017:13:21:31.676567865 +0000] - ERR - libdb - BDB2014 Unable to allocate memory for mutexes from the region
[26/Nov/2017:13:21:31.744285973 +0000] - ERR - libdb - BDB0061 PANIC: Cannot allocate memory

Something is very odd with your environment. I'll need the following log output:

cat /var/log/slapd-INSTANCE-NAME/errors | grep spal

Your partially installed IPA instance (before you run ipa-server-install -u) will work for this. Then send me the log from /tmp/spal.log.

We try to determine memory limits from a number of places. These are:

  • rlimit
  • cgroup
  • /proc/meminfo

The only time we've seen this kind of behaviour is when your /proc/meminfo does not provite MemAvailable, and we resolved that. This looks more like your /proc/meminfo may not exist at all, and we fall back to cgroup, which has a very large available (9223372021559820288).

So in addition, I want to see your lxc:

  • /proc/meminfo
  • /sys/fs/cgroup/memory/memory.soft_limit_in_bytes
  • /sys/fs/cgroup/memory/memory.limit_in_bytes
  • /sys/fs/cgroup/memory/memory.usage_in_bytes

Thanks,

Metadata Update from @firstyear:
- Custom field component adjusted to None
- Custom field origin adjusted to None
- Custom field reviewstatus adjusted to None
- Custom field version adjusted to None

2 years ago

No 'spal' in errors:

# cat /var/log/dirsrv/slapd-PIPEBREAKER-PL/errors | grep -i spal
# 

/tmp/spal.log does not exist. If it were here previously, it was deleted on reboot.

/proc/meminfo:

MemTotal:       9007199254740991 kB
MemFree:        9007199254589103 kB
MemAvailable:   9007199254589103 kB
Buffers:               0 kB
Cached:           113196 kB
SwapCached:       105524 kB
Active:            39236 kB
Inactive:          92728 kB
Active(anon):      17272 kB
Inactive(anon):     1612 kB
Active(file):      21964 kB
Inactive(file):    91116 kB
Unevictable:           0 kB
Mlocked:             944 kB
SwapTotal:      16777212 kB
SwapFree:       14824956 kB
Dirty:             14200 kB
Writeback:            12 kB
AnonPages:       6222176 kB
Mapped:           618516 kB
Shmem:            370032 kB
Slab:                  0 kB
SReclaimable:          0 kB
SUnreclaim:            0 kB
KernelStack:       37456 kB
PageTables:       124448 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    24912936 kB
Committed_AS:   20983636 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
HardwareCorrupted:     0 kB
AnonHugePages:   s:        0 kB
ShmemPmdMapped:        0 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:     2600024 kB
DirectMap2M:    14016512 kB

cgroup files:

# cat /sys/fs/cgroup/memory/memory.soft_limit_in_bytes
9223372036854771712
# cat /sys/fs/cgroup/memory/memory.limit_in_bytes
9223372036854771712
# cat /sys/fs/cgroup/memory/memory.usage_in_bytes
14551904256

I don't think this is an issue anymore. The perl tools are also deprecated. Closing ticket, but if the problem happens with dscreate please reopen this ticket

Metadata Update from @mreynolds:
- Issue close_status updated to: wontfix
- Issue status updated to: Closed (was: Open)

11 months ago

Login to comment on this ticket.

Metadata