#49392 Unable to retrieve /proc/meminfo : MemAvailable:
Closed: wontfix 6 years ago Opened 6 years ago by markovich.

Issue Description

I'm trying to install ipa-server-4.5.0-21.0.1.el7_4.1.2.x86_64 on Oracle Enterprise Linux but getting error:

[Setup] Info Could not import LDIF file '/var/lib/dirsrv/boot.ldif'. Error: 768. Output: importing data ...
[04/Oct/2017:11:55:53.798978140 -0400] - ERR - spal_meminfo_get - Unable to retrieve /proc/meminfo : MemAvailable:
...

cat /proc/meminfo
MemTotal: 16170720 kB
MemFree: 9051152 kB
Buffers: 11280 kB
Cached: 3490240 kB
SwapCached: 0 kB
Active: 5041772 kB
Inactive: 1337116 kB
Active(anon): 2878404 kB
Inactive(anon): 8128 kB
Active(file): 2163368 kB
Inactive(file): 1328988 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 2097148 kB
SwapFree: 2097148 kB
Dirty: 400 kB
Writeback: 0 kB
AnonPages: 2877664 kB
Mapped: 66880 kB
Shmem: 8876 kB
Slab: 562760 kB
SReclaimable: 519908 kB
SUnreclaim: 42852 kB
KernelStack: 7408 kB
PageTables: 12624 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 10182508 kB
Committed_AS: 5700160 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 31364 kB
VmallocChunk: 34359694544 kB
HardwareCorrupted: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 8192 kB
DirectMap2M: 2088960 kB
DirectMap1G: 14680064 kB

Package Version and Platform

ipa-server-4.5.0-21.0.1.el7_4.1.2.x86_64

cat /etc/oracle-release
Oracle Linux Server release 7.4

uname -a
Linux FQDN 3.8.13-118.19.7.el7uek.x86_64 #2 SMP Fri Sep 15 18:15:47 PDT 2017 x86_64 x86_64 x86_64 GNU/Linux

This is also a VM on Xen hypervisor, wuth HVM virtualization with PV drivers. Architecture x64.

Steps to reproduce

  1. Clean Oracle Linux 7.4
  2. Installing ipa-server from yum
  3. ipa-server-install --hostname=myhostname --domain=mydomain.com --realm=MYDOMAIN.COM --ds-password=password--master-password=password--admin-password=password--unattended --debug

Actual results

Done configuring NTP daemon (ntpd).
Configuring directory server (dirsrv). Estimated time: 30 seconds
[1/45]: creating directory server instance
[error] RuntimeError: failed to create DS instance Command '/usr/sbin/setup-ds.pl --silent --logfile - -f /tmp/tmpPQPUX_' returned non-zero exit status 1

More debug info is here:

2017-10-04T15:55:52Z DEBUG calling setup-ds.pl
2017-10-04T15:55:52Z DEBUG Starting external process
2017-10-04T15:55:52Z DEBUG args=/usr/sbin/setup-ds.pl --silent --logfile - -f /tmp/tmpPQPUX_
2017-10-04T15:55:56Z DEBUG Process finished, return code=1
2017-10-04T15:55:56Z DEBUG stdout=[17/10/04:11:55:56] - [Setup] Info Could not import LDIF file '/var/lib/dirsrv/boot.ldif'. Error: 768. Output: importing data ...
[04/Oct/2017:11:55:53.798978140 -0400] - ERR - spal_meminfo_get - Unable to retrieve /proc/meminfo : MemAvailable:
[04/Oct/2017:11:55:53.900526100 -0400] - INFO - ldbm_instance_config_cachememsize_set - force a minimal value 512000
[04/Oct/2017:11:55:53.902864577 -0400] - ERR - spal_meminfo_get - Unable to retrieve /proc/meminfo : MemAvailable:
[04/Oct/2017:11:55:53.923965959 -0400] - ERR - spal_meminfo_get - Unable to retrieve /proc/meminfo : MemAvailable:
[04/Oct/2017:11:55:53.945262395 -0400] - ERR - spal_meminfo_get - Unable to retrieve /proc/meminfo : MemAvailable:
[04/Oct/2017:11:55:53.953918605 -0400] - INFO - dblayer_instance_start - Import is running with nsslapd-db-private-import-mem on; No other process is allowed to access the database
[04/Oct/2017:11:55:53.961341875 -0400] - ERR - spal_meminfo_get - Unable to retrieve /proc/meminfo : MemAvailable:
[04/Oct/2017:11:55:53.963334442 -0400] - INFO - check_and_set_import_cache - pagesize: 4096, available bytes 9223372030878334975, process usage 13570048
[04/Oct/2017:11:55:53.964858651 -0400] - INFO - check_and_set_import_cache - Import allocates 144115185741308KB import cache.
[04/Oct/2017:11:55:53.966412876 -0400] - ERR - spal_meminfo_get - Unable to retrieve /proc/meminfo : MemAvailable:
[04/Oct/2017:11:55:53.968456383 -0400] - NOTICE - dblayer_select_ncache - Setting ncache to: 34359738 to keep each chunk below 4Gbytes
tcmalloc: large alloc 1649270784 bytes == 0x7f370367e000 @ 0x7f36ffd8cbf3 0x7f36ffdaccf2 0x7f36f751bb25 0x7f36f74e5c17 0x7f36f74ef019 0x7f36f74e9260 0x7f36f74e9a40 0x7 f36f335bf61 0x7f36f33a32ff 0x7f37009720e2 0x7f36fd5a1c05 0x7f3700972ad7
tcmalloc: large alloc 584852226048 bytes == (nil) @ 0x7f36ffd8cbf3 0x7f36ffdaccf2 0x7f36f751bb25 0x7f36f74e5c17 0x7f36f74a7d4a 0x7f36f74e92fc 0x7f36f74e9a40 0x7f36f335 bf61 0x7f36f33a32ff 0x7f37009720e2 0x7f36fd5a1c05 0x7f3700972ad7
[04/Oct/2017:11:55:55.213645769 -0400] - ERR - libdb - BDB0147 malloc: 584852224468: Cannot allocate memory
[04/Oct/2017:11:55:55.233320159 -0400] - ERR - libdb - BDB2014 Unable to allocate memory for mutexes from the region
[04/Oct/2017:11:55:55.244128535 -0400] - ERR - libdb - BDB0061 PANIC: Cannot allocate memory
tcmalloc: large alloc 1649270784 bytes == 0x7f370367e000 @ 0x7f36ffd8cbf3 0x7f36ffdaccf2 0x7f36f751bb25 0x7f36f74e5c17 0x7f36f74ef019 0x7f36f74f0023 0x7f36f74e961c 0x7 f36f74e9a40 0x7f36f335bf61 0x7f36f33a32ff 0x7f37009720e2 0x7f36fd5a1c05 0x7f3700972ad7
[04/Oct/2017:11:55:56.472105344 -0400] - ERR - libdb - BDB1544 process-private: unable to find environment
[04/Oct/2017:11:55:56.481460130 -0400] - ERR - dblayer_instance_start - Unable to open new DB_ENV for import/export! -30973
[04/Oct/2017:11:55:56.483212941 -0400] - ERR - dblayer_instance_start - Failure BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery (-30973)

Could not import LDIF file '/var/lib/dirsrv/boot.ldif'. Error: 768. Output: importing data ...
[04/Oct/2017:11:55:53.798978140 -0400] - ERR - spal_meminfo_get - Unable to retrieve /proc/meminfo : MemAvailable:
[04/Oct/2017:11:55:53.900526100 -0400] - INFO - ldbm_instance_config_cachememsize_set - force a minimal value 512000
[04/Oct/2017:11:55:53.902864577 -0400] - ERR - spal_meminfo_get - Unable to retrieve /proc/meminfo : MemAvailable:
[04/Oct/2017:11:55:53.923965959 -0400] - ERR - spal_meminfo_get - Unable to retrieve /proc/meminfo : MemAvailable:
[04/Oct/2017:11:55:53.945262395 -0400] - ERR - spal_meminfo_get - Unable to retrieve /proc/meminfo : MemAvailable:
[04/Oct/2017:11:55:53.953918605 -0400] - INFO - dblayer_instance_start - Import is running with nsslapd-db-private-import-mem on; No other process is allowed to access the database
[04/Oct/2017:11:55:53.961341875 -0400] - ERR - spal_meminfo_get - Unable to retrieve /proc/meminfo : MemAvailable:
[04/Oct/2017:11:55:53.963334442 -0400] - INFO - check_and_set_import_cache - pagesize: 4096, available bytes 9223372030878334975, process usage 13570048
[04/Oct/2017:11:55:53.964858651 -0400] - INFO - check_and_set_import_cache - Import allocates 144115185741308KB import cache.
[04/Oct/2017:11:55:53.966412876 -0400] - ERR - spal_meminfo_get - Unable to retrieve /proc/meminfo : MemAvailable:
[04/Oct/2017:11:55:53.968456383 -0400] - NOTICE - dblayer_select_ncache - Setting ncache to: 34359738 to keep each chunk below 4Gbytes
tcmalloc: large alloc 1649270784 bytes == 0x7f370367e000 @ 0x7f36ffd8cbf3 0x7f36ffdaccf2 0x7f36f751bb25 0x7f36f74e5c17 0x7f36f74ef019 0x7f36f74e9260 0x7f36f74e9a40 0x7 f36f335bf61 0x7f36f33a32ff 0x7f37009720e2 0x7f36fd5a1c05 0x7f3700972ad7
tcmalloc: large alloc 584852226048 bytes == (nil) @ 0x7f36ffd8cbf3 0x7f36ffdaccf2 0x7f36f751bb25 0x7f36f74e5c17 0x7f36f74a7d4a 0x7f36f74e92fc 0x7f36f74e9a40 0x7f36f335 bf61 0x7f36f33a32ff 0x7f37009720e2 0x7f36fd5a1c05 0x7f3700972ad7
[04/Oct/2017:11:55:55.213645769 -0400] - ERR - libdb - BDB0147 malloc: 584852224468: Cannot allocate memory
[04/Oct/2017:11:55:55.233320159 -0400] - ERR - libdb - BDB2014 Unable to allocate memory for mutexes from the region
[04/Oct/2017:11:55:55.244128535 -0400] - ERR - libdb - BDB0061 PANIC: Cannot allocate memory
tcmalloc: large alloc 1649270784 bytes == 0x7f370367e000 @ 0x7f36ffd8cbf3 0x7f36ffdaccf2 0x7f36f751bb25 0x7f36f74e5c17 0x7f36f74ef019 0x7f36f74f0023 0x7f36f74e961c 0x7 f36f74e9a40 0x7f36f335bf61 0x7f36f33a32ff 0x7f37009720e2 0x7f36fd5a1c05 0x7f3700972ad7
[04/Oct/2017:11:55:56.472105344 -0400] - ERR - libdb - BDB1544 process-private: unable to find environment
[04/Oct/2017:11:55:56.481460130 -0400] - ERR - dblayer_instance_start - Unable to open new DB_ENV for import/export! -30973
[04/Oct/2017:11:55:56.483212941 -0400] - ERR - dblayer_instance_start - Failure BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery (-30973)

[17/10/04:11:55:56] - [Setup] Fatal Error: Could not create directory server instance 'MYDOMAIN-COM'.
Error: Could not create directory server instance 'MYDOMAIN-COM'.
[17/10/04:11:55:56] - [Setup] Fatal Exiting . . .
Log file is '-'

Exiting . . .
Log file is '-'

2017-10-04T15:55:56Z DEBUG stderr=
2017-10-04T15:55:56Z DEBUG Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 504, in start_creation
run_step(full_msg, method)
File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 494, in run_step
method()
File "/usr/lib/python2.7/site-packages/ipaserver/install/dsinstance.py", line 559, in __create_instance
raise RuntimeError("failed to create DS instance %s" % e)
RuntimeError: failed to create DS instance Command '/usr/sbin/setup-ds.pl --silent --logfile - -f /tmp/tmpPQPUX_' returned non-zero exit status 1

2017-10-04T15:55:56Z DEBUG [error] RuntimeError: failed to create DS instance Command '/usr/sbin/setup-ds.pl --silent --logfile - -f /tmp/tmpPQPUX_' returned non-zero exit status 1
2017-10-04T15:55:56Z DEBUG File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 172, in execute
return_value = self.run()
File "/usr/lib/python2.7/site-packages/ipapython/install/cli.py", line 333, in run
cfgr.run()
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 368, in run
self.execute()
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 392, in execute
for _nothing in self._executor():
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 434, in runner
exc_handler(exc_info)
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 463, in _handle_execute_exception
self._handle_exception(exc_info)
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 453, in _handle_exception
six.reraise(exc_info)
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 424, in __runner
step()
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 421, in <lambda>
step = lambda: next(self.__gen)
File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
six.reraise(
exc_info)
File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
value = gen.send(prev_value)
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 658, in _configure
next(executor)
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 434, in __runner
exc_handler(exc_info)
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 463, in _handle_execute_exception
self._handle_exception(exc_info)
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 521, in _handle_exception
self.__parent._handle_exception(exc_info)
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 453, in _handle_exception
six.reraise(exc_info)
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 518, in _handle_exception
super(ComponentBase, self)._handle_exception(exc_info)
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 453, in _handle_exception
six.reraise(
exc_info)
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 424, in __runner
step()
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 421, in <lambda>
step = lambda: next(self.__gen)
File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
six.reraise(*exc_info)
File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
value = gen.send(prev_value)
File "/usr/lib/python2.7/site-packages/ipapython/install/common.py", line 63, in _install
for _nothing in self._installer(self.parent):
File "/usr/lib/python2.7/site-packages/ipaserver/install/server/__init
.py", line 578, in main
master_install(self)
File "/usr/lib/python2.7/site-packages/ipaserver/install/server/install.py", line 248, in decorated
func(installer)
File "/usr/lib/python2.7/site-packages/ipaserver/install/server/install.py", line 758, in install
setup_pkinit=not options.no_pkinit)
File "/usr/lib/python2.7/site-packages/ipaserver/install/dsinstance.py", line 344, in create_instance
self.start_creation(runtime=30)
File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 504, in start_creation
run_step(full_msg, method)
File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 494, in run_step
method()
File "/usr/lib/python2.7/site-packages/ipaserver/install/dsinstance.py", line 559, in __create_instance
raise RuntimeError("failed to create DS instance %s" % e)

2017-10-04T15:55:56Z DEBUG The ipa-server-install command failed, exception: RuntimeError: failed to create DS instance Command '/usr/sbin/setup-ds.pl --silent --logfil e - -f /tmp/tmpPQPUX_' returned non-zero exit status 1
2017-10-04T15:55:56Z ERROR failed to create DS instance Command '/usr/sbin/setup-ds.pl --silent --logfile - -f /tmp/tmpPQPUX_' returned non-zero exit status 1
2017-10-04T15:55:56Z ERROR The ipa-server-install command failed. See /var/log/ipaserver-install.log for more information

Expected results

Success:)


https://pagure.io/389-ds-base/blob/master/f/ldap/servers/slapd/slapi_pal.c#_203

First, the issue look like OEL doesn't provide a correct meminfo file because it lacks memAvailable.

Second though, it looks like we are setting a value that could be too small instead. But this shouldn't happen because we read memTotal and we fall back to that as the ultimate cap available.

https://pagure.io/389-ds-base/blob/master/f/ldap/servers/slapd/slapi_pal.c#_227

So I suspect here you are getting an rlimit or a cgroup limit. Can you check:

/sys/fs/cgroup/memory/memory.soft_limit_in_bytes
/sys/fs/cgroup/memory/memory.limit_in_bytes
/etc/security/limits.conf

It might also be a systemd memory limit, but I don't know how those are configured, you may need to do something like:

sudo systemctl cat dirsrv@.service

Metadata Update from @firstyear:
- Custom field component adjusted to None
- Custom field origin adjusted to None
- Custom field reviewstatus adjusted to None
- Custom field type adjusted to None
- Custom field version adjusted to None

6 years ago

Metadata Update from @firstyear:
- Issue assigned to firstyear

6 years ago

Perhaps a generic solution is to have a minimum output on memavail, IE 64MB or similar, so that we always return something?

 sudo cat /sys/fs/cgroup/memory/memory.limit_in_bytes
[sudo] password for william: 
9223372036854771712

AHHH I know what it is. Your memavail is 0, so you read int max from cgroup, then:

[04/Oct/2017:11:55:53.963334442 -0400] - INFO - check_and_set_import_cache - pagesize: 4096, available bytes 9223372030878334975, process usage 13570048
[04/Oct/2017:11:55:53.964858651 -0400] - INFO - check_and_set_import_cache - Import allocates 144115185741308KB import cache.

So what needs to happen is if memavail == 0, memavail = memtotal.

Metadata Update from @firstyear:
- Custom field reviewstatus adjusted to review (was: None)
- Issue set to the milestone: 1.3.7.0

6 years ago

Metadata Update from @mreynolds:
- Custom field reviewstatus adjusted to ack (was: review)

6 years ago

cat /sys/fs/cgroup/memory/memory.soft_limit_in_bytes
9223372036854775807

cat /sys/fs/cgroup/memory/memory.soft_limit_in_bytes
9223372036854775807

cat /etc/security/limits.conf

/etc/security/limits.conf

This file sets the resource limits for the users logged in via PAM.

It does not affect resource limits of the system services.

Also note that configuration files in /etc/security/limits.d directory,

which are read in alphabetical order, override the settings in this

file in case the domain is the same or more specific.

That means for example that setting a limit for wildcard domain here

can be overriden with a wildcard setting in a config file in the

subdirectory, but a user specific setting here can be overriden only

with a user specific setting in the subdirectory.

Each line describes a limit for a user in the form:

<domain> <type> <item> <value>

Where:

<domain> can be:

- a user name

- a group name, with @group syntax

- the wildcard *, for default entry

- the wildcard %, can be also used with %group syntax,

for maxlogin limit

<type> can have the two values:

- "soft" for enforcing the soft limits

- "hard" for enforcing hard limits

<item> can be one of the following:

- core - limits the core file size (KB)

- data - max data size (KB)

- fsize - maximum filesize (KB)

- memlock - max locked-in-memory address space (KB)

- nofile - max number of open file descriptors

- rss - max resident set size (KB)

- stack - max stack size (KB)

- cpu - max CPU time (MIN)

- nproc - max number of processes

- as - address space limit (KB)

- maxlogins - max number of logins for this user

- maxsyslogins - max number of logins on the system

- priority - the priority to run user process with

- locks - max number of file locks the user can hold

- sigpending - max number of pending signals

- msgqueue - max memory used by POSIX message queues (bytes)

- nice - max nice priority allowed to raise to values: [-20, 19]

- rtprio - max realtime priority

<domain> <type> <item> <value>

* soft core 0

* hard rss 10000

@student hard nproc 20

@faculty soft nproc 20

@faculty hard nproc 50

ftp hard nproc 0

@student - maxlogins 4

End of file

commit 0ea523c
To ssh://git@pagure.io/389-ds-base.git
4cd1a24..0ea523c master -> master

commit 87c118c
To ssh://git@pagure.io/389-ds-base.git
adc6a8b..87c118c 389-ds-base-1.3.7 -> 389-ds-base-1.3.7

commit eaede00
To ssh://git@pagure.io/389-ds-base.git
6cb1999..eaede00 389-ds-base-1.3.6 -> 389-ds-base-1.3.6

Metadata Update from @firstyear:
- Issue close_status updated to: fixed
- Issue status updated to: Closed (was: Open)

6 years ago

389-ds-base is moving from Pagure to Github. This means that new issues and pull requests
will be accepted only in 389-ds-base's github repository.

This issue has been cloned to Github and is available here:
- https://github.com/389ds/389-ds-base/issues/2451

If you want to receive further updates on the issue, please navigate to the github issue
and click on subscribe button.

Thank you for understanding. We apologize for all inconvenience.

Metadata Update from @spichugi:
- Issue close_status updated to: wontfix (was: fixed)

3 years ago

Login to comment on this ticket.

Metadata