Ticket 48383 - import tasks with dynamic buffer sizes
Bug Description: db2index, ldif2db and others all use a fifo buffer max size
defined by:
./ldap/servers/slapd/back-ldbm/import.c:48:
job->fifo.bsize = (inst->inst_cache.c_maxsize/10) << 3;
Where c_maxsize is the value of the cn=ldbm backend's cachememsize value
This is very arbitrary, and in a database with few, but large objects, it can
cause the admin to increase cachememsize to a value that could cause an OOM
situation.
Worse, as an admin, because this is effectively .8 of the cachememsize, it's
hard to determine the origin of the message in the error:
import_log_notice(job, "WARNING: skipping entry \"%s\"", slapi_entry_get_dn(e));
import_log_notice(job, "REASON: entry too large (%lu bytes) for "
"the buffer size (%lu bytes)", (long unsigned int)newesize, (long unsigned int)job->fifo.bsize);
Fix Description: We now change two things:
* First we set bsize to be the backend cachesize.
* Second, when we go to use the buffer, we check if the entry will fit into
it. If it won't fit, we trigger a check of avaliable bytes of ram. If we have
sufficient we resize the buffer accordingly. If we have insufficent, we skip
the entry as before, with a large error message:
[22/Feb/2016:14:29:47 +1000] - import userRoot: REASON: entry too large
(10020345 bytes) for the buffer size (409600 bytes), and we were UNABLE
to expand buffer.
[22/Feb/2016:14:29:47 +1000] - import userRoot: CRITICAL: skipping entry
"uid=user8,ou=People,dc=example,dc=com" ending line 1042454 of file
"/tmp/slapd-standalone.bck/var/lib/dirsrv/slapd-standalone/ldif/standalone.ldif"
This will make *all* our command line tools more robust, able to function in a
greater variety of environments, and will make the admin experience better.
https://fedorahosted.org/389/ticket/48383
Author: wibrown
Review by: nhosoi, lkrispen (Thanks!)