daf40aa Ticket 48383 - import tasks with dynamic buffer sizes

Authored and Committed by William Brown 8 years ago
    Ticket 48383 - import tasks with dynamic buffer sizes
    
    Bug Description:  db2index, ldif2db and others all use a fifo buffer max size
    defined by:
    
    ./ldap/servers/slapd/back-ldbm/import.c:48:
            job->fifo.bsize = (inst->inst_cache.c_maxsize/10) << 3;
    
    Where c_maxsize is the value of the cn=ldbm backend's cachememsize value
    
    This is very arbitrary, and in a database with few, but large objects, it can
    cause the admin to increase cachememsize to a value that could cause an OOM
    situation.
    
    Worse, as an admin, because this is effectively .8 of the cachememsize, it's
    hard to determine the origin of the message in the error:
    
    import_log_notice(job, "WARNING: skipping entry \"%s\"", slapi_entry_get_dn(e));
    import_log_notice(job, "REASON: entry too large (%lu bytes) for "
      "the buffer size (%lu bytes)", (long unsigned int)newesize, (long unsigned int)job->fifo.bsize);
    
    Fix Description:  We now change two things:
    
    * First we set bsize to be the backend cachesize.
    * Second, when we go to use the buffer, we check if the entry will fit into
    it. If it won't fit, we trigger a check of avaliable bytes of ram. If we have
    sufficient we resize the buffer accordingly. If we have insufficent, we skip
    the entry as before, with a large error message:
    
    [22/Feb/2016:14:29:47 +1000] - import userRoot: REASON: entry too large
            (10020345 bytes) for the buffer size (409600 bytes), and we were UNABLE
            to expand buffer.
    [22/Feb/2016:14:29:47 +1000] - import userRoot: CRITICAL: skipping entry
            "uid=user8,ou=People,dc=example,dc=com" ending line 1042454 of file
            "/tmp/slapd-standalone.bck/var/lib/dirsrv/slapd-standalone/ldif/standalone.ldif"
    
    This will make *all* our command line tools more robust, able to function in a
    greater variety of environments, and will make the admin experience better.
    
    https://fedorahosted.org/389/ticket/48383
    
    Author: wibrown
    
    Review by: nhosoi, lkrispen (Thanks!)
    
        
file modified
+1 -1