#49198 At backend startup, autotune access a non initialized dblayer struct
Opened 2 years ago by tbordaz. Modified 4 months ago

Issue Description

At startup of the dblayer, autotune accesses 'li->li_dblayer_private->dblayer_home_directory' that is not yet initialized.
It will be initialized in 'dblayer_start'.
ldbm_back_start

  retval = ldbm_back_start_autotune(li);
  if (retval != 0) {
      slapi_log_err(SLAPI_LOG_ERR, "ldbm_back_start", "Failed to set database tuning on backends\n");
      return SLAPI_FAIL_GENERAL;
  }                                                                                                                                                           
  ...
  retval = dblayer_start(li,DBLAYER_CLEAN_RECOVER_MODE);

This can be seen as 'dblayer_home_directory' was not yet initialized with 'li_directory'

#0  dblayer_get_home_dir (li=<optimized out>, dbhome=dbhome@entry=0x0) at ldap/servers/slapd/back-ldbm/dblayer.c:488
#1  0x00007fa8cc1204bf in dblayer_get_full_inst_dir (li=<optimized out>, inst=inst@entry=0x7fa8db2ffac0,
  buf=buf@entry=0x7ffc448ffb90 "", buflen=buflen@entry=4096) at ldap/servers/slapd/back-ldbm/dblayer.c:1161
#2  0x00007fa8cc1207ea in dblayer_get_id2entry_size (inst=inst@entry=0x7fa8db2ffac0) at ldap/servers/slapd/back-ldbm/dblayer.c:1804
#3  0x00007fa8cc1776d0 in ldbm_back_start_autotune (li=0x7fa8db155580) at ldap/servers/slapd/back-ldbm/start.c:229
#4  ldbm_back_start (pb=<optimized out>) at ldap/servers/slapd/back-ldbm/start.c:356
#5  0x00007fa8d9623a7b in plugin_call_func (list=0x7fa8db155d80, operation=operation@entry=212, pb=0x7fa8db328d40,
  call_one=call_one@entry=1) at ldap/servers/slapd/plugin.c:2072
#6  0x00007fa8d962412c in plugin_call_one (pb=<optimized out>, operation=212, list=<optimized out>)
  at ldap/servers/slapd/plugin.c:2020
#7  plugin_dependency_startall (argc=argc@entry=5, argv=argv@entry=0x7ffc449020a8, plugin_list=plugin_list@entry=0x0,
  errmsg=<optimized out>, operation=212) at ldap/servers/slapd/plugin.c:1791
#8  0x00007fa8d96246c3 in plugin_startall (argc=argc@entry=5, argv=argv@entry=0x7ffc449020a8, plugin_list=plugin_list@entry=0x0)
  at ldap/servers/slapd/plugin.c:1992
#9  0x00007fa8d9f02788 in main (argc=5, argv=0x7ffc449020a8) at ldap/servers/slapd/main.c:1089
(gdb) frame 3
#3  0x00007fa8cc1776d0 in ldbm_back_start_autotune (li=0x7fa8db155580) at ldap/servers/slapd/back-ldbm/start.c:229
229         db_size = dblayer_get_id2entry_size(inst);
(gdb) print li->li_dblayer_private->dblayer_home_directory
$4 = 0x0
(gdb) print li->li_directory
$5 = 0x7fa8db309bd0 "/var/lib/dirsrv/slapd-TESTRELM-TEST/db"

The consequence is incorrect sizing computation and alarming messages like

WARN - dblayer_get_home_dir - Db home directory is not set. Possibly nsslapd-directory (optionally nsslapd-db-home-directory) is missing in the config file.

Package Version and Platform

1.3.6 any platform

Steps to reproduce

  1. Strangely this problem is not systematic, reproduce while installing ipa (4.5) with DS 1.3.6
    2.
    3.

Actual results

Expected results


Note that this bug is possibly related to RH BZ 1435122

Metadata Update from @tbordaz:
- Custom field reviewstatus adjusted to new
- Custom field type adjusted to defect

2 years ago

Hmmm okay. Do you want me to investigate this?

The autotuning is just showing the problem, but it's not the cause. looking at this, auto tune is run after ldbm_config_internal_set(li, CONFIG_DIRECTORY, "get default");, and this function is meant to set li->li_directory. So the issue is probably actually in ldbm_config_directory_set() in ldbm_config.c not setting li->li_directory correctly.

We can see this because the message

WARN - dblayer_get_home_dir - Db home directory is not set. Possibly nsslapd-directory (optionally nsslapd-db-home-directory) is missing in the config file.

Occurs inside of ldbm_config_directory_set(), which indicates that the following condition is violated:

 slapi_entry_attr_find(entries[0], "nsslapd-directory", &attr); 
...
s = slapi_value_get_string( v )
...
if (NULL == s || '\0' == s || 0 == PL_strcmp(s, "(null)")) {
     slapi_log_err(SLAPI_LOG_ERR,

This indicates there is an issue with the nsslapd-directory attribute in dse.ldif.

I am not sure the issue comes from setting of li->li_directory.
In process I debugged ("Issue Description"), it was correctly set in ldbminfo (li) but not yet copied in dblayer_private (li->li_dblayer_private). This is done in dblayer_start.

My understanding is that autotune, calls dblayer_get_id2entry_size/dblayer_get_full_inst_dir/dblayer_get_home_dir that access li->li_dblayer_private before dblayer_start is called. So li->li_dblayer_private->dblayer_home_directory is not yet set.

Note: Reading the code, I found it as minor issue (just an incorrect log).
However if I completely prevent the call to autotune, it fixes 1435122.
So either there is a side effect of 49198 or an other issue in autotune.

Ahhh, I missed it was dblayer_private.

Well, the autotuning code has always called id2entry, that was there before I started changing the autotune. So this has been a problem for a long time. Either it's a race of some kind, or its an issue with how this backend is being made during ipa creation. I've not seen the issue with offline or online creation so far, so I wonder what the issue is.

Arguable, shouldn't we set dblayer_home_directory in private before we start? Is there a reason not to?

The following TEST patch is preventing

WARN - dblayer_get_home_dir - Db home directory is not set. Possibly nsslapd-directory (optionally nsslapd-db-home-directory) is missing in the config file.

The attached patch is just a test one where it is using li->li_directory if dblayer_home_directory is not set. I have no strong opinion whether it is better to fallback to li_directory or rely on previously set dblayer_home_directory

BUT this patch does NOT fix RH BZ 1435122. It exists a side effect of calling autotune that prevent the access to the suffix. I will open a separated ticket for that.

0001-49198-test-fix.patch

Thinking about this issue, does id2entry exist on the affected system? What if the issue is during setup-ds.pl, the ldif2db is failing to import the db, which leaves id2entry un-created.

That seems to be the only idea I have as to why that value isn't populated,

Closing as this was fixed via issue 49204

Metadata Update from @mreynolds:
- Custom field reviewstatus reset (from new)
- Issue close_status updated to: duplicate
- Issue status updated to: Closed (was: Open)

2 years ago

Metadata Update from @tbordaz:
- Issue status updated to: Open (was: Closed)

2 years ago

Metadata Update from @tbordaz:
- Issue assigned to tbordaz

2 years ago

Reopening this ticket as it was a different issue than 49204, although it was found during investigation of 49204.
IMHO the attached fix is valid as until backend is startup, private dblayer_home_directory is not initialized and attempt to read it will return NULL home directory.

This bug is minor because so far it only log unexpected/scary message (i.e. "Db home directory is not set.").

Metadata Update from @tbordaz:
- Custom field origin adjusted to IPA
- Custom field reviewstatus adjusted to review

2 years ago

Just to confirm, home_dir and li->li_directory are the same values yes? Otherwise, I'm happy with this fix.

Metadata Update from @firstyear:
- Custom field reviewstatus adjusted to ack (was: review)

2 years ago

Checking that li_directory is equivalent to dblayer_home_directory is not immediate.

Under normal run, dblayer_home_directory is set to li_directory (nsslapd-directory). Now the function dblayer_get_home_dir can return nsslapd-db-home-directory, even if dblayer_home_directory==li_directory. Using nsslapd-db-home-directory looks quite experimental and if someone is still using it.
So under normal run, I would say yes: dblayer_home_directory==li_directory.

Under restore/backup/export/import/online_index, it looks to be the same.
Under offline index, I think it can be different where it uses a separated environment <nsslapd-directory's parent="">/dbenv.

I will rework the fix and see if I can make it more specific to the use case: how to run autotune before dblayer_start is called.

Metadata Update from @mreynolds:
- Issue set to the milestone: 1.3.6.0

2 years ago

Metadata Update from @tbordaz:
- Issue set to the milestone: 1.3.7 backlog (was: 1.3.6.0)

2 years ago

@tbordaz - is this still applicable? In 1.4.2 I want the db home set by default, so it would be nice to get his in.

Metadata Update from @mreynolds:
- Issue set to the milestone: 1.4.2 (was: 1.3.7 backlog)

4 months ago

Login to comment on this ticket.

Metadata