The fix looks good to me. ACK

During the review I noticed that sometime (e.g. bdb_get_home_dir(), no_diskspace()), it is using li->li_directory to retrieve BDB_CONFIG(li)->bdb_home_directory. I guess they are equal but from different structure. Could it be changed to only use BDB_CONFIG(li)->bdb_home_directory ?

The fix looks good to me. ACK

thanks, lets wait for Marks tests

During the review I noticed that sometime (e.g. bdb_get_home_dir(), no_diskspace()), it is using li->li_directory to retrieve BDB_CONFIG(li)->bdb_home_directory. I guess they are equal but from different structure. Could it be changed to only use BDB_CONFIG(li)->bdb_home_directory ?

yes. will look into it

Something is still off, look like its the transaction log that is still in the db_home_dir.

So I ran "dscreate interactive" and choose to create the sample entries, and at startup I get an operations error

[06/Dec/2019:08:29:26.417827172 -0500] conn=2 op=7 ADD dn="dc=kvm-01-guest27,dc=lab,dc=eng,dc=rdu2,dc=redhat,dc=com"
[06/Dec/2019:08:29:26.422793357 -0500] conn=2 op=7 RESULT err=1 tag=105 nentries=0 etime=0.005032307
[06/Dec/2019:08:29:26.455490174 -0500] conn=2 op=8 UNBIND
[06/Dec/2019:08:29:26.455505484 -0500] conn=2 op=8 fd=65 closed - U1

[06/Dec/2019:08:29:26.402547929 -0500] - ERR - bdb_instance_start - dbp->open("userroot/id2entry.db") failed: No such file or directory (2)
[06/Dec/2019:08:29:26.403191026 -0500] - ERR - bdb_instance_start - Instance userroot is not registered as a db data directory. Please restart the server to create it.
[06/Dec/2019:08:29:26.403797436 -0500] - ERR - bdb_instance_start - Failure No such file or directory (2)
[06/Dec/2019:08:29:26.404400053 -0500] - ERR - ldbm_instance_postadd_instance_entry_callback - ldbm_instnace_start (userroot) failed (2)
[06/Dec/2019:08:29:26.406009447 -0500] - ERR - dblayer_open_file - The instance path /var/lib/dirsrv/slapd-localhost/db is not registered for db_data_dir, although userroot is a relative path.
[06/Dec/2019:08:29:26.406879313 -0500] - ERR - entryrdn_index_read_ext - Opening the index failed: BDB0092 Unknown error: -1(-1)
[06/Dec/2019:08:29:26.407446309 -0500] - ERR - dn2entry_ext - Failed to get id for dc=kvm-01-guest27,dc=lab,dc=eng,dc=rdu2,dc=redhat,dc=com from entryrdn index (-1)
[06/Dec/2019:08:29:26.407925698 -0500] - ERR - dblayer_open_file - The instance path /var/lib/dirsrv/slapd-localhost/db is not registered for db_data_dir, although userroot is a relative path.
[06/Dec/2019:08:29:26.408438041 -0500] - ERR - entryrdn_index_read_ext - Opening the index failed: BDB0092 Unknown error: -1(-1)
[06/Dec/2019:08:29:26.408865318 -0500] - ERR - dn2entry_ext - Failed to get id for dc=kvm-01-guest27,dc=lab,dc=eng,dc=rdu2,dc=redhat,dc=com from entryrdn index (-1)
[06/Dec/2019:08:29:26.409465499 -0500] - ERR - dblayer_open_file - The instance path /var/lib/dirsrv/slapd-localhost/db is not registered for db_data_dir, although userroot is a relative path.
[06/Dec/2019:08:29:26.410268551 -0500] - ERR - entryrdn_index_read_ext - Opening the index failed: BDB0092 Unknown error: -1(-1)
[06/Dec/2019:08:29:26.410813474 -0500] - ERR - dn2entry_ext - Failed to get id for dc=kvm-01-guest27,dc=lab,dc=eng,dc=rdu2,dc=redhat,dc=com from entryrdn index (-1)
[06/Dec/2019:08:29:26.417919523 -0500] - ERR - dblayer_open_file - The instance path /var/lib/dirsrv/slapd-localhost/db is not registered for db_data_dir, although userroot is a relative path.
[06/Dec/2019:08:29:26.418742251 -0500] - ERR - entryrdn_index_read_ext - Opening the index failed: BDB0092 Unknown error: -1(-1)
[06/Dec/2019:08:29:26.419298503 -0500] - ERR - dn2entry_ext - Failed to get id for dc=kvm-01-guest27,dc=lab,dc=eng,dc=rdu2,dc=redhat,dc=com from entryrdn index (-1)
[06/Dec/2019:08:29:26.419896892 -0500] - ERR - dblayer_open_file - The instance path /var/lib/dirsrv/slapd-localhost/db is not registered for db_data_dir, although userroot is a relative path.
[06/Dec/2019:08:29:26.420401750 -0500] - ERR - get_copy_of_entry - Operation error fetching Null DN (6c21bf27-182c11ea-92db8c5e-d4b76509), error -1.
[06/Dec/2019:08:29:26.420825659 -0500] - ERR - dblayer_open_file - The instance path /var/lib/dirsrv/slapd-localhost/db is not registered for db_data_dir, although userroot is a relative path.
[06/Dec/2019:08:29:26.421227358 -0500] - ERR - entryrdn_index_read_ext - Opening the index failed: BDB0092 Unknown error: -1(-1)
[06/Dec/2019:08:29:26.421678710 -0500] - ERR - dn2entry_ext - Failed to get id for dc=kvm-01-guest27,dc=lab,dc=eng,dc=rdu2,dc=redhat,dc=com from entryrdn index (-1)
[06/Dec/2019:08:29:26.422078333 -0500] - ERR - get_copy_of_entry - Operation error fetching dc=kvm-01-guest27,dc=lab,dc=eng,dc=rdu2,dc=redhat,dc=com (null), error -1.

ls /var/lib/dirsrv/slapd-localhost/db/userroot/

DBVERSION id2entry.db

ls /dev/shm/dirsrv/slapd-localhost/

__db.001 __db.002 __db.003 DBVERSION log.0000000001

this is a bit weird, yesterday I had seen log.00001 also in /dev/shm, but with this patch it was in the db directory, maybe I need to try to modify the template dse.ldif.

Could you post what you have in your dse.ldif for

cn=config,cn=ldbm database,....
cn=bdb,cn=config,cn=ldbm database,....

ok, now I added the nsslapd-db-home-directory to the template dse.ldif and get the txn log unde /dev/shm,

but the test, starting with an import, succeeds

looks like a bit more investigation needed

this is a bit weird, yesterday I had seen log.00001 also in /dev/shm, but with this patch it was in the db directory, maybe I need to try to modify the template dse.ldif.
Could you post what you have in your dse.ldif for
cn=config,cn=ldbm database,....
cn=bdb,cn=config,cn=ldbm database,....

dn: cn=config,cn=ldbm database,cn=plugins,cn=config
...
nsslapd-directory: /var/lib/dirsrv/slapd-localhost/db
nsslapd-backend-implement: bdb

dn: cn=bdb,cn=config,cn=ldbm database,cn=plugins,cn=config
objectClass: extensibleobject
objectClass: top
cn: bdb
...
nsslapd-db-logdirectory: /var/lib/dirsrv/slapd-localhost/db
nsslapd-db-home-directory: /dev/shm/dirsrv/slapd-localhost

I discovered two issues,
- if nsslapd-db-logdirectory is not set, we don't set it in the DBENV, so BDB will use the dbhome directory. As long as db-dir and db-home dir are the same its fine, but otherwise txn logs will go into the wrong dir. I have a fix for that.

• if ns-slapd ldif2db is run as first command for a new instance, it will try to autoupgrade and use the settings (eg db-home) found, and create the cn=bdb entry.
  But in ldif2db mode the dse.ldif is not written. It didn#t have an impact when db-home was not set, but now there is a strange behaviour that after the next start we will have set:
  nsslapd-db-home-directory: (null)

I do not yet understand this, but will try to write the dse.ldif after auto upgrade

I think with the ldif2db issue, what situation occurs where we run ldif2db as the first command? I think in dscreate we don't have the capability to import via ldif2db during setup so this situation "shouldn't" occur. So I'm curious about how it does happen and why?

I fixed a few more issues handling homedir and log dir and running upgrade in import mode.

Some of these issue already exist in 1.4.1, I am still investigating another issue, which also always existed:

If you start the server, run an online restore, the db-home-directory will be reset to (null). This happens because in the first phase of online restore dblayer_close() is called and the db-home-dir and data dir are freed and not properly rest afterwards. If then the dse.ldif is written again it will write NULL for nsslapd-db-home-directory (and after restart will fail).

Need to investigate and thik about this more.

Is there a need to re-write the dse.ldif after the online restore? IE are details of the server changing as we perform the restore?

The hacky solution is obviously to cache those attributes and re-add them to the config before the dse.ldif is written. But maybe I'm missing something ....

Is there a need to re-write the dse.ldif after the online restore? IE are details of the server changing as we perform the restore?
The hacky solution is obviously to cache those attributes and re-add them to the config before the dse.ldif is written. But maybe I'm missing something ....

There is no specific writing of the dse.ldif in the online restore, the change you are probably referring to is in bdb_config_upgrade_dse_info() and this is only called if the dse is upgraded from an older state, and then it has to be written, but this only will happen in normal mode or in any offline mode. For an online restore the server was already started in normal mode and the dse.ldif was written or not.

The problem with online restore is that the databases are closed with dblayer_close() and the the restore is performed and the instances are restarted, but some config settings eg dbhome were lost. Either we prevent the unsetting of this in dblayer_close under some conditions or reread the dse.ldif to set the correct paths again - need to find the most simple variant.

BTW, I think this problem was always there if you have set db_home_directory to a special path and run on online restore, but this seems to be a rare combination

@lkrispen, the patch looks good to me but I am sorry to jump with a new question..

dont_ever_write_dse_files is set by default (prevent write of dse.ldif). It is reset (allow write of dse.ldif) only for normal/upgrade_db/restore.

The patch adds dse_unset_dont_ever_write_dse_files (allow write of dse.ldif) in bdb_config_upgrade_dse_info. This function is called by dblayer_setup/bdb_config_load_dse_info AT THE CONDITION there is no bdb entry.
dblayer_setup is called during several execmode (import/archive/export/index...).
So I wonder if the patch can allow write of dse.ldif in execmode that were prevented before.

dblayer_setup is called during several execmode (import/archive/export/index...).
So I wonder if the patch can allow write of dse.ldif in execmode that were prevented before.

yes, that is the intention of the patch, befor in these exec modes there was no change of dse.ldif expected and the writing of the dse.ldif was prevented by default. With the patch now if there IS a change we also need to write it

:) I knew it was a stupid question !
Does it remain a case where we do not want to write dse.ldif ? If not we could simply get rid of dont_ever_write_dse_files flag/setting

I think the main reason to avoid writing the dse.ldif is that it always has to write the complete file and this is a big overhead, so if it can be avoided it should be avoided

@lkrispen thanks for these explanations. The patch looks good to me, let's see if @mreynolds and @firstyear raise others concerns.

added a fix for the online restore problem

I have no further concerns :)

@lkrispen Maybe this isn't working for me because it has not been merged yet? :-)

@lkrispen Maybe this isn't working for me because it has not been merged yet? :-)

it isn't merged because I was waiting to see if it works for you :-)

This still does not work. When creating an instance it fails with an "operations error"

Errors log:

[05/Mar/2020:14:53:34.470949921 -0500] - ERR - bdb_instance_start - dbp->open("userroot/id2entry.db") failed: No such file or directory (2)
[05/Mar/2020:14:53:34.471491944 -0500] - ERR - bdb_instance_start - Instance userroot is not registered as a db data directory. Please restart the server to create it.
[05/Mar/2020:14:53:34.472108550 -0500] - ERR - bdb_instance_start - Failure No such file or directory (2)
[05/Mar/2020:14:53:34.472627081 -0500] - ERR - ldbm_instance_postadd_instance_entry_callback - ldbm_instnace_start (userroot) failed (2)
[05/Mar/2020:14:53:34.474314217 -0500] - ERR - dblayer_open_file - The instance path /var/lib/dirsrv/slapd-localhost/db is not registered for db_data_dir, although userroot is a relative path.
[05/Mar/2020:14:53:34.475116943 -0500] - ERR - entryrdn_index_read_ext - Opening the index failed: BDB0092 Unknown error: -1(-1)
[05/Mar/2020:14:53:34.475746430 -0500] - ERR - dn2entry_ext - Failed to get id for o=mark from entryrdn index (-1)
[05/Mar/2020:14:53:34.476279636 -0500] - ERR - dblayer_open_file - The instance path /var/lib/dirsrv/slapd-localhost/db is not registered for db_data_dir, although userroot is a relative path.
[05/Mar/2020:14:53:34.476744923 -0500] - ERR - entryrdn_index_read_ext - Opening the index failed: BDB0092 Unknown error: -1(-1)
[05/Mar/2020:14:53:34.477191518 -0500] - ERR - dn2entry_ext - Failed to get id for o=mark from entryrdn index (-1)
[05/Mar/2020:14:53:34.477713715 -0500] - ERR - dblayer_open_file - The instance path /var/lib/dirsrv/slapd-localhost/db is not registered for db_data_dir, although userroot is a relative path.
[05/Mar/2020:14:53:34.478614604 -0500] - ERR - entryrdn_index_read_ext - Opening the index failed: BDB0092 Unknown error: -1(-1)
[05/Mar/2020:14:53:34.479349849 -0500] - ERR - dn2entry_ext - Failed to get id for o=mark from entryrdn index (-1)
[05/Mar/2020:14:53:34.484491264 -0500] - ERR - dblayer_open_file - The instance path /var/lib/dirsrv/slapd-localhost/db is not registered for db_data_dir, although userroot is a relative path.
[05/Mar/2020:14:53:34.485764183 -0500] - ERR - entryrdn_index_read_ext - Opening the index failed: BDB0092 Unknown error: -1(-1)
[05/Mar/2020:14:53:34.489157235 -0500] - ERR - dn2entry_ext - Failed to get id for o=mark from entryrdn index (-1)
[05/Mar/2020:14:53:34.490733938 -0500] - ERR - dblayer_open_file - The instance path /var/lib/dirsrv/slapd-localhost/db is not registered for db_data_dir, although userroot is a relative path.
[05/Mar/2020:14:53:34.491372500 -0500] - ERR - get_copy_of_entry - Operation error fetching Null DN (fefcf327-5f1a11ea-a98eebd5-107f6ab8), error -1.
[05/Mar/2020:14:53:34.491956523 -0500] - ERR - dblayer_open_file - The instance path /var/lib/dirsrv/slapd-localhost/db is not registered for db_data_dir, although userroot is a relative path.
[05/Mar/2020:14:53:34.492524119 -0500] - ERR - entryrdn_index_read_ext - Opening the index failed: BDB0092 Unknown error: -1(-1)
[05/Mar/2020:14:53:34.493065612 -0500] - ERR - dn2entry_ext - Failed to get id for o=mark from entryrdn index (-1)
[05/Mar/2020:14:53:34.493607443 -0500] - ERR - get_copy_of_entry - Operation error fetching o=mark (null), error -1.

# ls /dev/shm/dirsrv/slapd-localhost/
__db.001  __db.002  __db.003  DBVERSION

This looks good to me....

# ls /var/lib/dirsrv/slapd-localhost/db
log.0000000001  userroot
# ls /var/lib/dirsrv/slapd-localhost/db/userroot/
DBVERSION  id2entry.db

Configuation

 dn: cn=config,cn=ldbm database,cn=plugins,cn=config
...
nsslapd-directory: /var/lib/dirsrv/slapd-localhost/db
dn: cn=userroot,cn=ldbm database,cn=plugins,cn=config
...
nsslapd-directory: /var/lib/dirsrv/slapd-localhost/db/userroot

dn: cn=bdb,cn=config,cn=ldbm database,cn=plugins,cn=config
...
nsslapd-db-logdirectory: /var/lib/dirsrv/slapd-localhost/db
nsslapd-db-home-directory: /dev/shm/dirsrv/slapd-localhost

So everything looks correct now, except for how the server is interacting with the database:

[05/Mar/2020:14:53:34.470949921 -0500] - ERR - bdb_instance_start - dbp->open("userroot/id2entry.db") failed: No such file or directory (2)
[05/Mar/2020:14:53:34.471491944 -0500] - ERR - bdb_instance_start - Instance userroot is not registered as a db data directory. Please restart the server to create it.
[05/Mar/2020:14:53:34.472108550 -0500] - ERR - bdb_instance_start - Failure No such file or directory (2)
[05/Mar/2020:14:53:34.472627081 -0500] - ERR - ldbm_instance_postadd_instance_entry_callback - ldbm_instnace_start (userroot) failed (2)
[05/Mar/2020:14:53:34.474314217 -0500] - ERR - dblayer_open_file - The instance path /var/lib/dirsrv/slapd-localhost/db is not registered for db_data_dir, although userroot is a relative path.

I have a test system you can look at if you want to look into it...

If I restart the server then I can update/initialize the database and everything works correctly. So this seems to be the critical error here:

bdb_instance_start - Instance userroot is not registered as a db data directory. Please restart the server to create it.

ACK

389-ds-base is moving from Pagure to Github. This means that new issues and pull requests
will be accepted only in 389-ds-base's github repository.

This pull request has been cloned to Github as issue and is available here:
- https://github.com/389ds/389-ds-base/issues/3825

If you want to continue to work on the PR, please navigate to the github issue,
download the patch from the attachments and file a new pull request.

Thank you for understanding. We apologize for all inconvenience.