#49562 integrate changelog database to main database
Closed: wontfix 3 years ago by tbordaz. Opened 6 years ago by lkrispen.

On masters and hubs the multimaster replication plugin maintains a changelog database containing the changes to be replayed to other servers. Historically this was a separate databse environment, but with the need to write the changelog in the same transaction as the change operation it was changed to use the main database environment.

But it still maintains the changelog for all backends in a single directory and manages th file names to open and close the changelog database files. This has an overhead in many respects, but it also prevents the move to a replacable backend database library, eg LMDB does not allow the use of dedicated database files.

Therefor, as a first step, the changlog database files should be fully integrated to the backend database.

Design document (in progress): http://www.port389.org/docs/389ds/design/integrate-changelog-database-and-backend-database.html


I will upload my current patches, which are incomplete, but allow to assess what direction it goes. With the three patches normal replication, upgrade and ldif export should work.
What is missing is a cleanup of locking and managing cl access for state changes.

Metadata Update from @lkrispen:
- Custom field component adjusted to None
- Custom field origin adjusted to None
- Custom field reviewstatus adjusted to None
- Custom field type adjusted to None
- Custom field version adjusted to None

6 years ago

Metadata Update from @mreynolds:
- Issue set to the milestone: 1.4.0

6 years ago

Metadata Update from @mhonek:
- Issue assigned to mhonek

5 years ago

Metadata Update from @lkrispen:
- Issue assigned to lkrispen (was: mhonek)

4 years ago

I will continue on this.

There is one topic related to the changelog I would like to discuss and to address in this work: changelog export/import.

The changelog export might have some interest on its own, to check changes like in the audit log (although there is the audit log and dbscan).
The changelog import has only one use case: if changelog encryption is enabled reimport when certificate change makes the encryption key unusable.

There is a ticket to avoid the encrypting of the symmetric key #49525, but this is not decided yet and even if behaviour will change at least one reimport will be needed.

Now to my issue: we offer a separate changelog export/import online, but in my opinion this does not guarantee consistency of database and changelog, between export and import there could be changes, even during the export there could be changes which are not yet commited and will be missed in the export.

If the changelog is moved to the database I would like to allow only a synchronous import of database and changelog. I suggest to extend the "db2ldif" mode to also export the changelog and to import this combo.
The online export task, for informational use, could remain as it is

Metadata Update from @vashirov:
- Issue priority set to: normal
- Issue set to the milestone: 1.4.3 (was: 1.4.0)

4 years ago

@lkrispen a question. If an admin took backup before that change is applied. So backup with database and changelog that are not in the same directory. Can the backup be restored on a version running with this ticket.

@tbordaz Do you mean this order of operations?

  • Backup is taken (be and seperate cl)
  • 389 is upgraded (be and cl merged)
  • backup is restored

@firstyear exactly. Thanks for making it more clear. I think it is difficult to achieve and not sure it worth the effort, but just wanted to be sure of the status of old backup with this ticket.

@tbordaz I would think in this case the old backup would not work ... we could advise the use of db2ldif -r instead?

@tbordaz I would think in this case the old backup would not work ... we could advise the use of db2ldif -r instead?

you are getting to the hard stuff :-)

First, I have to admit I did not test it, I was happy to remove the tons of extra code from backup/restore to handle the changelog db dir. And id not think of a reuse of an old backup.

What I think will happen is that the changelog in the backup will be ignored and at startup the changelog for the backend will be recreated.
Using db2ldif -r will not make a difference, it will contain repl meta data in the ldif, but not the changelog, after import it will be recreated as well.

In the new code I have a db2ldif option -R which also does an ldif export of the changelog and can be reimported, but this does not help in the described scenario.

So there are two options:
1] accept the loss of changelog when rstoring an old backup, doc and warn ...
2] make it work. When the new code is staring the first time it does an upgrade and renames and moves the changelog db, this could probably be done for restore, but would bring back some legacy code, need to check how much effort it would be.

@lkrispen Thanks for the precise explanations.
I vote for option 1), make it simple. IMHO it is acceptable to have this limitation on old backup.

I was even considering a 3rd option: old backup are not supported.
Do you anticipate any risk to restore old backup (with loose of changelog) with the new layout ?

Just a dummy question, taking a old backup db_env refers to the old changelog. Is it feasible to "simply" move/del_add the old changelog to the db location ?

I think it will be your option 3]

Things are worse than I expected. A backup also contains the transaction logs which will be read in the recovery phase of the restore. And they contain references to the old changelog files - and restore will fail.

But there is even another problem: if the server is started the first time with the new changelog code on an existing instance it will automatically migrate the existing changelog from the old location to the backend - using bdb commands - and leaving traces of the old file in the txn logs. It can eventaually cause problems on later backup/restore, probably depending on the number of checkpoints written. needs further investigation

If it will be option 3 we probably need to have a way to refuse the attempt to restore to prevent these bad states occuring. And probably a path to complete a restore in that situation.

Changelog moving from global config entry to per backend config entry, the CLI is also impacted with the ticket https://pagure.io/389-ds-base/issue/51189

Metadata Update from @tbordaz:
- Issue close_status updated to: fixed
- Issue status updated to: Closed (was: Open)

3 years ago

389-ds-base is moving from Pagure to Github. This means that new issues and pull requests
will be accepted only in 389-ds-base's github repository.

This issue has been cloned to Github and is available here:
- https://github.com/389ds/389-ds-base/issues/2621

If you want to receive further updates on the issue, please navigate to the github issue
and click on subscribe button.

Thank you for understanding. We apologize for all inconvenience.

Metadata Update from @spichugi:
- Issue close_status updated to: wontfix (was: fixed)

3 years ago

Login to comment on this ticket.

Metadata