#49562 integrate changelog database to main database
Opened 2 years ago by lkrispen. Modified a month ago

On masters and hubs the multimaster replication plugin maintains a changelog database containing the changes to be replayed to other servers. Historically this was a separate databse environment, but with the need to write the changelog in the same transaction as the change operation it was changed to use the main database environment.

But it still maintains the changelog for all backends in a single directory and manages th file names to open and close the changelog database files. This has an overhead in many respects, but it also prevents the move to a replacable backend database library, eg LMDB does not allow the use of dedicated database files.

Therefor, as a first step, the changlog database files should be fully integrated to the backend database.

Design document (in progress): http://www.port389.org/docs/389ds/design/integrate-changelog-database-and-backend-database.html

I will upload my current patches, which are incomplete, but allow to assess what direction it goes. With the three patches normal replication, upgrade and ldif export should work.
What is missing is a cleanup of locking and managing cl access for state changes.

Metadata Update from @lkrispen:
- Custom field component adjusted to None
- Custom field origin adjusted to None
- Custom field reviewstatus adjusted to None
- Custom field type adjusted to None
- Custom field version adjusted to None

2 years ago

Metadata Update from @mreynolds:
- Issue set to the milestone: 1.4.0

2 years ago

Metadata Update from @mhonek:
- Issue assigned to mhonek

a year ago

Metadata Update from @lkrispen:
- Issue assigned to lkrispen (was: mhonek)

7 months ago

I will continue on this.

There is one topic related to the changelog I would like to discuss and to address in this work: changelog export/import.

The changelog export might have some interest on its own, to check changes like in the audit log (although there is the audit log and dbscan).
The changelog import has only one use case: if changelog encryption is enabled reimport when certificate change makes the encryption key unusable.

There is a ticket to avoid the encrypting of the symmetric key #49525, but this is not decided yet and even if behaviour will change at least one reimport will be needed.

Now to my issue: we offer a separate changelog export/import online, but in my opinion this does not guarantee consistency of database and changelog, between export and import there could be changes, even during the export there could be changes which are not yet commited and will be missed in the export.

If the changelog is moved to the database I would like to allow only a synchronous import of database and changelog. I suggest to extend the "db2ldif" mode to also export the changelog and to import this combo.
The online export task, for informational use, could remain as it is

Metadata Update from @vashirov:
- Issue priority set to: normal
- Issue set to the milestone: 1.4.3 (was: 1.4.0)

a month ago

Login to comment on this ticket.