#4302 [RFE] Move replication topology to the shared tree
Closed: Fixed None Opened 4 years ago by simo.

The current way to handle replication, is cumbersome and makes it difficult to properly handle replication topology.

Create a new replication topology subtree under the cn=etc tree that will have a simplified schema to list the replication agreements between servers, and have a global configs for things like excluded replication attributes.

The idea is that admins would change objects in this shared tree and these changes would then be reflected in actual replication agreement changes in cn=config as a consequence.

Advantages (in no particular order):

  • it becomes possible to easily visualize the topology w/o having to contact multiple servers
  • it is possible to centrally remove a server and have all other server remove the relevant replication agreement w/o having to contact every single server
  • it is possible to have a topology checker that uses graph theory
  • it is possible to use said checker to prevent split brain situations by simply denying (LDAP_UNWILLING_TO_PERFORM) changes that would break the graph
  • it is possible to create new agreements remotely w/o direct access to the replica just by virtue of replication of the shared tree.
  • it will be possible to create multiple replication typologies for different database (ie one for main tree and one for CA data) and have distinct checks for all of them.

Some or all of the following tickets would be automatically or easily
resolved: #4286, #3829, #3480, #3143, #3053, #3742(?), #3058, #2358(?), #2163,

In the first version, the Web UI should be able to display at least read only graph of the replicas.


We agreed that this will be very useful for many of our other tickets and plan to do it in 4.1, if we can squeeze it in. During 4.0 time frame we would like to have at least a design of this feature.

There are three different levels of features to implement with this request

1] providing all replication configuration information consitsent across all deployed servers on all servers, eg to easily visualize the replication topology.
It will be achieved by having the replication information in a suffix, where it will be replicated.
This should not only cover replication agreements as in the ticket description, it should also contain the replicas itself, without knowing which replicas exist its not possible to verify connectivity.

2] Allowing to do sanity checks on replication configuration, denying modifications which would break replication topology or issue warnings. It might not be possible to verify each modification isolated, eg when adding a new replica it is not connected before adding the replication agreements, but there could be a warning or help indicating which next steps would have to be performed.
This could be handled by a plugin, intercepting and verifying the modifications to objects in the shared configuration tree. Ideally the checks would not be hardcoded, but allow the definition of rules to be applied, eg topology type (ring, star, mesh,..), min connectivity,.....

3] The idea is that the configuration information in the shared tree should define the effective replication information. So a modification in the shared config should trigger a change in the replication config entries and then trigger a change in the in memory replication objects. This again could be done by a custom plugin, which analyzes the changes to the shared config and translates this to modification of the entries under cn=config.
The question is how should direct modifications be treated. Even if the advice is to only modify the shared config, it is still possible to do direct modifications of the cn=config entries or even edit the dse.ldif. A custom plugin could also capture changes to cn=config, but changes to dse.ldif are out of reach.

I think not every part of the feature has to be handled in the custom plugin, we could als ask for enhancements in 389 itself. There could be an extension to the replica and replication agreement entries to reference a master entry in the shared tree. DS would check these entries at startup or at modifications of the cn=config entries, so it would catch all direct modifications, a custom plugin could trigger a direct change or rebuild of replica config or the replication plugin could also monitor changes to the shared config objects.

So I propose to split the implementation into two parts:
1] enhance DS to handle config entries in the shared tree. This would allow to consistently handle config changes either applied to the shared config, cn=config mods or des.ldif changes.
This feature might also be interesting to other DS deployments, not only IPA
2] provide an IPA plugin to do consistency checks and topology verification

I am not convinced it is a good idea to move the actual replication agreements in the shared tree.
You'd have a ton of objects there that are only needed by a single server, and instead gets replicated around, what's the point ?

My idea to keep consistency is that at startup the replication-topology plugin reads the shared tree and overwrites any changes that may have bee made to cn=config independently. This takes care of any manual modification to dse.ldif. The replication topology plugin also monitors cn=config and prevent changes to replication agreements that it manages directly (note: not just all agreements, I think it is ok to allow custom agreements to solve special issues, if the admin knows what it is doing).

Replying to [comment:5 simo]:

I am not convinced it is a good idea to move the actual replication agreements in the shared tree.
You'd have a ton of objects there that are only needed by a single server, and instead gets replicated around, what's the point ?

I thought that's the point: to have full visibility and control of the full replication topology on each server - allowing a centralized visualization, verification and management of the replication topology.

My idea to keep consistency is that at startup the replication-topology plugin reads the shared tree and overwrites any changes that may have bee made to cn=config independently. This takes care of any manual modification to dse.ldif. The replication topology plugin also monitors cn=config and prevent changes to replication agreements that it manages directly (note: not just all agreements, I think it is ok to allow custom agreements to solve special issues, if the admin knows what it is doing).
In may proposal you also have custom agreements, only if you reference an agreement entry it will be used. This will assure that the 'desired' configuration is always used. If you want to handle it at startup, the order of the startup of the replication plugin and repl-topology plugin is a challenge

Just to clarify, there are two issues and my proposal has two parts:

1] Add a config attribute to the replication agreement entry cn=config
nsslapd-replica-agmt-configdn: <dn of="" a="" configuration="" entry="">

at startup or when the attr is modified the repl agreement object is built based on this entry. If the attr doesn't exist, the agreement is created as usual

2] Decide where these config entries live, only on on server or replicated

The ideas of the replicated tree was to allow to manage the topology from one place, i.e when you are connected to any replica. If I want to see how replicas are connected - topology, add or remove an agreement, ping replica or even turn it off from one interface the data stored in shared tree should allow me to do so.

Replying to [comment:8 dpal]:

The ideas of the replicated tree was to allow to manage the topology from one place, i.e when you are connected to any replica. If I want to see how replicas are connected - topology, add or remove an agreement, ping replica or even turn it off from one interface the data stored in shared tree should allow me to do so.

yes, that is my understanding. But if you are connected to replicaA and wantto modify a replication agreement for replicaB this info needs to be available an replicaA and teh change replicated to replicaB (and others)

Simo's plan was to have a 389-ds plugin that would monitor changes to this area of the tree and apply them into cn=config as needed. This is how you can manage agreements on serverB from serverA.

Rob, but in comment 6 Simo said he doesn't want the info to be replicated, that's whay we have this discussion

Replying to [comment:11 lkrispen]:

Rob, but in comment 6 Simo said he doesn't want the info to be replicated, that's whay we have this discussion

I did not say that, and here lie the confusion.

What I said is that I do not want the entries in cn=config to be put directly in the shared tree as they are not the best representation for dealing with the topology.

Take N as the number of masters, and T as the number of connections between masters.

You'd have NT2 objects in the tree where, instead, I want only (ideally) N or T depending on whether we want a master centric view or a connection centric view (ie do we care about the nodes or the edges in the graph ?).

The idea was to represent masters and their connections with much simpler and easier to handle objects in the shared tree in LDAP, and use the plugin to monitor changes and write out cn=config entries as the consequence of changes to those objects.
The plugin would use graph based theory for the checks and eventually for preventing changes from happening.

In the future, ideally, self-repair would also be on the list:

  • for example if 2 admins sever links at the same time from 2 servers in a way that "after replication" would cause the graph to break in 2 disconnected graphs, then the plugin would prevent that from being translated into changes to the replication agreements, and instead automatically self-modify to re-establish the connections and replicate around the change.

In order to do this the topology objects and the a ctual cn=config objects must be separated, and the latter only be locally modified by the topology checking plugin, not by the replication plugin.

These are thoughts in progress, so confusion is no surprise. I did also not mean to just have the cn=config entries entries moved/copied to the shared tree – only the relevant info which should be possible to be managed from any server.
I understand your point about the potential large number entries depending on the number of master and connections. You can reduce the number of entries, but not the complexity. If you have an entry for each master, this entry must contain the information for all the connections affecting this master, so you will have one attribute or a group of attributes for each connection. If you want to have a group of attributes you somehow need to group them, if you want on eattribute and add some configuration attributes like skipped or stripped attributes I think the syntax will become horrible and the time when you want to extend which config params should be controlled will come. In fact there are sub entries in one entry and in my opinion it is much more clear and extendable if they are real entries.

Regarding the question if the replication plugin should handle parts of this or if everything should be handled by a custom plugin: i think replication plugin knows best about configuration and handles this for changes, but it could learn to take some parameters from a specific entry – we already do this in a way since the use of groups as replica binddns was implemented.

Replying to [comment:13 lkrispen]:

These are thoughts in progress, so confusion is no surprise. I did also not mean to just have the cn=config entries entries moved/copied to the shared tree – only the relevant info which should be possible to be managed from any server.
I understand your point about the potential large number entries depending on the number of master and connections. You can reduce the number of entries, but not the complexity. If you have an entry for each master, this entry must contain the information for all the connections affecting this master, so you will have one attribute or a group of attributes for each connection. If you want to have a group of attributes you somehow need to group them, if you want one attribute and add some configuration attributes like skipped or stripped attributes I think the syntax will become horrible and the time when you want to extend which config params should be controlled will come. In fact there are sub entries in one entry and in my opinion it is much more clear and extendable if they are real entries.

I think this can be accomplished by "member/memberof" style attribute pair. We can name it "connector" and "connectee" it can probably be maintained by a referential integrity plugin. The attribute "connector" will have the DNs of other masters the replica is connected to as a client. The "connectee" will be auto managed attribute and will hold the list DNs of masters the replica is serving as a server. What am I missing?

Replying to [comment:14 dpal]:

Replying to [comment:13 lkrispen]:

These are thoughts in progress, so confusion is no surprise. I did also not mean to just have the cn=config entries entries moved/copied to the shared tree – only the relevant info which should be possible to be managed from any server.
I understand your point about the potential large number entries depending on the number of master and connections. You can reduce the number of entries, but not the complexity. If you have an entry for each master, this entry must contain the information for all the connections affecting this master, so you will have one attribute or a group of attributes for each connection. If you want to have a group of attributes you somehow need to group them, if you want one attribute and add some configuration attributes like skipped or stripped attributes I think the syntax will become horrible and the time when you want to extend which config params should be controlled will come. In fact there are sub entries in one entry and in my opinion it is much more clear and extendable if they are real entries.

I think this can be accomplished by "member/memberof" style attribute pair. We can name it "connector" and "connectee" it can probably be maintained by a referential integrity plugin. The attribute "connector" will have the DNs of other masters the replica is connected to as a client. The "connectee" will be auto managed attribute and will hold the list DNs of masters the replica is serving as a server. What am I missing?

Why do we need to keep the fiction of asymmetrical connections for ipa masters ?

Are we ever planning on supporting one-way replicaiton agreements ? I don't think we do with full masters.

Replying to [comment:15 simo]:

Are we ever planning on supporting one-way replicaiton agreements ? I don't think we do with full masters.

I suspect that would be the case when we have RO replicas.

Replying to [comment:16 dpal]:

Replying to [comment:15 simo]:

Are we ever planning on supporting one-way replicaiton agreements ? I don't think we do with full masters.

I suspect that would be the case when we have RO replicas.

Ok, In any case, my idea was to represent agreements in the objects not the servers, so we would have 1 object per agreement. An agreement can then be mono-directional or bi-directional, but always have only 2 parties in it (I call them left and right).

Replying to [comment:17 simo]:

Replying to [comment:16 dpal]:

Replying to [comment:15 simo]:

Are we ever planning on supporting one-way replicaiton agreements ? I don't think we do with full masters.

I suspect that would be the case when we have RO replicas.

Ok, In any case, my idea was to represent agreements in the objects not the servers, so we would have 1 object per agreement. An agreement can then be mono-directional or bi-directional, but always have only 2 parties in it (I call them left and right).

That is an option too. The only difference is that we already have master objects so that we can just extend them to have relations instead of creating a separate object class to express the relations. Other then that it does not really matter.

patch for initial implementation attached, next will work on the management commands and then do full testing in IPA environment.
Comments are welcome

Thanks! But I think it would be better to submit the patch to freeipa-devel list. We do all our patch-related technical discussions there.

master:

  • 25bf0c6 ds plugin - manage replication topology in the shared tree

master:

  • 4bcc254 install part - manage topology in shared tree

master:

  • 41662eb server-find and server-show commands

master:

  • 4e05ffa plugin uses 1 as minimum domain level to become active no calculation based on plugin version
  • f87324d crash when removing a replica

master:

  • 8457edc accept missing binddn group

master:

  • b189e66 topology: ipa management commands

master:

  • 4232c39 topology: allow only one node to be specified in topologysegment-refresh
  • 2661a86 topology: hide topologysuffix-add del mod commands

master:

  • 7cf82cf move replications managers group to cn=sysaccounts,cn=etc,$SUFFIX
  • 99ce650 add entries required by topology plugin on update
  • c9cbb14 rename topologysegment_refresh to topologysegment_reinitialize
  • 5089dde disallow mod of topology segment nodes
  • b3c2a4b make sure the agremment rdn match the rdn used in the segment
  • 056518a v2-reject modifications of endpoints and connectivity of a segment
  • 6b153ba topology: restrict direction changes
  • bb6c0b9 topology: fix swapped topologysegment-reinitialize behavior
  • 45dcced ipa-replica-manage: Do not allow topology altering commands from DL 1
  • d58bdf2 server: add "del" command
  • e9e4509 ipa-replica-manage: adjust del to work with managed topology

master:

  • 659b88b topology: check topology in ipa-replica-manage del
  • 5397150 Verify replication topology for a suffix

The functionality is there. From now on, the feature is in bugfixing mode.

w00t! :)

Awesome... Perhaps a small blog post or similar on how this works would be really helpful?

Cheers!

As per freeipa-devel thread, we will postpone activation of the Topology plugin to next, shortly followed FreeIPA 4.3 release.

It will be disabled in 4.2 - see #5097. Sorry for the delay, but want to make this the best value for the users and not release it as fully baked.

Metadata Update from @simo:
- Issue assigned to lkrispen
- Issue set to the milestone: FreeIPA 4.3

2 years ago

Login to comment on this ticket.

Metadata