#48286 allow management of fractional replication at the replica level
Closed: wontfix 3 years ago by spichugi. Opened 8 years ago by lkrispen.

In ticket #48266 problems with fraction replication were handled when there were many changes which had to be skipped during replication, because they were filtered out by fractional replication configuration.
If the definition of "local" attributes is not specfic for a special replication agreement, but is common to all agreements, the attribute is only locally used, then these changes do not need to be logged in the changelog.
Allow configuring of attribute for a replica, not an agreement, to be local only - this reduces writing to the changelog and processing of the changelog during replication sessions


Note that if we skip recording updates in CL, we may also need to skip the update in the RUV.
Else we will have RUV that is ahead of CL and on restart we may decide to reset CL.

I think it is fine to skip updates from CL and RUV (may be the modification could be easier) but may be it can create corner case issue if the entries contains CSN that are ahead of what is in the RUV

This RFE targets performance improvements on DS side, it is not an urgent RFE for FreeIPA (see 5313) as updates for 'last successful Auth' will be avoided by default.
The fix ​https://fedorahosted.org/389/ticket/48266 is enough to prevent replication issue.

Metadata Update from @tbordaz:
- Issue set to the milestone: FUTURE

7 years ago

Assume we decide to keep local, for performance benefit, lastSuccessfulAuth at the replica level. So when the attribute is updated we do not log it in CL. But on a direct update, modifiersname and modifytimestamp operational attributes are also updated. If we decide to log modifiersname/modifytimestamp in the CL it reduces the benefit of this RFE (it will both write the CL and creates network traffic). If we decide to not log modifiersname/modifytimestamp in the CL, then those attributes will diverge on the topology.

Metadata Update from @tbordaz:
- Custom field reviewstatus adjusted to None
- Issue close_status updated to: None

4 years ago

That could cause some confusion I think, because modifiers name/ts, some people may use for auditing ....

My question here is what happens when we add another replication agreement and attributes that were previously local now become replicated? Would they start being added to the CL at that point?

@firstyear thanks for your feedback. I agree it could be confusing. Now it could be acceptable limitation for some application, like IPA, completely hiding ldap internals.

IPA is stripping modifersname and modifytimestamps, so in the context of IPA the change of moving modifersname and modifytimestamps in addtion to lastSuccessfulAuth to the replica level, will be beneficial for performance.

Regarding your question about conflict RA vs Replica I think there is no problem. If replica does not log in CL the updates (lastSuccessfulAuth, modifersname and modifytimestamps) then the RA will not find it in the CL. If RA stips those attributes or not, it will not find them in the updates to send. In that sense replica tuning has priority over the RA.

Regarding the sizing. I think the change is not complex (new replica config params and taking them into account in write_changelog_and_ruv), is quite easy to test and impacts documentation.

I see three concerns:
- The benefice is to avoid disk IO/space by skipping updating the CL and replication trafic . But the tuning could be complex if for example an update impacts both stripped and not stripped attribute it will be useless. Should we warn the admin about updates partially stripped ?
- Stripped attributes will be timestamped (csn) in the entries with CSN that can be ahead of the RUV. I believe it is not a concern for URP but corner cases are frequent in replication
- If the update is completely skipped from the CL then the RUV should not be updated (else CL may be recreated at next startup). The server will act like if the update never happened (like if it was failing operation, cleared RID, error condition....). Then it is like the operation never happened but it is logged in access/audit log.

Certainly having some really good test plans and handling of possible states that may exist would be the advice I'd have here :)

Ha, looks like I opened this ticket four years ago when I thought it was a good idea.

I am no longer so sure about this, the problem of iterating over many changes in the changelog has become less severe after introducing the keep alive entry.
And keeping the RUV and the changelog consistent was a difficult task, adding new cases, and handling modify attrs is a least challenging.

We should first try to determine the performance gain, this could be done by comparing replication performance with lastSuccessfulAuth enabled/disabled. When disabled nothing should be written to the changelog (it would also avoid to update the attr, but for a rough estimate should be good enough).

So @tbordaz what is the bigger picture here and the problem this change tries to solve? I'm curious about what this is.

Metadata Update from @mreynolds:
- Issue set to the milestone: 1.4.5 (was: FUTURE)

3 years ago

389-ds-base is moving from Pagure to Github. This means that new issues and pull requests
will be accepted only in 389-ds-base's github repository.

This issue has been cloned to Github and is available here:
- https://github.com/389ds/389-ds-base/issues/1617

If you want to receive further updates on the issue, please navigate to the github issue
and click on subscribe button.

Thank you for understanding. We apologize for all inconvenience.

Metadata Update from @spichugi:
- Issue close_status updated to: wontfix
- Issue status updated to: Closed (was: Open)

3 years ago

Login to comment on this ticket.

Metadata