#48955 improve changelog buffer size management
Closed: wontfix 3 years ago by spichugi. Opened 7 years ago by lkrispen.

In a replication session a changelog buffer is used to read key(csn)/value(changes) pairs from the changelog database using a cursor and the DB_MULTIPLE_KEY flag.

The intention is to reduce database access, but the drawback in an active server is that while processing the buffer changes may be received which are inserted at the beginning of the buffer. So the effect is that at teh next bufferload the buffer start is reset and many chaneg which were already sent are processed again and skipped.

With the fix for #48766 the buffer size used was limited and only if data didn't fit it was temporarily increased.
Now, with the latest tests for reliab15, it showed that for large entries this can become inefficient as well, requiring many reallocs and buffer loads.

There is no optimal buffer size for all scenarios, so it would be best if the buffer size adapts to the effect of the last iteration.

I propose do do somthing like:

introduce two new fields, buf_record_cnt_last and buf_record_skipped_last
start with an arbitary not too big buffer

in clcache_cursor_get do:

buf_processd = buf_buf_record_cnt - buf_buf_record_cnt_last
buf_skipped = buf_record_skipped - buf_record_skipped_last
if buf_processed < MIN ==> increase buffer
if buf_skipped / buf_processed < XX % ==>increase (almost all entries were sent)
if buf_skipped / buf_processed > yy % ==> reduce buffer (too many skipped)


Metadata Update from @lkrispen:
- Issue set to the milestone: 1.3.6.0

7 years ago

Metadata Update from @mreynolds:
- Custom field component reset (from Replication - General)
- Issue close_status updated to: None
- Issue set to the milestone: 1.3.7.0 (was: 1.3.6.0)

6 years ago

Metadata Update from @mreynolds:
- Custom field component adjusted to None
- Custom field reviewstatus adjusted to None
- Issue set to the milestone: 1.4 backlog (was: 1.3.7.0)
- Issue tagged with: RFE

6 years ago

389-ds-base is moving from Pagure to Github. This means that new issues and pull requests
will be accepted only in 389-ds-base's github repository.

This issue has been cloned to Github and is available here:
- https://github.com/389ds/389-ds-base/issues/2014

If you want to receive further updates on the issue, please navigate to the github issue
and click on subscribe button.

Thank you for understanding. We apologize for all inconvenience.

Metadata Update from @spichugi:
- Issue close_status updated to: wontfix
- Issue status updated to: Closed (was: Open)

3 years ago

Login to comment on this ticket.

Metadata