It's worth noting, that even on basic tests, this was significantly faster on my machine:

before:
35 passed, 3 skipped, 150 warnings in 239.14s (0:03:59)
after:
35 passed, 3 skipped, 150 warnings in 173.03s (0:02:53)

That's roughly a 25% speedup.

Is 'threads' the final nsslapd-threadnumber ?
If yes, it looks very high.

The fix looks good but I have a doubt. Default was 30. Unless workers are very slow or doing very long job, I would expect 30 workers would be enough for most of load with rapid operations. By any chance do you have searchrate/modrate numbers that shows #workers > 50 or 100 are beneficial ?

@tbordaz it sets the number of threads based on how many CPU hardware threads are presented by the OS. so if you have a 4 core machine, it's 4. If it's 256 it's 256. If you have an 8 core with hyper threads, it would be 16.

The <512 there is a cap that we don't exceed. Not a minimum.

Another data point: search rate on a 4 CPU machine, 1 client with 1-10 threads:

@vashirov wow, those numbers are stunning. I think the scaling at the high end (10 client threads) is more important than the low numbbers since we do need to consider high concurrency as a key workload for us.

yes, it looks good at first sight. But you need to see what happens in a mixed load, I think write operations can easiiy block 4 threads and delay all binds and searches.

We should not optimize for one specific load pattern

I think that this may not be true going forward, as with lmdb on a concurrent cache design we can only have a single active writer, which means that we could distinguish between read operations and write operations in the thread pool, to guarantee that bind/read is always seperate.

Flip side of this, is we could just also have many many readers stalling writers causing them to stall too.

Saying this, I still agree that @vashirov can do some more of his excellent load testing to check this patch, :)

Here are the test results with 1-30 client threads and 4, 8, 16, 24 worker threads:
https://fedorapeople.org/groups/389ds/ci/pr51073/search.html
https://fedorapeople.org/groups/389ds/ci/pr51073/modify.html
https://fedorapeople.org/groups/389ds/ci/pr51073/mixed.html
https://fedorapeople.org/groups/389ds/ci/pr51073/auth.html

This is only with this patch applied, no other tunings.

Thanks for all these runs !

@vashirov for mixed load is it sync operation ? is it accounting MOD+SEARCH as one operation ? It is looking like the search rate is hidden by the mod rate.

For MODs at the moment we can not really conclude of a benefit of high/lower workers. For searchs there is still an unexpected significant negative impact of #workers.

@vashirov for mixed load is it sync operation ? is it accounting MOD+SEARCH as one operation ? It is looking like the search rate is hidden by the mod rate.

It was a SRCH followed by the MOD:

[12/May/2020:08:23:26.169046247 +0200] conn=1868 op=3715 SRCH base="dc=example,dc=com" scope=2 filter="(uid=user.8306)" attrs="givenName sn mail"
[12/May/2020:08:23:26.169258473 +0200] conn=1868 op=3715 RESULT err=0 tag=101 nentries=1 etime=0.000289852
[12/May/2020:08:23:26.169500575 +0200] conn=1868 op=3716 MOD dn="uid=user.8306,ou=People,dc=example,dc=com"
[12/May/2020:08:23:26.181633127 +0200] conn=1868 op=3716 RESULT err=0 tag=103 nentries=0 etime=0.012312424

I will add another test with async SRCH and MOD.

My analysis of what this shows is that search seems to improve with more threads, but something causes contention leading to the loss - so lower threads == less contention yielding the cpu-matched threads to give better search throughput and latency. It appears in the mixed workload our writes are heavily impacting the searches, so I think our write path is likely to be preventing search performance improvement. Regardless, it didn't make it worse, so I'm of course in favour of this change :)

IMHO LGTM, like William said it's not hurting the numbers. It's definitely an improvement, and if we need to fine tune at a later date so be it.

Actually there was something I'd like to see tested with this change. A machine with more CPU/cores.

So we tested a 4 core machine and setting the thread number to 4 was great, but what about a 16 core system with varying worker threads. Do we same improvement if we set the thread number to 16 vs 32 or 8 or 4?

@vashirov - would it be hard to reserve a system with this hardware and run one more rounds of tests?

When I did investigation for another potential customer a few years ago I also saw that setting the thread number to the number of cores gave the best performance. I think this is definitely an improvement over what we had, ack

389-ds-base is moving from Pagure to Github. This means that new issues and pull requests
will be accepted only in 389-ds-base's github repository.

This pull request has been cloned to Github as issue and is available here:
- https://github.com/389ds/389-ds-base/issues/4126

If you want to continue to work on the PR, please navigate to the github issue,
download the patch from the attachments and file a new pull request.

Thank you for understanding. We apologize for all inconvenience.