git patch file (master)
0001-Trac-Ticket-412-memberof-performance-enhancement.patch

Fix description:
memberof.c: replaced DN strings with Slapi_DN and set the
normalized info to Slapi_Value flags. It reduces the number
of slaip_dn_normalize_ext call by ~25%.

attr.c, slapi-plugin.h: introduced a new API slapi_attr_
value_cmp_ext which takes Slapi_Value instead of struct
berval. By replacing with Slapi_Value, the value flag
(e.g., normalized info) can be passed to the syntax plugin.

value.c: changed slapi_value_compare to call slapi_attr_
value_cmp_ext instead of slapi_attr_value_cmp.

What sort of performance increase do these changes give you in your testing?

Replying to [comment:2 nkinder]:

What sort of performance increase do these changes give you in your testing?

Unfortunately, there was no observable performance gain. Let me explain it with the test env next...

testcase tarball
testcase.tar.gz

Tarball contains the following files:
-rw------- nhosoi/nhosoi 4315 2012-07-19 09:48 memof_mycom_core1.ldif
-rw------- nhosoi/nhosoi 50713 2012-07-16 11:54 memof_mycom_core2.1k.ldif
-rw-rw-r-- nhosoi/nhosoi 1480889 2012-07-16 11:58 mycom1k.ldif
-rw-rw-r-- nhosoi/nhosoi 403 2012-07-19 09:59 memof_script.sh

HowTo run the test script:
Set up 389-ds-base (suffix: o=my.com)
Enable MemberOf plugin; let memberofgroupattr the value uniquemember.
ldif2db -n <INSTANCE> -i /path/to/memof_mycom_core1.ldif
start the server

cd /path/to/
ldapmodify ... -af memof_mycom_core2.1k.ldif
sh -x memof_script.sh <PORT> <DIRECTORY_MANAGER_PASSWORD>

The script adds a group entry which contains 1000 uniqueMembers, then wait for all the member entries have the "memberof" attribute value.

Result:
Note: With/Without the attached patch: 0001-Trac-Ticket-412-memberof-performance-enhancement.patch​, there was no difference in the elapsed time.

Straight ldapmodify ...; sh memof_script.sh: 14 seconds
ldapmodify ...; restart the server; sh memof_script.sh: 27 seconds
ldapmodify ...; restart the server; ldapsearch <all>; sh memof_script.sh: 27 seconds
(I.e, the difference is not due to the entries in the entry cache or not)

Enable betxn of the Memberof Plugin
dn: cn=MemberOf Plugin,cn=plugins,cn=config
nsslapd-pluginType: betxnpostoperation
Straight ldapmodify ...; sh memof_script.sh: < 1 second
ldapmodify ...; restart the server; ldapsearch <all>; sh memof_script.sh: < 1 second

Patch looks good.

I guess we had better start testing IPA again with betxn plugins enabled . . .

Replying to [comment:7 rmeggins]:

Patch looks good.

I guess we had better start testing IPA again with betxn plugins enabled . . .

Thank you for the review, Rich! I ran the acceptance level multi plugin test cases with the related plugins enabled (with this patch). It passed 100%. Next, I'm running the stress level...

I'm running the stress level...
The Stress level test passed 100%, too!

Reviewed by Rich (Thank you!!!)

Pushed to master.

$ git merge trac412
Updating 0b5cdb4..6a21e8e
Fast-forward
ldap/servers/plugins/memberof/memberof.c | 342 ++++++++++++++++--------------
ldap/servers/slapd/attr.c | 69 +++++--
ldap/servers/slapd/slapi-plugin.h | 23 ++-
ldap/servers/slapd/value.c | 8 +-
4 files changed, 263 insertions(+), 179 deletions(-)

$ git push
Counting objects: 73, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (37/37), done.
Writing objects: 100% (37/37), 6.74 KiB, done.
Total 37 (delta 30), reused 0 (delta 0)
To ssh://git.fedorahosted.org/git/389/ds.git
0b5cdb4..6a21e8e master -> master

Ticket has been cloned to Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=842440

Added initial screened field value.

389-ds-base is moving from Pagure to Github. This means that new issues and pull requests
will be accepted only in 389-ds-base's github repository.

This issue has been cloned to Github and is available here:
- https://github.com/389ds/389-ds-base/issues/412

If you want to receive further updates on the issue, please navigate to the github issue
and click on subscribe button.

Thank you for understanding. We apologize for all inconvenience.