#411 [RFE] mods optimizer
Closed: wontfix None Opened 8 years ago by nhosoi.

Comparing the following 2 mods in one modify, the first one is much faster.
Tested wit 10K values, the first case took 23 sec. vs. the second 80 sec.

mods[0].mod_op: LDAP_MOD_ADD
mods[0].mod_type: uniqueMember
mods[0].mod_values: <value_0>
                    <value_1>
                       ...
                    <value_N>

mods[0].mod_op: LDAP_MOD_ADD
mods[0].mod_type: uniqueMember
mods[0].mod_values: <value_0>
mods[1].mod_op: LDAP_MOD_ADD
mods[1].mod_type: uniqueMember
mods[1]mod_values: <value_1>
         ...
mods[N].mod_op: LDAP_MOD_ADD
mods[N].mod_type: uniqueMember
mods[N]mod_values: <value_N>

We may be able to introduce mods optimizer for some cases like the above.

Note: there are some operations which should not be optimized/cancelled.
For instance, mods could be used for check-and-increment as follows. In the example, if foo=1 does not exist, the modify should fail and the next add should not happen.

mods[0].mod_op: LDAP_MOD_DELETE
mods[0].mod_type: foo
mods[0].mod_values: 1
mods[1].mod_op: LDAP_MOD_ADD
mods[1].mod_type: foo
mods[1]mod_values: 2

How to run the test program modattrs:

{{{
1. Compile modattrs (You may need to modify the server port MY_PORT and the directory manager password MGR_PW)
$ gcc -g -I /usr/include/mozldap -o modattrs modattrs.c -lldap60 -lpthread
2. Set up 389-ds-base with the Suffix "o=my.com"
3. Add the following group
dn: cn=Accounting Managers,ou=Groups,o=my.com
nsUniqueId: fddf0105-cab411e1-bc6184ba-ac438f09
objectClass: top
objectClass: groupOfUniqueNames
objectClass: eduPerson
cn: Accounting Managers
ou: groups
description: People who can manage accounting entries
4. Run modattrs
4-1. one ldap modify request modifies (adds) one attribute value.
./modattrs [-n <number_of_attribute_values_to_add>]
4-2. one ldap modify request modifies (adds) all attribute values stored in one big mod. (the first case of the Description)
./modattrs [-n <number_of_attribute_values_to_add>] one
4-3. one ldap modify request modifies (adds) all attribute values, where each value is store in each mod. (the second case of the Description)
./modattrs [-n <number_of_attribute_values_to_add>] many
}}}

Test results:
{{{
attribute type: uniqueMember (DN string)
number_of_attribute_values_to_add: 1000
no option (4-1): 71 sec
one (4-2): <1 sec
many (4-3): 1 sec
}}}

{{{
attribute type: uniqueMember (DN string)
number_of_attribute_values_to_add: 10000
one (4-2): 23 sec
many (4-3): 80 sec
}}}

{{{
attribute type: eduPersonEntitlement (Directory string)
number_of_attribute_values_to_add: 1000
no option (4-1): 78 sec
one (4-2): 1 sec
many (4-3): 2 sec
}}}

{{{
attribute type: eduPersonEntitlement (Directory string)
number_of_attribute_values_to_add: 10000
one (4-2): 83 sec
many (4-3): 211 sec
}}}

Additional info:
{{{
attribute type: uniqueMember (DN string)
number_of_attribute_values_to_add: 1000
no option (4-1): 71 sec
no option (4-1) with plugins off(): 66 sec
}}}
{{{
attribute type: uniqueMember (DN string)
number_of_attribute_values_to_add: 10000
one (4-2): 23 sec
one (4-2) with plugins off(
): 20 sec
many (4-3): 80 sec
many (4-3) with plugins off(*): 75 sec
}}}

(*) disabled plugins: Auto Membership Plugin, Managed Entries, Linked Attributes
Note: MemberOf, referential integrity postoperation, Distributed Numeric Assignment Plugin, attribute uniqueness are off, by default.

So what does modattrs.c do that we can't do with ldapmodify and an LDIF file?

Replying to [comment:4 rmeggins]:

So what does modattrs.c do that we can't do with ldapmodify and an LDIF file?

A good question. :) All can be done by scripts, I believe. It was just handy for me. :p

set default ticket origin to Community

Added initial screened field value.

Test results...

Without patch

{{{
./modattrs -n 10000 many
Start 10000 modify (Many): Fri Sep 6 11:11:29 2013
End 10000 modify (Many): Fri Sep 6 11:11:54 2013
(25 sec)
}}}

Results with patch:

{{{

./modattrs -n 10000 many
Start 10000 modify (Many): Fri Sep 6 11:08:23 2013
End 10000 modify (Many): Fri Sep 6 11:08:23 2013
(0 sec)
}}}

Since the results are good, and no valgrind complaints, I'll be preparing the patch for review...

One reason why a user may want to add values one at a time is for error checking. If I attempt to add a single value, I can know exactly if there was an error with that particular value - already exists, invalid syntax, etc. Same with deletion - value doesn't exist. Is this substantially different with the new code? For example, what does ldapmodify -c -S errfile report in the errfile? If the behavior is different, we need to document the new behavior.

Replying to [comment:19 rmeggins]:

One reason why a user may want to add values one at a time is for error checking. If I attempt to add a single value, I can know exactly if there was an error with that particular value - already exists, invalid syntax, etc. Same with deletion - value doesn't exist. Is this substantially different with the new code? For example, what does ldapmodify -c -S errfile report in the errfile? If the behavior is different, we need to document the new behavior.

This fix takes this "single" mod operation:

dn: uid=m,dc=example,dc=com
changetype: modify
delete: description
description: 1
-
delete: description
description: 2
-
delete: description
description: 3

and converts it to:

dn: uid=m,dc=example,dc=com
changetype: modify
delete: description
description: 1
description: 2
description: 3

In either case, if one value does not exist, the entire mod operation will fail. I found no difference between these two "mods" when one of the values does not exist. (Error 16)

Replying to [comment:20 mreynolds]:

Replying to [comment:19 rmeggins]:
<snip>
In either case, if one value does not exist, the entire mod operation will fail. I found no difference between these two "mods" when one of the values does not exist. (Error 16)

Great! Thanks.

git merge ticket411
Updating 3adc242..8358b43
Fast-forward
ldap/servers/slapd/modify.c | 76 +++++++++++++++++++++++++++++++++++++++++++
1 files changed, 76 insertions(+), 0 deletions(-)

git push origin master
Counting objects: 11, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (6/6), 1.83 KiB, done.
Total 6 (delta 4), reused 0 (delta 0)
To ssh://git.fedorahosted.org/git/389/ds.git
3adc242..8358b43 master -> master

commit 8358b43
Author: Mark Reynolds mreynolds@redhat.com
Date: Fri Sep 6 11:55:21 2013 -0400

Metadata Update from @nhosoi:
- Issue assigned to mreynolds
- Issue set to the milestone: 1.3.2 - 09/13 (September)

3 years ago

389-ds-base is moving from Pagure to Github. This means that new issues and pull requests
will be accepted only in 389-ds-base's github repository.

This issue has been cloned to Github and is available here:
- https://github.com/389ds/389-ds-base/issues/411

If you want to receive further updates on the issue, please navigate to the github issue
and click on subscribe button.

Thank you for understanding. We apologize for all inconvenience.

Metadata Update from @spichugi:
- Issue close_status updated to: wontfix (was: Fixed)

16 days ago

Login to comment on this ticket.

Metadata