User-Agent:       Mozilla/5.0 (Windows NT 5.1; rv:7.0.1) Gecko/20100101
Firefox/7.0.1

The entryusn.db4 index is not maintained correctly when a large number of
replicated operations arrive almost simultaneously. As a result, several
entryusn numbers in the index may correspond to one entryid and the searches on
entryusn return incorrect results.

Manual reindexing on entryusn resolves the problem.

Reproducible: Always

Steps to Reproduce:
I will describe our configuration where it is reproducible every time.

1. Configure three 389ds servers (in our case, it's 1.2.9.8 version with the
additional bugfix patch for the bug 735121).

2. Configure multimaster replication from each server to two others. While
configuring replicas, exclude entryusn and memberOf from replication.

Important: do not disable the replication of modifyTimestamp or modifiersName.
It's thanks to these attributes that a high load may be produced in this
configuration.

3. Create a sufficiently large groups with nested subgroups (~200 entries per
group is usually sufficient).

4. In a single (it is important!) operation make a replace or add/delete of all
or many uniqueMember attribute values of the group. It will generate the
following events :

* an avalanche of changes of the memberOf and entryusn attributes in a lot of
entries. These changes are not replicated, so it's ok.

* each changed entry has its modifyTimestamp and modifiersName attribute
changed at first directly and then by memberOf plugin.

* the changes are replicated to two other replicas. On these replicas memberOf
replaces the corresponding attributes and once again change modifyTimestamp and
modifiersName.

* As a result for each change we have two "returns to the sender" from two
replicas because of modifyTimestamp and modifiersName. Here is the typical ldap
audit log example :

time: 20111011210009
dn: cn=groupe dsi restreint,ou=par entite,ou=groupes
globaux,ou=groupes,dc=id,dc=polytechnique,dc=edu
changetype: modify
add: uniqueMember
uniqueMember: uid=andrey.ivanov,ou=Personnel,ou=Utilisateurs,dc=id,dc=polytech
 nique,dc=edu
-
replace: modifiersname
modifiersname: cn=X LDAP Root
-
replace: modifytimestamp
modifytimestamp: 20111011190009Z
-
replace: entryusn
entryusn: 69538
-

time: 20111011210011
dn:
uid=andrey.ivanov,ou=personnel,ou=utilisateurs,dc=id,dc=polytechnique,dc=edu
changetype: modify
replace: modifiersName
modifiersName: cn=MemberOf Plugin,cn=plugins,cn=config
-
replace: modifyTimestamp
modifyTimestamp: 20111011190011Z
-
replace: entryusn
entryusn: 69542
-

time: 20111011210015
dn:
uid=andrey.ivanov,ou=personnel,ou=utilisateurs,dc=id,dc=polytechnique,dc=edu
changetype: modify
replace: modifiersName
modifiersName: cn=MemberOf Plugin,cn=plugins,cn=config
-
replace: modifyTimestamp
modifyTimestamp: 20111011190016Z
-
replace: entryusn
entryusn: 69545
-



So, to resume, one change of the first replica generates two "returned"
replicated operations. With a sufficiently large initial group modification we
have a sufficiently large number of changes arriving at the same moment.
Actual Results:
After this operation the examining of entryusn index shows a large site of
duplicates. Here is an example :

dbscan -r -f entryusn.db4|less

=69294
        649
=69357
        649
=69387
        649
=69490
        649
=69511
        649

The entry with id 649 (in id2entry.db4) contains 69511 (on two other replicas
entryusn for this entry is 69891 and 69780, so the replication of entryusn is
out of game).

The number of duplicates can easily be found by comparing the number of all
entries and unique entries :

dbscan -r -f entryusn.db4| grep -v ^= | sort |wc
   5232   12074   71602

dbscan -r -f entryusn.db4| grep -v ^= | sort |uniq|wc
   5109   11951   70820

The searches of entryUSN=smth and entryUSN>=smth, naturally, do not return the
expected results since the index is corrupted.


Expected Results:
I am unable te reproduce the problem without replication or by changing just
one single independent attribute (like 'description').
I think this problem has to do something with heavy internal updates (memberOf
plugin) and a lot of replicated entries arriving simultaneously.

Reindexing recreates the index correctly:

db2index.pl -v -D "cn=Directory Manager" -w - -n userRoot -t entryusn
...

dbscan -r -f entryusn.db4| grep -v ^= | sort |wc
   5109   11951   70820

dbscan -r -f entryusn.db4| grep -v ^= | sort |uniq|wc
   5109   11951   70820

The failure is informed to entryusn by by setting LDAP_MOD_IGNORE
in the mod.  Entryusn plugin bepostop_modify uses the info to
determine to increment the entryusn or not.