#48252 db2index creates index entry from deleted records
Closed: Fixed None Opened 3 years ago by tscherf.


1) Create the entry

# ldapadd -cvvvv -D "cn=Directory Manager" -W -f /root/ldif/create-posix-user2.ldif
ldap_initialize( <DEFAULT> )
add objectClass:
        top
        account
        posixAccount
        shadowAccount
add cn:
        Non Secure User
add uid:
        user80
add uidNumber:
        80
add gidNumber:
        80
add homeDirectory:
        /home/insecure
add loginShell:
        /bin/bash
add userpassword:
        {CLEAR}redhat
adding new entry "uid=user80,ou=People,dc=coe,dc=muc,dc=redhat,dc=com"
modify complete

2) Check the index file

# dbscan -f /var/lib/dirsrv/slapd-tscherf54/db/userRoot/gidNumber.db4 |grep "=80"
=80

3) Delete the entry

# ldapdelete -vx -D "cn=Directory Manager" -W "uid=user80,ou=People,dc=coe,dc=muc,dc=redhat,dc=com"
ldap_initialize( <DEFAULT> )
deleting entry "uid=user80,ou=People,dc=coe,dc=muc,dc=redhat,dc=com"

4) Verify the value from deleted entry has been removed from index

# dbscan -f /var/lib/dirsrv/slapd-tscherf54/db/userRoot/gidNumber.db4 |grep "=80"
#

5) Run db2index again

# /usr/lib64/dirsrv/slapd-tscherf54/db2index.pl  -n userRoot -D "cn=Directory Manager" -W -t gidNumber
adding new entry "cn=db2index_2015_8_19_10_34_41, cn=index, cn=tasks, cn=config"

6) Value from removed entry shows up in index file again

# dbscan -f /var/lib/dirsrv/slapd-tscherf54/db/userRoot/gidNumber.db4 |grep "=80"
=80

7) ldapsearch fails to find the removed entry and there is also nothing related in the error log:

ldapsearch -x gidNumber=80

extended LDIF

LDAPv3

base <dc=coe,dc=muc,dc=redhat,dc=com> (default) with scope subtree

filter: gidNumber=80

requesting: ALL

search result

search: 2
result: 0 Success

numResponses: 1


I tried to verify the issue on a single master system but I failed. The issue is only there when tombstone entries exists in the database. When I enable the USN-Plugin on a single master system, the issue pops up again.

Replying to [comment:1 tscherf]:

I tried to verify the issue on a single master system but I failed. The issue is only there when tombstone entries exists in the database. When I enable the USN-Plugin on a single master system, the issue pops up again.

Please note that USN plugin turns an entry to a tombstone entry to store the USN value in the deleted entry even if the server is a standalone server. Thus, the behaviour sounds consistent.

Replying to [comment:2 nhosoi]:

Replying to [comment:1 tscherf]:

I tried to verify the issue on a single master system but I failed. The issue is only there when tombstone entries exists in the database. When I enable the USN-Plugin on a single master system, the issue pops up again.

Please note that USN plugin turns an entry to a tombstone entry to store the USN value in the deleted entry even if the server is a standalone server. Thus, the behaviour sounds consistent.

Right, that was the reason why I enabled the plugin to confirm the issue has to do with tombstone records in the changelog.

I will now clone this bug into BZ since I also have a customer reporting this issue.

the fix now completely skips indexing of attributes except tombstonecsn and rdn-
But don't we need the (objectclass=nstombstone) index ?

Replying to [comment:11 lkrispen]:

the fix now completely skips indexing of attributes except tombstonecsn and rdn-
But don't we need the (objectclass=nstombstone) index ?

Thanks, Ludwig. Indeed. The special treatment is needed...
$ ldapsearch -LLLx ... -b "dc=example,dc=com" "(objectclass=nstombstone)" dn
dn: nsuniqueid=ed2d6596-478711e5-9d10cc15-0a6ddd94,uid=tuser,ou=People,dc=example,dc=com

$ db2index.pl -t objectclass ...

$ ldapsearch ... -b "dc=example,dc=com" "(objectclass=nstombstone)" dn <== returns none. :(
$

Reviewed by Mark (Thank you!!)

Attachment 0001-Ticket-48252-lib389-adding-get_bin_dir-and-dbscan.patch​ added
git patch file (master) -- lib389
Pushed to master:
2f78164..c48301a master -> master
commit c48301a63c1ea13ebd904e4a30a71fa53165d0da

Attachment 0001-Ticket-48252-db2index-creates-index-entry-from-delet.patch​ added
git patch file (master)
Attachment 0002-Ticket-48252-CI-test-added-test-cases-for-ticket-482.patch​ added
git patch file (master) -- CI test
Pushed to master:
2dbbb9d..b011ae3 master -> master
commit ca4d38d
commit b011ae3

Pushed to 389-ds-base-1.3.4:
626f2d7..2231ee4 389-ds-base-1.3.4 -> 389-ds-base-1.3.4
commit 86b9627
commit 2231ee4

Pushed to 389-ds-base-1.3.3:
7f63b58..ef0a295 389-ds-base-1.3.3 -> 389-ds-base-1.3.3
commit ef0a295

Note: the patch cannot be applied to 1.2.11 directly. Backport is needed.

Reviewed by Rich (Thank you!!)

Pushed to master:
ee96ec7..4277f2b master -> master
commit 4277f2b

Pushed to 389-ds-base-1.3.4:
326f73f..8c9c2fc 389-ds-base-1.3.4 -> 389-ds-base-1.3.4
commit 8c9c2fc

Pushed to 389-ds-base-1.3.3:
90cf9fd..f10e2e6 389-ds-base-1.3.3 -> 389-ds-base-1.3.3
commit f10e2e6

Pushed to 389-ds-base-1.2.11:
1e2793d..6a1f2f0 389-ds-base-1.2.11 -> 389-ds-base-1.2.11
commit 6a1f2f0

Metadata Update from @nhosoi:
- Issue assigned to nhosoi
- Issue set to the milestone: 1.2.11.33

2 years ago

Login to comment on this ticket.

Metadata