#47313 Indexed search with filter containing '&' and "!" with attribute subtypes gives wrong result
Closed: wontfix None Opened 8 years ago by pj101.

Server version: 389DS v1.2.10.26 on CentOS 5.x x86_64

I have not tested it on 1.2.11 and 1.3.0.

How to reproduce:

Add a new multi-value attribute to the schema, say, X-Flags. Index this attribute on presence and equality (in my tests no substring index was present).

Maybe a test with one of the pre-defined attributes also works (with the same index types)

Create an entry containing this attribute with a subtype, e.g.:

uid=login,dc=example,dc=com
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: inetOrgPerson
objectClass: X-Misc
uid: login
...
X-Flags;en: test

Now make a search using the following filter:
(&(objectClass=inetOrgPerson)(!(X-Flags;fr=test)))

The previously created entry should be returned by this search. However it is not returned. If we eliminate the indexing on X-Flags by changing this search to a substring search (adding "" to test)
(&(objectClass=inetOrgPerson)(!(X-Flags;fr=test
)))
then the entry is returned. So the problem is obviously in the usage of index. If i disable the indexes on this attribute, all the searches return correct results.

However, simple searches without '&' work just fine, both with and without indexes:
(!(X-Flags;fr=test)) returns the entry correctly.


I could not reproduce the problem... Here's what I did...
1. Added X-Flags and X-Misc to /etc/dirsrv/slapd-<INST>/schema/99user.ldif. Then, restarted the server.
{{{
dn: cn=schema

attributeTypes: ( 2.16.840.1.113730.3.1.123456.1
NAME 'X-Flags'
DESC 'Test attribute type'
SYNTAX 1.3.6.1.4.1.1466.115.121.1.15 X-ORIGIN 'test' )

objectClasses: ( 2.16.840.1.113730.3.2.123456.2
NAME 'X-Misc'
SUP inetOrgPerson
STRUCTURAL
MAY ( X-Flags ) X-ORIGIN 'test' )
}}}
2. Added index for X-Flags:
{{{
$ ldapmodify ... << EOF
dn: cn=X-Flags,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
objectClass: top
objectClass: nsIndex
cn: X-Flags
nsSystemIndex: false
nsIndexType: eq
nsIndexType: pres
EOF
}}}
3. Added an entry:
{{{
$ ldapmodify ... << EOF
dn: uid=tuser0,ou=People,dc=example,dc=com
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: inetOrgPerson
objectClass: X-Misc
cn: test user0
sn: user0
uid: tuser0
givenName: test
roomNumber: 1000
mail: tuser0@example.com
userPassword: {CLEAR}tuser0
X-Flags;en: test
EOF
}}}
4. Make sure the entry is indexed in X-Flags.db:
{{{

dbscan -f /var/lib/dirsrv/slapd-<INST>/db/userRoot/X-Flags.db -r

  • 10
    =test
    10
    }}}
    5. Search with the filter: (&(objectClass=inetOrgPerson)(!(X-Flags;fr=test))). I get the expected results.
    {{{
    $ ldapsearch ... -b "dc=example,dc=com" '(&(objectClass=inetOrgPerson)(!(X-Flags;fr=test)))' X-Flags
    dn: uid=tuser0,ou=People,dc=example,dc=com
    X-Flags;en: test

$ ldapsearch ... -b "dc=example,dc=com" '(&(objectClass=inetOrgPerson)(X-Flags;en=test))' X-Flags
dn: uid=tuser0,ou=People,o=my.com
X-Flags;en: test
}}}
Please let me know if I'm missing something...

I'd like you to try one thing, the step 4 -- Make sure the entry is indexed in X-Flags.db. If you don't see the X-Flags value(s) in the index file, please re-index the attribute as "db2index -n <your_backend> -t X-Flags" and retry the search.

Hi Noriko,

ok, i'll try to do the same steps tomorrow on our test server. As i said, i have tested only 1.2.10.x branch on CentOS 5.x. If you made your tests on a more recent version of 38DS and on RHEL6/Fedora, maybe the indexing code (or berkley db is more recents) have changed a bit thus correcting the bug...

I've tested on our dev server and following your steps it does work correctly. It does also work in simple cases if i use dbgen.pl.

However, it does not work for our production data (just in case, the entry is correctly indexed in x-flags.db, the data is imported from *.ldif, the server is completely reinstalled 100% automatically with scripts), and this problem is reproducible each time. I will try to narrow down the problem deleting progressively and simplifying the production data.

LDIF file to reproduce the bug
bug-search.ldif

Well, i've finally managed to craft an ldif export that reproduces the bug. It's rather special - if you delete any of the entries from this ldif, the bug disappears :) So it took me some time to find a minimum set of data (reducing from 25000 objects in our production server to ~20 entries) and then to anonymize the entries.

So, in order to reproduce the problem you need to choose the suffix ''"dc=id,dc=polytechnique,dc=edu"'' (i do not know whether the precise suffix is important or not) when installing, then add the schema and indexing as you have done in your steps 1,2.

Stop the server, then import the whole ldif file i will attach:
{{{ldif2db -n userRoot -i /tmp/bug.ldif}}}

Start the server, check the index file (your step 4):
{{{
[root@ldap-model ~]# /Local/dirsrv/bin/dbscan -r -f /Local/dirsrv/var/lib/dirsrv/slapd-ldap-model/db/userRoot/X-Flags.db4
+
15
=test
15
}}}

Now, search and check whether the entry '''"uid=consult"''' appears in the result:

{{{
ldapsearch ... -b "ou=Comptes Mail,ou=Comptes generiques,ou=Utilisateurs,dc=id,dc=polytechnique,dc=edu" '(&(objectClass=inetOrgPerson)(!(X-Flags;fr=test)))' uid X-Flags

extended LDIF

LDAPv3

base <ou=Comptes Mail,ou=Comptes generiques,ou=Utilisateurs,dc=id,dc=polytechnique,dc=edu> with scope subtree

filter: (&(objectClass=inetOrgPerson)(!(X-Flags;fr=test)))

requesting: uid X-Flags

phi, Comptes Mail, Comptes generiques, Utilisateurs, id.polytechnique.edu

dn: uid=phi,ou=Comptes Mail,ou=Comptes generiques,ou=Utilisateurs,dc=id,dc=pol
ytechnique,dc=edu
uid: phi

trex.meca, Comptes Mail, Comptes generiques, Utilisateurs, id.polytechnique

.edu
dn: uid=trex.meca,ou=Comptes Mail,ou=Comptes generiques,ou=Utilisateurs,dc=id,
dc=polytechnique,dc=edu
uid: trex.meca

tuchka, Comptes Mail, Comptes generiques, Utilisateurs, id.polytechnique.ed

u
dn: uid=tuchka,ou=Comptes Mail,ou=Comptes generiques,ou=Utilisateurs,dc=id,dc=
polytechnique,dc=edu
uid: tuchka

search result

search: 2
result: 0 Success

numResponses: 4

numEntries: 3

ldapsearch ... -b "ou=Comptes Mail,ou=Comptes generiques,ou=Utilisateurs,dc=id,dc=polytechnique,dc=edu" '(&(objectClass=inetOrgPerson)(!(X-Flags;fr=test*)))' uid X-Flags

extended LDIF

LDAPv3

base <ou=Comptes Mail,ou=Comptes generiques,ou=Utilisateurs,dc=id,dc=polytechnique,dc=edu> with scope subtree

filter: (&(objectClass=inetOrgPerson)(!(X-Flags;fr=test*)))

requesting: uid X-Flags

phi, Comptes Mail, Comptes generiques, Utilisateurs, id.polytechnique.edu

dn: uid=phi,ou=Comptes Mail,ou=Comptes generiques,ou=Utilisateurs,dc=id,dc=pol
ytechnique,dc=edu
uid: phi

consult, Comptes Mail, Comptes generiques, Utilisateurs, id.polytechnique.e

du
dn: uid=consult,ou=Comptes Mail,ou=Comptes generiques,ou=Utilisateurs,dc=id,dc
=polytechnique,dc=edu
uid: consult
X-Flags;en: test

trex.meca, Comptes Mail, Comptes generiques, Utilisateurs, id.polytechnique

.edu
dn: uid=trex.meca,ou=Comptes Mail,ou=Comptes generiques,ou=Utilisateurs,dc=id,
dc=polytechnique,dc=edu
uid: trex.meca

tuchka, Comptes Mail, Comptes generiques, Utilisateurs, id.polytechnique.ed

u
dn: uid=tuchka,ou=Comptes Mail,ou=Comptes generiques,ou=Utilisateurs,dc=id,dc=
polytechnique,dc=edu
uid: tuchka

search result

search: 2
result: 0 Success

numResponses: 5

numEntries: 4

}}}

Thanks for the reproducer! I could duplicate the bug. Investigating it now...

Bug description: Index db files do not contain the subtype knowledge,
which is only in the primary id2entry db and entries in the memory.
If the search filter includes subtype in the NOT condition and
the type is indexed, the condition is mistakenly simplified to
the one equivalent to not having the subtype.

E.g., if the given filter is (&(cn=A)(!(cn;fr=ABC en)), it's
evaluated as (&(cn=A
)(!(cn=ABC en)).

Fix description: If a filter contains a subtype in NOT condition,
we give up using the index and leave the not evaluation to the
search return code.

Reviewed by Rich (Thank you!!)

Pushed to master: commit fae0068

https://bugzilla.redhat.com/show_bug.cgi?id=1041732
{{{
--- Comment #15 from Rich Megginson rmeggins@redhat.com ---
Created attachment 836423
--> https://bugzilla.redhat.com/attachment.cgi?id=836423&action=edit
stacktrace 2

The problem is at list_candidates():855:
} else if ( ftype == LDAP_FILTER_AND ) {
if (isnot && !idl_is_allids(tmp)) {
the search returned tmp == NULL (*err = -30988 DB_NOTFOUND) because
objectclass=mepManagedEntry was not found. This problem was introduced with
this commit:

commit fae0068
Author: Noriko Hosoi nhosoi@redhat.com
Date: Wed Apr 17 14:55:56 2013 -0700

Ticket #47313 - Indexed search with filter containing '&' and "!" with

attribute subtypes gives wrong result
}}}

Description: commit fae0068 introduced
a bug, which occurs when a filter includes NOT and one of the results
from the subfilters returns NONE. This patch backoffs the last section
of the commit fae0068 with an improvement
-- avoiding unnecessary idl duplication.

Also, adding (NULL == idl) checks to idl_common.c.

Reviewed by rmeggins (Thank you, Rich!)

Pushed to master:
8a4bbc7..a71633d master -> master
commit a71633d

Pushed to 389-ds-base-1.3.2:
3f28f45..c8b44e1 389-ds-base-1.3.2 -> 389-ds-base-1.3.2
commit c8b44e1

git patch file (master) -- CI test: add test case for ticket 47313
0001-Ticket-47313-CI-test-add-test-case-for-ticket-47313.patch

Pushed the CI test to master:
d50f994..3939aaf master -> master
commit 3939aaf

Hi Noriko,

if i understand correctly, this patch did not make it to 1.3.1? I've just installed RHEL7 with system-supplied 389ds (389-ds-base-1.3.1.6-25.el7.x86_64) and the bug is still present. Do i need to use the 1.3.2 branch (not too stable?) in order to eliminate the bug?

Andrey IVANOV

Replying to [comment:20 pj101]:

Hi Andrey,

if i understand correctly, this patch did not make it to 1.3.1?
Correct. This patch was checked in only to 389-ds-base-1.3.2 branch and newer.

I've just installed RHEL7 with system-supplied 389ds (389-ds-base-1.3.1.6-25.el7.x86_64) and the bug is still present. Do i need to use the 1.3.2 branch (not too stable?) in order to eliminate the bug?
Correct, again. It is getting stabler. But the test is still in progress...

Thanks,
--noriko

Metadata Update from @nhosoi:
- Issue assigned to nhosoi
- Issue set to the milestone: 1.3.2 - 04/13 (April)

4 years ago

389-ds-base is moving from Pagure to Github. This means that new issues and pull requests
will be accepted only in 389-ds-base's github repository.

This issue has been cloned to Github and is available here:
- https://github.com/389ds/389-ds-base/issues/650

If you want to receive further updates on the issue, please navigate to the github issue
and click on subscribe button.

Thank you for understanding. We apologize for all inconvenience.

Metadata Update from @spichugi:
- Issue close_status updated to: wontfix (was: Fixed)

8 months ago

Login to comment on this ticket.

Metadata