Server version: 389DS v1.2.10.26 on CentOS 5.x x86_64
I have not tested it on 1.2.11 and 1.3.0.
How to reproduce:
Add a new multi-value attribute to the schema, say, X-Flags. Index this attribute on presence and equality (in my tests no substring index was present).
Maybe a test with one of the pre-defined attributes also works (with the same index types)
Create an entry containing this attribute with a subtype, e.g.:
uid=login,dc=example,dc=com objectClass: top objectClass: person objectClass: organizationalPerson objectClass: inetOrgPerson objectClass: X-Misc uid: login ... X-Flags;en: test
Now make a search using the following filter: (&(objectClass=inetOrgPerson)(!(X-Flags;fr=test)))
The previously created entry should be returned by this search. However it is not returned. If we eliminate the indexing on X-Flags by changing this search to a substring search (adding "" to test) (&(objectClass=inetOrgPerson)(!(X-Flags;fr=test))) then the entry is returned. So the problem is obviously in the usage of index. If i disable the indexes on this attribute, all the searches return correct results.
However, simple searches without '&' work just fine, both with and without indexes: (!(X-Flags;fr=test)) returns the entry correctly.
I could not reproduce the problem... Here's what I did... 1. Added X-Flags and X-Misc to /etc/dirsrv/slapd-<INST>/schema/99user.ldif. Then, restarted the server. {{{ dn: cn=schema
attributeTypes: ( 2.16.840.1.113730.3.1.123456.1 NAME 'X-Flags' DESC 'Test attribute type' SYNTAX 1.3.6.1.4.1.1466.115.121.1.15 X-ORIGIN 'test' )
objectClasses: ( 2.16.840.1.113730.3.2.123456.2 NAME 'X-Misc' SUP inetOrgPerson STRUCTURAL MAY ( X-Flags ) X-ORIGIN 'test' ) }}} 2. Added index for X-Flags: {{{ $ ldapmodify ... << EOF dn: cn=X-Flags,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config objectClass: top objectClass: nsIndex cn: X-Flags nsSystemIndex: false nsIndexType: eq nsIndexType: pres EOF }}} 3. Added an entry: {{{ $ ldapmodify ... << EOF dn: uid=tuser0,ou=People,dc=example,dc=com objectClass: top objectClass: person objectClass: organizationalPerson objectClass: inetOrgPerson objectClass: X-Misc cn: test user0 sn: user0 uid: tuser0 givenName: test roomNumber: 1000 mail: tuser0@example.com userPassword: {CLEAR}tuser0 X-Flags;en: test EOF }}} 4. Make sure the entry is indexed in X-Flags.db: {{{
10 =test 10 }}} 5. Search with the filter: (&(objectClass=inetOrgPerson)(!(X-Flags;fr=test))). I get the expected results. {{{ $ ldapsearch ... -b "dc=example,dc=com" '(&(objectClass=inetOrgPerson)(!(X-Flags;fr=test)))' X-Flags dn: uid=tuser0,ou=People,dc=example,dc=com X-Flags;en: test
$ ldapsearch ... -b "dc=example,dc=com" '(&(objectClass=inetOrgPerson)(X-Flags;en=test))' X-Flags dn: uid=tuser0,ou=People,o=my.com X-Flags;en: test }}} Please let me know if I'm missing something...
I'd like you to try one thing, the step 4 -- Make sure the entry is indexed in X-Flags.db. If you don't see the X-Flags value(s) in the index file, please re-index the attribute as "db2index -n <your_backend> -t X-Flags" and retry the search.
Hi Noriko,
ok, i'll try to do the same steps tomorrow on our test server. As i said, i have tested only 1.2.10.x branch on CentOS 5.x. If you made your tests on a more recent version of 38DS and on RHEL6/Fedora, maybe the indexing code (or berkley db is more recents) have changed a bit thus correcting the bug...
I've tested on our dev server and following your steps it does work correctly. It does also work in simple cases if i use dbgen.pl.
However, it does not work for our production data (just in case, the entry is correctly indexed in x-flags.db, the data is imported from *.ldif, the server is completely reinstalled 100% automatically with scripts), and this problem is reproducible each time. I will try to narrow down the problem deleting progressively and simplifying the production data.
LDIF file to reproduce the bug bug-search.ldif
Well, i've finally managed to craft an ldif export that reproduces the bug. It's rather special - if you delete any of the entries from this ldif, the bug disappears :) So it took me some time to find a minimum set of data (reducing from 25000 objects in our production server to ~20 entries) and then to anonymize the entries.
So, in order to reproduce the problem you need to choose the suffix ''"dc=id,dc=polytechnique,dc=edu"'' (i do not know whether the precise suffix is important or not) when installing, then add the schema and indexing as you have done in your steps 1,2.
Stop the server, then import the whole ldif file i will attach: {{{ldif2db -n userRoot -i /tmp/bug.ldif}}}
Start the server, check the index file (your step 4): {{{ [root@ldap-model ~]# /Local/dirsrv/bin/dbscan -r -f /Local/dirsrv/var/lib/dirsrv/slapd-ldap-model/db/userRoot/X-Flags.db4 + 15 =test 15 }}}
Now, search and check whether the entry '''"uid=consult"''' appears in the result:
{{{ ldapsearch ... -b "ou=Comptes Mail,ou=Comptes generiques,ou=Utilisateurs,dc=id,dc=polytechnique,dc=edu" '(&(objectClass=inetOrgPerson)(!(X-Flags;fr=test)))' uid X-Flags
dn: uid=phi,ou=Comptes Mail,ou=Comptes generiques,ou=Utilisateurs,dc=id,dc=pol ytechnique,dc=edu uid: phi
.edu dn: uid=trex.meca,ou=Comptes Mail,ou=Comptes generiques,ou=Utilisateurs,dc=id, dc=polytechnique,dc=edu uid: trex.meca
u dn: uid=tuchka,ou=Comptes Mail,ou=Comptes generiques,ou=Utilisateurs,dc=id,dc= polytechnique,dc=edu uid: tuchka
search: 2 result: 0 Success
ldapsearch ... -b "ou=Comptes Mail,ou=Comptes generiques,ou=Utilisateurs,dc=id,dc=polytechnique,dc=edu" '(&(objectClass=inetOrgPerson)(!(X-Flags;fr=test*)))' uid X-Flags
du dn: uid=consult,ou=Comptes Mail,ou=Comptes generiques,ou=Utilisateurs,dc=id,dc =polytechnique,dc=edu uid: consult X-Flags;en: test
}}}
Thanks for the reproducer! I could duplicate the bug. Investigating it now...
Bug description: Index db files do not contain the subtype knowledge, which is only in the primary id2entry db and entries in the memory. If the search filter includes subtype in the NOT condition and the type is indexed, the condition is mistakenly simplified to the one equivalent to not having the subtype.
E.g., if the given filter is (&(cn=A)(!(cn;fr=ABC en)), it's evaluated as (&(cn=A)(!(cn=ABC en)).
Fix description: If a filter contains a subtype in NOT condition, we give up using the index and leave the not evaluation to the search return code.
git patch file (master) 0001-Ticket-47313-Indexed-search-with-filter-containing-a.patch
Reviewed by Rich (Thank you!!)
Pushed to master: commit fae0068
https://bugzilla.redhat.com/show_bug.cgi?id=1041732 {{{ --- Comment #15 from Rich Megginson rmeggins@redhat.com --- Created attachment 836423 --> https://bugzilla.redhat.com/attachment.cgi?id=836423&action=edit stacktrace 2
The problem is at list_candidates():855: } else if ( ftype == LDAP_FILTER_AND ) { if (isnot && !idl_is_allids(tmp)) { the search returned tmp == NULL (*err = -30988 DB_NOTFOUND) because objectclass=mepManagedEntry was not found. This problem was introduced with this commit:
commit fae0068 Author: Noriko Hosoi nhosoi@redhat.com Date: Wed Apr 17 14:55:56 2013 -0700
Ticket #47313 - Indexed search with filter containing '&' and "!" with
attribute subtypes gives wrong result }}}
Description: commit fae0068 introduced a bug, which occurs when a filter includes NOT and one of the results from the subfilters returns NONE. This patch backoffs the last section of the commit fae0068 with an improvement -- avoiding unnecessary idl duplication.
Also, adding (NULL == idl) checks to idl_common.c.
git patch file (master) -- Bug fix for bz 1041732 0001-Ticket-47313-Indexed-search-with-filter-containing-a.2.patch
Reviewed by rmeggins (Thank you, Rich!)
Pushed to master: 8a4bbc7..a71633d master -> master commit a71633d
Pushed to 389-ds-base-1.3.2: 3f28f45..c8b44e1 389-ds-base-1.3.2 -> 389-ds-base-1.3.2 commit c8b44e1
Ticket has been cloned to Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1044133
git patch file (master) -- CI test: add test case for ticket 47313 0001-Ticket-47313-CI-test-add-test-case-for-ticket-47313.patch
Pushed the CI test to master: d50f994..3939aaf master -> master commit 3939aaf
if i understand correctly, this patch did not make it to 1.3.1? I've just installed RHEL7 with system-supplied 389ds (389-ds-base-1.3.1.6-25.el7.x86_64) and the bug is still present. Do i need to use the 1.3.2 branch (not too stable?) in order to eliminate the bug?
Andrey IVANOV
Replying to [comment:20 pj101]:
Hi Andrey,
if i understand correctly, this patch did not make it to 1.3.1? Correct. This patch was checked in only to 389-ds-base-1.3.2 branch and newer. I've just installed RHEL7 with system-supplied 389ds (389-ds-base-1.3.1.6-25.el7.x86_64) and the bug is still present. Do i need to use the 1.3.2 branch (not too stable?) in order to eliminate the bug? Correct, again. It is getting stabler. But the test is still in progress...
if i understand correctly, this patch did not make it to 1.3.1? Correct. This patch was checked in only to 389-ds-base-1.3.2 branch and newer.
I've just installed RHEL7 with system-supplied 389ds (389-ds-base-1.3.1.6-25.el7.x86_64) and the bug is still present. Do i need to use the 1.3.2 branch (not too stable?) in order to eliminate the bug? Correct, again. It is getting stabler. But the test is still in progress...
Thanks, --noriko
Metadata Update from @nhosoi: - Issue assigned to nhosoi - Issue set to the milestone: 1.3.2 - 04/13 (April)
389-ds-base is moving from Pagure to Github. This means that new issues and pull requests will be accepted only in 389-ds-base's github repository.
This issue has been cloned to Github and is available here: - https://github.com/389ds/389-ds-base/issues/650
If you want to receive further updates on the issue, please navigate to the github issue and click on subscribe button.
subscribe
Thank you for understanding. We apologize for all inconvenience.
Metadata Update from @spichugi: - Issue close_status updated to: wontfix (was: Fixed)
Login to comment on this ticket.