#543 Sorting with attributes in ldapsearch gives incorrect result
Closed: Fixed None Opened 7 years ago by kavichitra.

I have performed sorting in ldapsearch operation with all the combination of two attributes (sn, telephonenumber). I have made the 'sn' values of all the entries same and the 'telephonenumber' values different so that the sorting order for all the search results will be same. But the sorting order for the search results are not same as expected. When I sort with "sn", "sn telephonenumber" and "telephonenumber sn" the ordering of the search results are same. The search results of "telephonenumber" sort is different from other results. At least the search results for sorting with "telephonenumber" and "telephonenumber sn" should give similar results as all the entries have same 'sn' value.
Note: It is reproduced in 389 DS-1.2.15

Reproducer details:

The data I have uploaded to the database is given below
dirsrv12@dirsrv12-VirtualBox:~$ cat data.ldif
dn: uid=scarter, ou=People, dc=asiapacific,dc=hpqcorp,dc=net
cn: Sam Carter
sn: Test sn
givenname: Sam
objectclass: top
objectclass: person
objectclass: organizationalPerson
objectclass: inetOrgPerson
uid: scarter
telephonenumber: 12

dn: uid=tmorris, ou=People, dc=asiapacific,dc=hpqcorp,dc=net
cn: Ted Morris
sn: Test sn
givenname: Ted
objectclass: top
objectclass: person
objectclass: organizationalPerson
objectclass: inetOrgPerson
uid: tmorris
telephonenumber: 1 3

dn: uid=kvaughan, ou=People, dc=asiapacific,dc=hpqcorp,dc=net
cn: Ken kvaughan
sn: Test sn
givenname: Kirsten
objectclass: top
objectclass: person
objectclass: organizationalPerson
objectclass: inetOrgPerson
uid: kvaughan
telephonenumber: 1 23

The sorting search results for all the combinations are given below
Sorting with sn
dirsrv12@dirsrv12-VirtualBox:~$ ldapsearch -h dirsrv12-VirtualBox -p 389 -b "ou=people,dc=asiapacific,dc=hpqcorp,dc=net" -s one -S "sn" -x "(objectclass=inetorgperson)" sn telephonenumber

extended LDIF

LDAPv3

base <ou=people,dc=asiapacific,dc=hpqcorp,dc=net> with scope oneLevel

filter: (objectclass=inetorgperson)

requesting: sn telephonenumber

scarter, People, asiapacific.hpqcorp.net

dn: uid=scarter,ou=People,dc=asiapacific,dc=hpqcorp,dc=net
sn: Test sn
telephonenumber: 12

tmorris, People, asiapacific.hpqcorp.net

dn: uid=tmorris,ou=People,dc=asiapacific,dc=hpqcorp,dc=net
sn: Test sn
telephonenumber: 1 3

kvaughan, People, asiapacific.hpqcorp.net

dn: uid=kvaughan,ou=People,dc=asiapacific,dc=hpqcorp,dc=net
sn: Test sn
telephonenumber: 1 23

search result

search: 2
result: 0 Success

numResponses: 4

numEntries: 3

Sorting with telephonenumber
dirsrv12@dirsrv12-VirtualBox:~$ ldapsearch -h dirsrv12-VirtualBox -p 389 -b "ou=people,dc=asiapacific,dc=hpqcorp,dc=net" -s one -S "telephonenumber" -x "(objectclass=inetorgperson)" sn telephonenumber

extended LDIF

LDAPv3

base <ou=people,dc=asiapacific,dc=hpqcorp,dc=net> with scope oneLevel

filter: (objectclass=inetorgperson)

requesting: sn telephonenumber

kvaughan, People, asiapacific.hpqcorp.net

dn: uid=kvaughan,ou=People,dc=asiapacific,dc=hpqcorp,dc=net
sn: Test sn
telephonenumber: 1 23

tmorris, People, asiapacific.hpqcorp.net

dn: uid=tmorris,ou=People,dc=asiapacific,dc=hpqcorp,dc=net
sn: Test sn
telephonenumber: 1 3

scarter, People, asiapacific.hpqcorp.net

dn: uid=scarter,ou=People,dc=asiapacific,dc=hpqcorp,dc=net
sn: Test sn
telephonenumber: 12

search result

search: 2
result: 0 Success

numResponses: 4

numEntries: 3

Sorting with telephonenumber sn
dirsrv12@dirsrv12-VirtualBox:~$ ldapsearch -h dirsrv12-VirtualBox -p 389 -b "ou=people,dc=asiapacific,dc=hpqcorp,dc=net" -s one -S "telephonenumber sn" -x "(objectclass=inetorgperson)" sn telephonenumber

extended LDIF

LDAPv3

base <ou=people,dc=asiapacific,dc=hpqcorp,dc=net> with scope oneLevel

filter: (objectclass=inetorgperson)

requesting: sn telephonenumber

scarter, People, asiapacific.hpqcorp.net

dn: uid=scarter,ou=People,dc=asiapacific,dc=hpqcorp,dc=net
sn: Test sn
telephonenumber: 12

tmorris, People, asiapacific.hpqcorp.net

dn: uid=tmorris,ou=People,dc=asiapacific,dc=hpqcorp,dc=net
sn: Test sn
telephonenumber: 1 3

kvaughan, People, asiapacific.hpqcorp.net

dn: uid=kvaughan,ou=People,dc=asiapacific,dc=hpqcorp,dc=net
sn: Test sn
telephonenumber: 1 23

search result

search: 2
result: 0 Success

numResponses: 4

numEntries: 3

Sorting with sn telephonenumber
dirsrv12@dirsrv12-VirtualBox:~$ ldapsearch -h dirsrv12-VirtualBox -p 389 -b "ou=people,dc=asiapacific,dc=hpqcorp,dc=net" -s one -S "sn telephonenumber" -x "(objectclass=inetorgperson)" sn telephonenumber

extended LDIF

LDAPv3

base <ou=people,dc=asiapacific,dc=hpqcorp,dc=net> with scope oneLevel

filter: (objectclass=inetorgperson)

requesting: sn telephonenumber

scarter, People, asiapacific.hpqcorp.net

dn: uid=scarter,ou=People,dc=asiapacific,dc=hpqcorp,dc=net
sn: Test sn
telephonenumber: 12

tmorris, People, asiapacific.hpqcorp.net

dn: uid=tmorris,ou=People,dc=asiapacific,dc=hpqcorp,dc=net
sn: Test sn
telephonenumber: 1 3

kvaughan, People, asiapacific.hpqcorp.net

dn: uid=kvaughan,ou=People,dc=asiapacific,dc=hpqcorp,dc=net
sn: Test sn
telephonenumber: 1 23

search result

search: 2
result: 0 Success

numResponses: 4

numEntries: 3


I think the telephone number sorting is wrong, in the telephonenumber syntax spaces and hyphens are insignificant and sorting should be based on the normalized values, so the order should be 12 123 13.

Is the combined sorting for other attrs like -S "sn givenname" working as expected?

Replying to [comment:1 lkrispen]:

I think the telephone number sorting is wrong, in the telephonenumber syntax spaces and hyphens are insignificant and sorting should be based on the normalized values, so the order should be 12 123 13.

Is the combined sorting for other attrs like -S "sn givenname" working as expected?

Sorry for the late reply.

I tried sorting by removing the spaces in telephonenumber. As the "sn" values are same for all the entries, the expected results for sorting with "telephonenumber sn" and "telephonenumber" should be same. But the search results are different for sorting with "telephonenumber sn" and "telephonenumber". For sorting with "telephonenumber sn" the order is 12 13 123 and with "telephonenumber" the order is 12 123 13.

I also checked the combined sorting for attributes sn and givenname. And I have made "sn" values of all the entries same. Here also the order for sorting with "givenname sn" and "givenname" are different.

In the case of ldapsearch sort with two attributes like telephonenumber and sn, the sort control will take first two entries and compare their “telephonenumber” values. If “telephonenumber” values are same then it will compare the “sn” values for the same entries.

When I have gone through the code, I found that for each attribute value type, there are separate functions to compare the values. For example tel_compare function will be called to compare two telephonenumber values and cis_compare function will be called to compare two sn values.

The values of two entries are compared in the function '''sort_attr_compare()''', input to the fumction are the two values to be compared and the compare function (eg.tel_compare).While calling the function sort_attr_compare, '''s->compare_fn''' is passed instead of using the local pointer '''this_one->compare_fn'''. I have provided the file name and the other details below. Can you please verify and give a clarification whether the below problem will create incorrect results in sorting ?

In the file /ldap/servers/slapd/back-ldbm/sort.c

static int compare_entries_sv(ID id_a, ID id_b, sort_spec s,baggage_carrier bc, int error)
{
.
.
for (this_one = (sort_spec_thing
)s; this_one ; this_one = this_one->next)
{

/ s – linked list which contains the sort attributes and address of their corresponding compare function in compare_fn
this_one – local pointer to move through the linked list
/
.
.
.
result = sort_attr_compare(value_a, value_b, s->compare_fn); / '''this_one->compare_fn''' should come instead of '''s->compare_fn''' /
.
.
}
.
.
}

result = sort_attr_compare(value_a, value_b, s->compare_fn); / this_one->compare_fn should come instead of s->compare_fn /

A good catch! You are right. We are using the wrong syntax there...

BTW, the code you mentioned is the implementation for the server side sorting. This command-line you are running does NOT use the server side sorting functionality. This sorting is done in the ldapsearch client code.
$ ldapsearch -h dirsrv12-VirtualBox? -p 389 -b "ou=people,dc=asiapacific,dc=hpqcorp,dc=net" -s one -S "telephonenumber sn" -x "(objectclass=inetorgperson)" sn telephonenumber

To enable the server side sorting, please use the search extension:
$ man ldapsearch
-E [!]ext[=extparam]
"[!]sss=[-]<attr[:oid]>[/[-]<attr[:oid]>...] (server side sorting)
I.e., this command-line would do the job.
$ ldapsearch -h dirsrv12-VirtualBox? -p 389 -b "ou=people,dc=asiapacific,dc=hpqcorp,dc=net" -s one -E "sss=telephonenumber/sn" -x "(objectclass=inetorgperson)" sn telephonenumber

I'm attaching the patch next...

Bug description: In the server side sorting compare function
compare_entries_sv, if multiple attribute types are specified,
the sort spec for each attribute is scanned one by one in the
for loop. In the for loop, instead of using the "current"
spec, the first spec is kept using. If the attribute types
have different syntaxes (e.g., cis, tel), then the first
syntax is unexpectedly selected for the second syntax.

Fix description: This patch correctly uses the current spec
in the for loop.

Reviewed by Nathan (Thank you!!)

Pushed to master: commit 95f425d

In the file /ldap/servers/slapd/back-ldbm/sort.c

The behaviour which was seen in the function '''compare_entries_sv''' also exists in this function '''compare_entries'''. If multiple attribute types are specified, the sort spec for each attribute is scanned one by one in the for loop. In the for loop, instead of using the "current" spec, the first spec is kept using.

Line no: 584 and 587

static int compare_entries(ID id_a, ID id_b, sort_spec s,baggage_carrier bc, int error)
{
.
.
.
for (this_one = (sort_spec_thing
)s; this_one ; this_one = this_one->next) {
.
.
.
if (!order) {
result = sort_attr_compare(value_a, value_b, '''s->compare_fn''');
} else {
/ If reverse, invert the sense of the comparison /
result = sort_attr_compare(value_b, value_a, '''s->compare_fn''');
}
.
.
}
.
.
.
}

Can someone verify this ?

In the source code (all the active branches), the function "compare_entries" is in "#if 0" with this comment:
/ USE THE _SV VERSION NOW /

So, the problematic function is not being used. Sorry about the confusion. We could fix the line even in the comment since it could be resurrected some time in the future.

Thank you for the report!

Note: the unused "compare_entries" in "#if 0" was removed to reduce confusion.

Pushed to 389-ds-base-1.2.11:
a682fe5..855d289 389-ds-base-1.2.11 -> 389-ds-base-1.2.11
commit 855d289

Pushed to 389-ds-base-1.3.0:
6d6354f..8356792 389-ds-base-1.3.0 -> 389-ds-base-1.3.0
commit 8356792

1.3.1 and newer already have this fix.

Metadata Update from @nhosoi:
- Issue assigned to nhosoi
- Issue set to the milestone: 1.2.11.22

2 years ago

Login to comment on this ticket.

Metadata