#47762 CoS cache re-scanning severely impacts performance
Closed: wontfix None Opened 10 years ago by coordinated.

A newly-installed directory server with 15 classic CoS rules and >200,000 entries suffers poor write performance.

After a small number (typically 6) of consecutive write requests (basic attribute changes to a single entry using ldapmodify) the ns-slapd process hits >100% CPU and stays there for at least 10 seconds per update, and blocks the client process attempting the update.

Enabling the Plug-ins logging shows that the CoS cache is being rebuilt after every attribute change - even though the attribute which is being changed does not have CoS rules.


The RHEL6.5 server is a VMware ESXi VM with 8GB RAM and 2x CPUs, running the latest EPEL package for RHEL6 (v1.2.11.15-32).

All attributes referred to by the CoS rules (mostly custom attributes) have been indexed. Replication has not been configured so far.

This was initially described in [389-users thread]] which [https://lists.fedoraproject.org/pipermail/389-users/2014-April/016922.html|continued here].

I will continue to add more details...

Thanks for looking into this,
Steve


Extract of error log showing CoS cache rebuilds
cos-errors.log

CoS template for user-type descripton and user classes list (relevant attributes only)
UserTypes_CoS_template.ldif

DSE config file, minus password and keys
dse_desensitized.ldif

Custom objectClasses and attributes
99user.ldif

We are still missing the COS templates mentioned in the cos definitions ldif file you provided:

{{{
ou=Departments,ou=Tables,dc=brighton,dc=ac,dc=uk
ou=Buildings,ou=Tables,dc=brighton,dc=ac,dc=uk
ou=Courses,ou=Tables,dc=brighton,dc=ac,dc=uk
ou=StaffTypes,ou=Tables,dc=brighton,dc=ac,dc=uk
ou=OperatorTypes,ou=Tables,dc=brighton,dc=ac,dc=uk
ou=StatusCodes,ou=Tables,dc=brighton,dc=ac,dc=uk
}}}

Sorry for the delay, Mark - now attached. I've only included minimally relevant attributes and a single example of each (the ones which match the example 'st8' user account).
Do let me know if you need any more of the structure - and whether you'd like me to test anything else or build a clearer test case...

Thanks (and much appreciated!),
Steve

Example entry from the usertype table
usertypes-min.ldif

Example entry from the department table
departments-min.ldif

Example entry from the stafftypes table
stafftypes-min.ldif

Example entry from the operatortype table
operatortypes-min.ldif

Example entry from the statuscodes table
statuscodes-min.ldif

Example entry from the buildings table
buildings-min.ldif

One thing I'd forgotten to mention: a quick test of the problem is how quickly the ''dirsrv'' service starts/restarts/stops. On my server, it takes well over a minute for any of these operations, whilst the CoS cache rebuilds.

(Do let me know if more info/test cases would be helpful...)

Unfortunately I can not reproduce the cos cache being rebuild by updating a non-cos attribute on 1.2.11.

[1] I created dc=brighton,dc=ac,dc=uk
[2] I added all the provided LDIFs, although you are still missing two:

[17/Apr/2014:10:17:49 -0400] - Skipping CoS Definition cn=cosFacultyDescription,ou=People,dc=brighton,dc=ac,dc=uk--no CoS Templates found, which should be added before the CoS Definition.
[17/Apr/2014:10:17:49 -0400] - Skipping CoS Definition cn=cosEduPersonAffiliation,ou=People,dc=brighton,dc=ac,dc=uk--no CoS Templates found, which should be added before the CoS Definition.

[3] I added the st8 user, and then I added 200K users(just like st8) to ou=people,dc=brighton,dc=ac,dc=uk. They all correctly return cos attributes/values

[4] I can update the same entry over and over, different entries, etc, but the cos cache does not rebuild, and performance is fine.

Obviously something is missing from my testcase. Maybe you could follow my exact steps in your environment - basically starting over. I'm attaching my ldif from my setup, maybe you can see if you can use it to reproduce your issue.

This is a duplicate of https://fedorahosted.org/389/ticket/47649 = which was not checked into 1.2.11 branch.

Closing this ticket as a duplicate.

Metadata Update from @coordinated:
- Issue assigned to mreynolds
- Issue set to the milestone: 1.2.11.30

7 years ago

389-ds-base is moving from Pagure to Github. This means that new issues and pull requests
will be accepted only in 389-ds-base's github repository.

This issue has been cloned to Github and is available here:
- https://github.com/389ds/389-ds-base/issues/1094

If you want to receive further updates on the issue, please navigate to the github issue
and click on subscribe button.

Thank you for understanding. We apologize for all inconvenience.

Metadata Update from @spichugi:
- Issue close_status updated to: wontfix (was: Duplicate)

4 years ago

Log in to comment on this ticket.

Metadata