https://bugzilla.redhat.com/show_bug.cgi?id=823882 (Red Hat Directory Server)
RFE Request 0. Proposed title of this feature request Feature Request: better memory management in logconv.pl 2. What is the nature and description of the request? Feature Request: better memory management in logconv.pl 3. Why does the customer need this? (List the business requirements here) Since with Directory Server Red Hat offer an enterprise solution, a company should be able to expect enterprise behaviour, in this case that is that the product won't break from regular use. This is what's happening with the use of logconv.pl: we use it to handle an undefined amount of logs, and we expect it to do anyhing BUT break the system and cause unavailability. Whether it be an error message (too many logs), a better handling of the logs (i.e. in chunks) or a better memory management, all is acceptable. For a script like this to crash a server is unacceptable in an enterprise environmemt. 4. How would the customer like to achieve this? (List the functional requirements here) Customer is expecting better memory Management in logconv.pl in below ways. - refusing to start processing logs altogether when logconv.pl calculates the logs are too big to be processed in memory - processing logs in sizeable chunks and later adding these processed chunks together to do a final report - fine-tune the memory usage of logconv.pl - finding temporary diskspace to flush parts of memory (of course with the option of still one of the previous three when disk space is too low) 5. For each functional requirement listed in question 4, specify how Red Hat and the customer can test to confirm the requirement is successfully implemented. Create a replica setup which generate 5 GB of daily logs & try to run logconv.pl. 6. Is there already an existing RFE upstream or in Red Hat bugzilla? No 7. How quickly does this need resolved? (desired target release) No time-line 8. Does this request meet the RHEL Bug and Feature Inclusion Criteria (please review) Yes 9. List the affected packages 389-ds-base 10. Would the customer be able to assist in testing this functionality if implemented? Yes
set default ticket origin to Community
Added initial screened field value.
I've designed logconv to write the verbose data to disk(default is /tmp). This keeps the memory relatively stable despite the size of the logs, but a lot of disk space is consumed. There is also a slight performance hit when printing the report as the formatted data has to read in from disk.
Here are the memory usage results:
Logconv.pl 6.1:
1 million lines: 2843256 k 5 million lines: 5118708 k
Logconv.pl 7.0:
1 million lines: 2026828 k 5 million lines: 2029848 k
5 million line Log:
log size: 126724k data size: 582764k (4.5 times larger that log size)
New logconv looks great!
The utility looks brand new. Probably, you may want to replace the copyright year with 2013? ;) 38 # Copyright (C) 2005 Red Hat, Inc.
git comment: Bug Description: logconv.pl memory usage is very high, and doesn't doesn't appear to be efficient. ^^^^^^^
Thanks for the review Noriko! Changed copyright, and fixed commit message.
git merge ticket419 Updating 4723735..16651d6 Fast-forward ldap/admin/src/logconv.pl | 2639 +++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 2639 insertions(+), 0 deletions(-) create mode 100755 ldap/admin/src/logconv.pl
[mareynol@localhost src]$ git push origin master Counting objects: 10, done. Delta compression using up to 4 threads. Compressing objects: 100% (5/5), done. Writing objects: 100% (6/6), 19.69 KiB, done. Total 6 (delta 3), reused 0 (delta 0) To ssh://git.fedorahosted.org/git/389/ds.git 4723735..16651d6 master -> master
New logconv.pl 0001-Ticket-419-logconv.pl-improve-memory-management.patch
Ticket has been cloned to Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1013134
To ssh://git.fedorahosted.org/git/389/ds.git fbece32..313dd8e 389-ds-base-1.2.11 -> 389-ds-base-1.2.11 commit aba08ee Author: Mark Reynolds mreynolds@redhat.com Date: Tue Jan 15 15:36:33 2013 -0500 commit 3badb6c Author: Mark Reynolds mreynolds@redhat.com Date: Tue Jan 15 12:41:23 2013 -0500
To ssh://git.fedorahosted.org/git/389/ds.git c96eaa0..9103b3e 389-ds-base-1.3.0 -> 389-ds-base-1.3.0 commit 9866682 Author: Mark Reynolds mreynolds@redhat.com Date: Tue Jan 15 15:36:33 2013 -0500 commit fc3c919 Author: Mark Reynolds mreynolds@redhat.com Date: Tue Jan 15 12:41:23 2013 -0500
Reopening... See bz1013134 for details.
The patch is already in the 1.2.11 branch! No need to reopen it. :)
Metadata Update from @mreynolds: - Issue assigned to mreynolds - Issue set to the milestone: 1.2.11.30
389-ds-base is moving from Pagure to Github. This means that new issues and pull requests will be accepted only in 389-ds-base's github repository.
This issue has been cloned to Github and is available here: - https://github.com/389ds/389-ds-base/issues/419
If you want to receive further updates on the issue, please navigate to the github issue and click on subscribe button.
subscribe
Thank you for understanding. We apologize for all inconvenience.
Metadata Update from @spichugi: - Issue close_status updated to: wontfix (was: Fixed)
Login to comment on this ticket.