Using two fully updated FreeIPA F20 masters with freeipa-server-3.3.5-1.fc20.x86_64 and 389-ds-base-1.3.2.23-1.fc20.x86_64, I've noticed that I'm getting a lot of the following errors in the 389 DS error log, especially at server startup.
"No original_tombstone for changenumber=511851,cn=changelog!!"
Most often, the same changenumber repeats over and over, but there are some other changenumbers listed as well. The changenumbers are different on each master. Sometimes the logging loops repeatedly for about 10 minutes or so, running the CPU to 100%
My concern is I'm preparing my thought process about the upcoming upgrade from F20 to F21 and it looks like I may need to create a new FreeIPA master and "migrate" the Dogtag stuff, etc. rather than a simple "yum distro-sync" on each master. Yuck!
What is the proper way to correct for these apparent errors and get these masters working with each other in a clean manner?
Ludwig Krispenz suggested that I open a ticket here.
[28/Sep/2014:09:46:17 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=522680,cn=changelog!! [28/Sep/2014:09:46:17 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=522681,cn=changelog!! [28/Sep/2014:09:46:18 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=522689,cn=changelog!! [28/Sep/2014:09:46:19 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=522691,cn=changelog!! [28/Sep/2014:09:46:19 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=522693,cn=changelog!! [28/Sep/2014:09:46:20 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=522694,cn=changelog!! [28/Sep/2014:09:46:20 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=522695,cn=changelog!! [28/Sep/2014:09:46:20 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=522697,cn=changelog!! [28/Sep/2014:09:46:21 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=522698,cn=changelog!! [28/Sep/2014:09:46:21 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=522699,cn=changelog!! [28/Sep/2014:09:46:21 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=522702,cn=changelog!! [28/Sep/2014:09:46:22 -0500] ldbm_back_delete - conn=0 op=0 [retry: 2] No original_tombstone for changenumber=522702,cn=changelog!! [28/Sep/2014:09:46:22 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=522703,cn=changelog!! [28/Sep/2014:09:46:22 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=522704,cn=changelog!! [28/Sep/2014:09:46:28 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=522754,cn=changelog!! [28/Sep/2014:09:46:28 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=522755,cn=changelog!! [28/Sep/2014:09:46:29 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=522758,cn=changelog!! [28/Sep/2014:09:50:22 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=525583,cn=changelog!! [28/Sep/2014:09:50:22 -0500] ldbm_back_delete - conn=0 op=0 [retry: 2] No original_tombstone for changenumber=525583,cn=changelog!! [28/Sep/2014:09:50:24 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=525601,cn=changelog!! [28/Sep/2014:09:51:40 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=526738,cn=changelog!! [28/Sep/2014:09:52:22 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=527371,cn=changelog!! [28/Sep/2014:09:52:23 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=527381,cn=changelog!! [28/Sep/2014:09:52:25 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=527390,cn=changelog!! [28/Sep/2014:09:52:27 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=527411,cn=changelog!! [28/Sep/2014:09:52:29 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=527415,cn=changelog!! [28/Sep/2014:09:52:30 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=527416,cn=changelog!! [28/Sep/2014:09:52:30 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=527417,cn=changelog!! [28/Sep/2014:09:52:31 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=527420,cn=changelog!! [28/Sep/2014:09:52:32 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=527428,cn=changelog!! [28/Sep/2014:09:52:33 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=527439,cn=changelog!! [28/Sep/2014:09:52:35 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=527467,cn=changelog!! [28/Sep/2014:09:52:44 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=527596,cn=changelog!! [28/Sep/2014:09:52:49 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=527667,cn=changelog!! [28/Sep/2014:09:52:51 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=527681,cn=changelog!! [28/Sep/2014:09:53:05 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=527894,cn=changelog!! [28/Sep/2014:09:53:10 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=527951,cn=changelog!! [28/Sep/2014:09:53:10 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=527958,cn=changelog!! [28/Sep/2014:09:53:14 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=527992,cn=changelog!! [28/Sep/2014:09:53:17 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=528013,cn=changelog!! [28/Sep/2014:09:53:18 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=528021,cn=changelog!!
Adding my comment from the mailing list:
I think the message was introduced when tombstone handling was partly rewritten to fix some bugs and improve entry cache management. It is logged if a delete transaction has to be retried and no tombstone entry is found. In the normal case this should not happen, but looks like this is in the retro changelog during changelog trimming and for the changelog a delete does not create a tombstone. In my opinion there should be an additional check before logging this message.
git patch file (master) 0001-Ticket-47912-Proper-handling-of-No-original_tombston.patch
Reviewed by Ludwig (Thank you!!)
Pushed to master: 3104346..36381c1 master -> master commit 36381c1
Pushed to 389-ds-base-1.3.3: e7a26dd..7df1729 389-ds-base-1.3.3 -> 389-ds-base-1.3.3 commit 7df1729
added RH bz 1235847 - handling of "No original_tombstone for changenumber" errors for a RHEL 7.1 report
Metadata Update from @nhosoi: - Issue assigned to nhosoi - Issue set to the milestone: 1.3.3 - 10/31 (October)
389-ds-base is moving from Pagure to Github. This means that new issues and pull requests will be accepted only in 389-ds-base's github repository.
This issue has been cloned to Github and is available here: - https://github.com/389ds/389-ds-base/issues/1243
If you want to receive further updates on the issue, please navigate to the github issue and click on subscribe button.
subscribe
Thank you for understanding. We apologize for all inconvenience.
Metadata Update from @spichugi: - Issue close_status updated to: wontfix (was: Fixed)
Login to comment on this ticket.