#47912 Proper handling of "No original_tombstone for changenumber" errors
Closed: wontfix None Opened 7 years ago by amessina.

Using two fully updated FreeIPA F20 masters with freeipa-server-3.3.5-1.fc20.x86_64 and 389-ds-base-1.3.2.23-1.fc20.x86_64, I've noticed that I'm getting a lot of the following errors in the 389 DS error log, especially at server startup.

"No original_tombstone for changenumber=511851,cn=changelog!!"

Most often, the same changenumber repeats over and over, but there are some other changenumbers listed as well. The changenumbers are different on each master. Sometimes the logging loops repeatedly for about 10 minutes or so, running the CPU to 100%

My concern is I'm preparing my thought process about the upcoming upgrade from F20 to F21 and it looks like I may need to create a new FreeIPA master and "migrate" the Dogtag stuff, etc. rather than a simple "yum distro-sync" on each master. Yuck!

What is the proper way to correct for these apparent errors and get these masters working with each other in a clean manner?

Ludwig Krispenz suggested that I open a ticket here.

[28/Sep/2014:09:46:17 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=522680,cn=changelog!!
[28/Sep/2014:09:46:17 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=522681,cn=changelog!!
[28/Sep/2014:09:46:18 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=522689,cn=changelog!!
[28/Sep/2014:09:46:19 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=522691,cn=changelog!!
[28/Sep/2014:09:46:19 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=522693,cn=changelog!!
[28/Sep/2014:09:46:20 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=522694,cn=changelog!!
[28/Sep/2014:09:46:20 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=522695,cn=changelog!!
[28/Sep/2014:09:46:20 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=522697,cn=changelog!!
[28/Sep/2014:09:46:21 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=522698,cn=changelog!!
[28/Sep/2014:09:46:21 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=522699,cn=changelog!!
[28/Sep/2014:09:46:21 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=522702,cn=changelog!!
[28/Sep/2014:09:46:22 -0500] ldbm_back_delete - conn=0 op=0 [retry: 2] No original_tombstone for changenumber=522702,cn=changelog!!
[28/Sep/2014:09:46:22 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=522703,cn=changelog!!
[28/Sep/2014:09:46:22 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=522704,cn=changelog!!
[28/Sep/2014:09:46:28 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=522754,cn=changelog!!
[28/Sep/2014:09:46:28 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=522755,cn=changelog!!
[28/Sep/2014:09:46:29 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=522758,cn=changelog!!
[28/Sep/2014:09:50:22 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=525583,cn=changelog!!
[28/Sep/2014:09:50:22 -0500] ldbm_back_delete - conn=0 op=0 [retry: 2] No original_tombstone for changenumber=525583,cn=changelog!!
[28/Sep/2014:09:50:24 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=525601,cn=changelog!!
[28/Sep/2014:09:51:40 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=526738,cn=changelog!!
[28/Sep/2014:09:52:22 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=527371,cn=changelog!!
[28/Sep/2014:09:52:23 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=527381,cn=changelog!!
[28/Sep/2014:09:52:25 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=527390,cn=changelog!!
[28/Sep/2014:09:52:27 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=527411,cn=changelog!!
[28/Sep/2014:09:52:29 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=527415,cn=changelog!!
[28/Sep/2014:09:52:30 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=527416,cn=changelog!!
[28/Sep/2014:09:52:30 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=527417,cn=changelog!!
[28/Sep/2014:09:52:31 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=527420,cn=changelog!!
[28/Sep/2014:09:52:32 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=527428,cn=changelog!!
[28/Sep/2014:09:52:33 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=527439,cn=changelog!!
[28/Sep/2014:09:52:35 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=527467,cn=changelog!!
[28/Sep/2014:09:52:44 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=527596,cn=changelog!!
[28/Sep/2014:09:52:49 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=527667,cn=changelog!!
[28/Sep/2014:09:52:51 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=527681,cn=changelog!!
[28/Sep/2014:09:53:05 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=527894,cn=changelog!!
[28/Sep/2014:09:53:10 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=527951,cn=changelog!!
[28/Sep/2014:09:53:10 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=527958,cn=changelog!!
[28/Sep/2014:09:53:14 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=527992,cn=changelog!!
[28/Sep/2014:09:53:17 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=528013,cn=changelog!!
[28/Sep/2014:09:53:18 -0500] ldbm_back_delete - conn=0 op=0 [retry: 1] No original_tombstone for changenumber=528021,cn=changelog!!

Adding my comment from the mailing list:

I think the message was introduced when tombstone handling was partly rewritten to fix some bugs and improve entry cache management.
It is logged if a delete transaction has to be retried and no tombstone entry is found. In the normal case this should not happen, but looks like this is in the retro changelog during changelog trimming and for the changelog a delete does not create a tombstone.
In my opinion there should be an additional check before logging this message.

Reviewed by Ludwig (Thank you!!)

Pushed to master:
3104346..36381c1 master -> master
commit 36381c1

Pushed to 389-ds-base-1.3.3:
e7a26dd..7df1729 389-ds-base-1.3.3 -> 389-ds-base-1.3.3
commit 7df1729

added RH
bz 1235847 - handling of "No original_tombstone for changenumber" errors
for a RHEL 7.1 report

Metadata Update from @nhosoi:
- Issue assigned to nhosoi
- Issue set to the milestone: 1.3.3 - 10/31 (October)

5 years ago

389-ds-base is moving from Pagure to Github. This means that new issues and pull requests
will be accepted only in 389-ds-base's github repository.

This issue has been cloned to Github and is available here:
- https://github.com/389ds/389-ds-base/issues/1243

If you want to receive further updates on the issue, please navigate to the github issue
and click on subscribe button.

Thank you for understanding. We apologize for all inconvenience.

Metadata Update from @spichugi:
- Issue close_status updated to: wontfix (was: Fixed)

2 years ago

Login to comment on this ticket.

Metadata