The problem with this is that QE uses master branch to test older version of DS. For example, these tests could start failing on 1.4.3...

Nice, this test always took a long time complete.

We are removing all the old perl and shell scripts in the same version, so really these changes are not needed. Just an FYI

Overall this is impressive work and removes a lot of code, But there is something missing. Ludwig did not update the new CLI tools, he just updated the old ones that we are removing in the same release. So this requires more lib389 changes. We could do that in a separate ticket though...

This is a really amazing piece of work, thanks @tbordaz! Something I'd like to see in the test suite is a db2bak for an old version, and a test that does an inplace upgrade to show we handle that case correctly. Is that possible? Perhaps can be another ticket or @vashirov can help?

@firstyear, thanks but the honour is deserved to @elkris.

This patch will introduce incompatibility. I think we should flag a vendorversion so that clients (CI tests, freeipa and others) can manage this.

@firstyear, thanks but the honour is deserved to @elkris.

thanks, but I just made a big change and you will have to deal with it to get it working correctly

This patch will introduce incompatibility. I think we should flag a vendorversion so that clients (CI tests, freeipa and others) can manage this.

@firstyear talks about backup/restore aand I think we could get this to work smoothly with backups from older version.

The biggest incompatibility is the deprecating of cn=changelog5, and if there is no way to handle this gracefully it should be flagged, you're right

maybe I wasn't clear :) I meant a test where we take an old database, and do the inplace upgrade to bring the changelog to the maindb. Does such a test exist?

Right I know two incompatibilities (cn=changelog5 and restore of old backup).

Regarding cn=changelog5, client admin applications (CI tests, freeipa installer, 389ds CLI, admin scripts) need to be modified. WIth that patch, there are several ways to discover what is the server behavior:

• add cn=changelog5 entry (and catch unwilling to perform)
• update the entry (and check that the MOD has no effect)
• check for cn=changelog,cn=.. (but fails for new instance)

IMHO this looks not convenient. We can also define a specific vendorVersion as transition. Or add a new attribute in rootDSE or cn=config.

We can also be smart, catching the client req and simulate expected behavior. But it looks complex and sometime impossible as some CL attributes are now specfic to each changelog.

@mreynolds, taking into account the compatibility of CI tests with older version. It uses a lib389.utils:ds_supports_new_changelog function based on vendorVersion.
If we keep this approach then we will need a new version (1.4.4.4) just before pushing this patch.

Right I know two incompatibilities (cn=changelog5 and restore of old backup).
Regarding cn=changelog5, client admin applications (CI tests, freeipa installer, 389ds CLI, admin scripts) need to be modified. WIth that patch, there are several ways to discover what is the server behavior:

add cn=changelog5 entry (and catch unwilling to perform)
update the entry (and check that the MOD has no effect)
check for cn=changelog,cn=.. (but fails for new instance)

IMHO this looks not convenient. We can also define a specific vendorVersion as transition. Or add a new attribute in rootDSE or cn=config.
We can also be smart, catching the client req and simulate expected behavior. But it looks complex and sometime impossible as some CL attributes are now specfic to each changelog.

The question is how smart we can and need to be. There is almost everything possible.
The only config which will no longer work is sppecifying a directory for the changelog files.

So a first step could be make changelogdir in the changelog objectclass a MAY and not a MUST.
When auto upgrading only remove the directory attribute and base the decision if upgrade needs to be done. When receiving an add of cn=changelog5 accept it but strip the directory attr

About the rest of the config attrs we could control it, eg by adding a config param to the MMR entry: changelogconfig : [specific|global|global_default]
where
specific: is what the patch implements, create and use cn=changelog
global: use the params of the changelog5 entry, either directly or always propagate changes to cn=changelog
global_default: use the values in cn=changelog5 if not specified in the specific config entry, otherwise the specific takes precedenc

With the rebases the status of the patch is:

• Fix so that changelog is specific to each (replicated) backend and use the main database (original patch)
• Fix to related to asan tests: use-after-free when remove/add/set replica, leaks CL trimming and cascading. (it remains few others leaks but will be handle with specific tickets.

• Fix CI tests: compatibility with previous versions, adapt test to new CL result is

  = 93 passed, 2 skipped, 11 xfailed, 6 xpassed, 380 warnings, 5 error in 3884.66 seconds =

  = 93 passed, 2 skipped, 11 xfailed, 6 xpassed, 380 warnings, 5 error in 3884.66 seconds =

If we comment this out can you add a reason why in a comment?

Same for all these below, explain why it's commented out so future readers of the code are not confused. :-)

@mreynolds patch updated. Waiting for last IPA tests results

IPA tests were successful, pushing that huge patch upstream

thank you very much, sorry to have left so much work for you

@elkris I wanted to drop you an email to let you know this great step forward.
Thanks to YOU, you did all job. @mreynolds also helped a lot with integration tests/fix in freeipa.

@elkris It's okay, we'll show up at your doorstep and expect to be fed in repayment for you leaving this work to us :)

Thanks very much @tbordaz and @mreynolds for getting this finished, it's a real achievement to see a huge piece of work like this merged!

389-ds-base is moving from Pagure to Github. This means that new issues and pull requests
will be accepted only in 389-ds-base's github repository.

This pull request has been cloned to Github as issue and is available here:
- https://github.com/389ds/389-ds-base/issues/4234

If you want to continue to work on the PR, please navigate to the github issue,
download the patch from the attachments and file a new pull request.

Thank you for understanding. We apologize for all inconvenience.