#48003 lib389 - build "suite" framework
Closed: Fixed None Opened 5 years ago by mreynolds.

Create a test suite, similar to TET's "basic" acceptance suite, to lib389. This test suite should perform a variety of basic/common operations and tasks.

Also, start the overall framework for "suite" tests(TET porting)


Looks great. A couple of trivial comments/quiestions...

1) It'd be nice to have a clean dse.ldif in the repo:
We don't need these attributes: creatorsName, modifiersName, createTimestamp, modifyTimestamp, numSubordinates.
We don't need o=NetscapeRoot related acis.
The value of nsslapd-pluginVersion could be none.

2) If we could also provide a standard style for checking the result might be nice?
E.g., the result of each operation in test_basic_ops could be checked by search, but we could just call a standard api for verifying the expected entry's existence (or not)?
Having the count of the entries in a db instance would be useful, too.
Maybe, we could add a doc how to add such a shared library functions to lib389?

3) clu/db2ldif_test.py is not finished yet?

    75      # test 1 function... 
    76      # test 2 function... 
    77      # ...

4) This question is more on my current task... I'm so glad you added cleanallruv testsuite. I'm now hoping to add a test case to check shutting down the servers while running the tasks (both cleanallruv and cleanruv_abort) does not crash the servers (ticket48005), do you think I could piggyback on the testsuite to check it? Or we'd better copy part of the testsuite and do the test separately?

Replying to [comment:3 nhosoi]:

Looks great. A couple of trivial comments/quiestions...

1) It'd be nice to have a clean dse.ldif in the repo:
We don't need these attributes: creatorsName, modifiersName, createTimestamp, modifyTimestamp, numSubordinates.
We don't need o=NetscapeRoot related acis.
The value of nsslapd-pluginVersion could be none.

I can clean it up. It is a broken dse.ldif that is meant to stop the server from starting up. It probably only needs one entry :-)

2) If we could also provide a standard style for checking the result might be nice?

You mean checking for an exception?

E.g., the result of each operation in test_basic_ops could be checked by search, but we could just call a standard api for verifying the expected entry's existence (or not)?
Having the count of the entries in a db instance would be useful, too.

Like a objectclass=* search on a backend?

Sorry I'm not sure I'm fully understanding your questions

Maybe, we could add a doc how to add such a shared library functions to lib389?

Like how to add a function to utils.py, or to the DirSrv object?

3) clu/db2ldif_test.py is not finished yet?

No, not yet, I was going to start writing skeleton scripts, but decided to wait. This patch is already quite large. I almost saw this patch as phase 1 of many phases.

  75      # test 1 function... 
  76      # test 2 function... 
  77      # ...

4) This question is more on my current task... I'm so glad you added cleanallruv testsuite. I'm now hoping to add a test case to check shutting down the servers while running the tasks (both cleanallruv and cleanruv_abort) does not crash the servers (ticket48005), do you think I could piggyback on the testsuite to check it? Or we'd better copy part of the testsuite and do the test separately?

It should definitely be in the suite, because that is also a feature of cleanallruv. You can stop the server in the middle of task, and it should resume at server startup. If you don't mind, I'll write that testcase(unless you already have one).

I can clean it up. It is a broken dse.ldif that is meant to stop the server from starting up. It probably only needs one entry :-)

Ah, I see. I thought it could be used as a template. If it is just a broken dse.ldif, never mind!

You mean checking for an exception?

Well, my thought is more basic. Like an added entry is really found in the db or a moved entry is really moved to the new location, or ... that level.

Test cases are often formed like:
set up a test env
do some test
get ther result
verify the result (e.g., comparing the result with some expected result)
I thought the verification part could be somehow formalized. But probably, it depends upon each test case.

Rather "Like how to add a function to utils.py, or to the DirSrv? object?" <-- this would work better -- sharing handy utilities!!

If you don't mind, I'll write that testcase(unless you already have one).

Thank you soooooo much, Mark!! I'm still working on the standalone cases. If you could cover the cleanruv tasks, I'd be much more confident on the patch being added!

$ py.test -v
=============================== test session starts ===
platform linux2 -- Python 2.7.5 -- pytest-2.4.2 -- /usr/bin/python
collected 61 items

basic/basic_test.py:72: test_basic_init PASSED
basic/basic_test.py:89: test_basic_ops PASSED
basic/basic_test.py:204: test_basic_import_export PASSED
basic/basic_test.py:270: test_basic_backup PASSED
basic/basic_test.py:306: test_basic_acl PASSED
basic/basic_test.py:427: test_basic_searches PASSED
basic/basic_test.py:465: test_basic_referrals PASSED
basic/basic_test.py:526: test_basic_systemctl PASSED
basic/basic_test.py:607: test_basic_ldapagent PASSED
basic/basic_test.py:642: test_basic_dse PASSED
basic/basic_test.py:662: test_basic_final PASSED
betxns/betxn_test.py:51: test_betxn_init PASSED
betxns/betxn_test.py:60: test_betxt_7bit PASSED
betxns/betxn_test.py:115: test_betxn_attr_uniqueness PASSED
betxns/betxn_test.py:169: test_betxn_final PASSED
clu/clu_test.py:51: test_clu_init PASSED
clu/clu_test.py:59: test_clu_pwdhash PASSED
clu/clu_test.py:84: test_clu_final PASSED
clu/db2ldif_test.py:51: test_db2ldif_init PASSED
clu/db2ldif_test.py:59: test_db2ldif_final PASSED
config/config_test.py:51: test_config_init PASSED
config/config_test.py:58: test_config_listen_backport_size PASSED
config/config_test.py:109: test_config_deadlock_policy PASSED
config/config_test.py:164: test_config_final PASSED
dynamic-plugins/test_dynamic_plugins.py:74: test_dynamic_plugins PASSED
dynamic-plugins/test_dynamic_plugins.py:457: test_dynamic_plugins_final PASSED
filter/filter_test.py:51: test_filter_init PASSED
filter/filter_test.py:58: test_filter_escaped PASSED
filter/filter_test.py:102: test_filter_search_original_attrs PASSED
filter/filter_test.py:124: test_filter_final PASSED
memory_leaks/range_search_test.py:52: test_range_search_init PASSED
memory_leaks/range_search_test.py:76: test_range_search PASSED
memory_leaks/range_search_test.py:128: test_range_search_final PASSED
password/password_test.py:51: test_password_init PASSED
password/password_test.py:59: test_password_delete_specific_password PASSED
password/password_test.py:118: test_password_final PASSED
password/pwdAdmin_test.py:62: test_pwdAdmin_init PASSED
password/pwdAdmin_test.py:173: test_pwdAdmin PASSED
password/pwdAdmin_test.py:388: test_pwdAdmin_final PASSED
password/pwdPolicy_test.py:51: test_pwdPolicy_init PASSED
password/pwdPolicy_test.py:58: test_pwdPolicy_final PASSED
replication/cleanallruv_test.py:456: test_cleanallruv_init PASSED
replication/cleanallruv_test.py:519: test_cleanallruv_clean PASSED
replication/cleanallruv_test.py:643: test_cleanallruv_clean_restart PASSED
replication/cleanallruv_test.py:791: test_cleanallruv_clean_force PASSED
replication/cleanallruv_test.py:930: test_cleanallruv_abort PASSED
replication/cleanallruv_test.py:1040: test_cleanallruv_abort_restart PASSED
replication/cleanallruv_test.py:1157: test_cleanallruv_abort_certify PASSED
replication/cleanallruv_test.py:1294: test_cleanallruv_stress_clean PASSED
replication/cleanallruv_test.py:1460: test_cleanallruv_final PASSED
rootdn_plugin/rootdn_plugin_test.py:54: test_rootdn_init PASSED
rootdn_plugin/rootdn_plugin_test.py:113: test_rootdn_access_specific_time PASSED
rootdn_plugin/rootdn_plugin_test.py:195: test_rootdn_access_day_of_week PASSED
rootdn_plugin/rootdn_plugin_test.py:280: test_rootdn_access_denied_ip PASSED
rootdn_plugin/rootdn_plugin_test.py:351: test_rootdn_access_denied_host PASSED
rootdn_plugin/rootdn_plugin_test.py:421: test_rootdn_access_allowed_ip PASSED
rootdn_plugin/rootdn_plugin_test.py:495: test_rootdn_access_allowed_host PASSED
rootdn_plugin/rootdn_plugin_test.py:568: test_rootdn_config_validate PASSED
rootdn_plugin/rootdn_plugin_test.py:746: test_rootdn_final PASSED
schema/test_schema.py:152: test_schema_comparewithfiles PASSED
schema/test_schema.py:196: test_schema_final PASSED

======================= 61 passed in 1296.17 seconds =====

Very nice to see all the tests PASSED!

So, if we add a "new" directory in the suites and "new_test.py" under it. Is it executed automagically?
dirsrvtests/suites/new/new_test.py

Replying to [comment:8 nhosoi]:

Very nice to see all the tests PASSED!

So, if we add a "new" directory in the suites and "new_test.py" under it. Is it executed automagically?
dirsrvtests/suites/new/new_test.py

Yes, if you use py.test. I ran "py.test -v" in /ds/dirsrvtests/suites - it recursively looks for all the files with "test" in their names, and executes them. If I would run "py.test -v" in ds/dirsrvtests, it would also execute all the ticket tests as well.

Replying to [comment:9 mreynolds]:

Replying to [comment:8 nhosoi]:

Very nice to see all the tests PASSED!

So, if we add a "new" directory in the suites and "new_test.py" under it. Is it executed automagically?
dirsrvtests/suites/new/new_test.py

Yes, if you use py.test. I ran "py.test -v" in /ds/dirsrvtests/suites - it recursively looks for all the files with "test" in their names, and executes them. If I would run "py.test -v" in ds/dirsrvtests, it would also execute all the ticket tests as well.

Beautiful! Thanks, Mark!!

1557df5..d33676f master -> master
commit d33676f
Author: Mark Reynolds mreynolds@redhat.com
Date: Tue Feb 17 12:18:50 2015 -0500

6a315fe..8e44437 389-ds-base-1.3.3 -> 389-ds-base-1.3.3
commit 8e44437

Reopening, need to add template scripts for all the plugins, and common functional areas. That way its more obvious where community members should add tests.

To ssh://git.fedorahosted.org/git/389/ds.git
430410d..6dcc12c master -> master
commit 6dcc12c
Author: Mark Reynolds mreynolds@redhat.com
Date: Wed Feb 18 17:32:05 2015 -0500

8e44437..20888a6 389-ds-base-1.3.3 -> 389-ds-base-1.3.3
commit 20888a6

Milestone lib389 1.0 deleted

Metadata Update from @mreynolds:
- Issue assigned to mreynolds

3 years ago

Login to comment on this ticket.

Metadata