Revise our upstream documentation and describe the optimal way to use lib389 and pytest.
Metadata Update from @spichugi: - Issue assigned to spichugi
Metadata Update from @spichugi: - Custom field Origin adjusted to QE - Issue priority set to: Normal
I've attached the draft version of the guide. My proposal is:
put these chapters in reStructuredText format to lib389 as Quick start guide
Basic workflow A ways to make your code better in a pytest way Fixtures Scope Parametrizing Test cases Parametrizing Marking test functions and selecting them for a run Asserting lib389 and python-ldap functions Constants Basic constants Add, Modify, and Delete Operations Search and Bind Operations Basic instance operations Setting up SSL/TLS Replication Basic configuration Replication tools
add docstrings to every module in lib389 and generate the rest of the docs with Sphinx
<img alt="Guidelines-for-using-pytest-and-lib389_47541162.html" src="/lib389/issue/raw/2aef109dbf0f23a359c15f209c6c523cebda4faf583125ff62e0a228f8c3b321-Guidelines-for-using-pytest-and-lib389_47541162.html" />
A super tiny fix on USN:
# Optionaly you can specify 'suffix', 'backend' and 'maxusn_to_delete' usnplugin.cleanup(suffix=None, backend=None, max_usn=None)
It is necessary to specify either suffix or backend otherwise this method will fail. I had a manual check about this in the same method, but we decided to move it inside task's _validate() method instead. Maybe it would be good to put it here as well in order to avoid confusion?
git remote add $(whoami) git+ssh://git.engineering.redhat.com/srv/git/users/$(whoami)/389-ds-base.git
I don't think applies in a public guide ;) similar for mojo links.
Use ./dirsrvtests/create_test.py tool to generate new test.py file. Usage:
Might be a good note here to say to avoid the "tickets" folder?
Go to the "beaker" branch of dirsrv-tests repo, to the beaker/upstream-tests dir http://git.app.eng.bos.redhat.com/git/dirsrv-tests.git
^ Internal only.
For more info check the source code at https://pagure.io/lib389/blob/master/f/lib389/_constants.py . If you need a constant, use, do not import all of them: from lib389._constants import CONSTNATS_YOU_NEED
Could we worth showing the tuple import syntax,
Add, Modify, and Delete Operations
^ "Manual add, modify ..."
Should probably recommend dsldapobjects over this though.
Setting up SSL/TLS
This section is showing CERT external auth. You can remove a fair bit of this section to "focus" it on the SSL/TLS setup. AS well, worth mentioning that standlanoe will detect ldaps if you set secureport.
DSLdapObject operations ... # Other parameters are carried out - it takes 'rdn' from properties, and 'basedn' from parent class users.create(properties=user_properties) # Get one DSLdapObject # with 'selector' user = users.get('testuser')
You can shortcut with user = users.create .... Because we return the newly created object.
# Delete user.delete()
This probably needs a bigger comment warning ;)
Memory Leak Testing (valgrind)
I'd say here that ASAN+LSAN are faster and detect more, plus are 100% seamless to tests. No need for valgrind.
Password
Looks like we are missing PBKDF2_SHA256 here ;)
Otherwise mate, this is looking great. A lot of these can easily become pydoc comments for the related objects. Great work!
I've applied the comments and have added more chapters that were missing (after some IRC discussions) After the review, I'll create rST formatted guide and I'll start to document the code.
The parts with internal actions won't be included in the upsteam rST guide.
<img alt="Guidelines-for-using-pytest-and-lib389_47541162.html" src="/lib389/issue/raw/files/6047ac156e06192837271afc2770a83f7f3206400669537bb1d8f43ab7b9a434-Guidelines-for-using-pytest-and-lib389_47541162.html" />
Metadata Update from @spichugi: - Custom field Review Status adjusted to None
Can't open provided link :(
After clicking it will be downloaded and you can open it with your browser. Also, you can just review the internal version.
<img alt="0001-Issue-77-Add-sphinx-documentation.patch" src="/lib389/issue/raw/files/95c83274b06e9b48993af02040a23194f075e48460efe04dbacfeb50cc2eaa51-0001-Issue-77-Add-sphinx-documentation.patch" />
Most of the comments were applied. Thanks!
Now the layout is: - index.rst - welcoming word, Table of Contents and Contact Us. - guidelines.rst - is included into index.rst and contains basic test development workflow, pytest features overview, and basic lib389 and python-ldap features. - replica.rst, agreement.rst, aci.rst, etc. - each contains the respective documentation.
Currently I do add docstrings to lib389 and rst files with references. It will be the next patch. Now, please, check if you agree with the existing format.
The generated example - https://fedorapeople.org/~spichugi/html/
Metadata Update from @spichugi: - Custom field Review Status adjusted to review (was: None)
Its a good start, ack
Metadata Update from @mreynolds: - Custom field Review Status adjusted to ack (was: review)
commit 3200669 Author: Simon Pichugin spichugi@redhat.com Date: Thu Aug 24 23:05:08 2017 +0200
I just read this, this is great. Thank you so much @spichugi this is great work!
<img alt="0001-Issue-77-Refactor-docstrings-in-rST-format-part-1.patch" src="/lib389/issue/raw/files/2e181d5321f194587899b1cbb37455d86d2754daeacb2d02c6de60aa9830b079-0001-Issue-77-Refactor-docstrings-in-rST-format-part-1.patch" />
Rewriting docstrings. Part 1.
It includes: _mapped_object.py, aci.py, agreement.py, backend.py, changelog.py, mappingTree.py, replica.py, repltools.py
Also, I've added a script for links to the code in the docs.
You can check it also here: https://fedorapeople.org/~spichugi/html/index.html
That looks great. There is a lot of content here, so I think it will take time to review to get it "right".
What's the process if I update a module, does it automatically just use the docstrings from the objects?
I think I'm happy to ack this, because if we start using it, we'll iron out any issues as we go.
Metadata Update from @spichugi: - Custom field Review Status adjusted to review (was: ack)
Yeah, so basically we have this main part in .rst file:
.. autoclass:: lib389.replica.Replica :members: :inherited-members:
:members: will parse all docstrings from public modules (it won't parse _method, and __func__) :inherited-members: will additionaly parse all methods that we have in parent class (but if the method is in our class, it will skip it in the parent).
Sure, feel free to set ack, if you're okay (I forgot to set 'review' yesterday)
Metadata Update from @firstyear: - Custom field Review Status adjusted to ack (was: review)
commit 3a498ae Author: Simon Pichugin spichugi@redhat.com Date: Tue Aug 29 19:29:15 2017 +0200
Somehow this broke the tests. dbdir not found in DirSrv
py.test -v -x -s tests/suites/setup_ds/setup_ds_test.py ================================================================================= test session starts ================================================================================= platform linux2 -- Python 2.7.13, pytest-2.9.2, py-1.4.34, pluggy-0.3.1 -- /usr/bin/python2 cachedir: tests/suites/setup_ds/.cache rootdir: /home/mareynol/source/ds389/389-ds-base/dirsrvtests/tests/suites/setup_ds, inifile: collected 2 items tests/suites/setup_ds/setup_ds_test.py::test_slapd_InstScriptsEnabled[true] INFO:setup_ds_test:set SER_INST_SCRIPTS_ENABLED to true INFO:setup_ds_test:create_instance - Installs the instance and Sets the value of InstScriptsEnabled to true OR false. INFO:setup_ds_test:Set up the instance and set the config_attr FAILED ====================================================================================== FAILURES ======================================================================================= _________________________________________________________________________ test_slapd_InstScriptsEnabled[true] _________________________________________________________________________ config_attr = 'true' @pytest.mark.parametrize("config_attr", ('true', 'false')) def test_slapd_InstScriptsEnabled(config_attr): """Tests InstScriptsEnabled attribute with "True" and "False" options :id: 02faac7f-c44d-4a3e-bf2d-1021e51da1ed :setup: Standalone instance with slapd.InstScriptsEnabled option as "True" and "False" :steps: 1. Execute setup-ds.pl with slapd.InstScriptsEnabled option as "True". 2. Check if /usr/lib64/dirsrv/slapd-instance instance script directory is created or not. 3. Execute setup-ds.pl with slapd.InstScriptsEnabled option as "False". 4. Check if /usr/lib64/dirsrv/slapd-instance instance script directory is created or not. :expectedresults: 1. Instance should be created. 2. /usr/lib64/dirsrv/slapd-instance instance script directory should be created. 3. Instance should be created. 4. /usr/lib64/dirsrv/slapd-instance instance script directory should not be created. """ log.info('set SER_INST_SCRIPTS_ENABLED to {}'.format(config_attr)) > standalone = create_instance(config_attr) tests/suites/setup_ds/setup_ds_test.py:62: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ tests/suites/setup_ds/setup_ds_test.py:23: in create_instance standalone = DirSrv(verbose=False) ../../../lib389/lib389/__init__.py:400: in __init__ self.__add_brookers__() ../../../lib389/lib389/__init__.py:318: in __add_brookers__ self.changelog = Changelog(self) ../../../lib389/lib389/changelog.py:29: in __init__ self.changelogdir = os.path.join(os.path.dirname(self.conn.dbdir), DEFAULT_CHANGELOG_DB) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <lib389.DirSrv object at 0x7f7ae2a22110>, name = 'dbdir' def __getattr__(self,name): if self.CLASSATTR_OPTION_MAPPING.has_key(name): return self.get_option(self.CLASSATTR_OPTION_MAPPING[name]) elif self.__dict__.has_key(name): return self.__dict__[name] else: raise AttributeError,'%s has no attribute %s' % ( > self.__class__.__name__,repr(name) ) E AttributeError: DirSrv has no attribute 'dbdir'
<img alt="0001-Issue-77-Fix-changelogdb-param-issue.patch" src="/lib389/issue/raw/files/486d6f41143e829fde00c722747e95d4ad03efafd0680b1e78f1c95ec693c3c0-0001-Issue-77-Fix-changelogdb-param-issue.patch" />
Thanks this patch seems to resolve the issue, ack
Thanks! Sorry for creating it...
commit bde1530 Author: Simon Pichugin spichugi@redhat.com Date: Tue Sep 12 07:58:23 2017 +0200
No problem, but this points out that we need to run the DS tests before pushing lib389 changes. Now, if we move lib389 inside of the DS source code then that will also cover the testing once 389-ds-base starts using PRs/jenkins, but we aren't there yet :-)
Yes, I agree, I have the exact same thoughts. :) It was a big patch and I forgot about the change... I thought my patch was only about docs that I've executed successfully.
<img alt="0001-Issue-lib389-77-Refactor-docstrings-in-rST-format-pa.patch" src="/lib389/issue/raw/files/a7f959f31cfd6f5b4f837dd715bf1d0beb96a182e30a0c1b13bfc4d3cdfbc905-0001-Issue-lib389-77-Refactor-docstrings-in-rST-format-pa.patch" />
@mreynolds, @firstyear, hi. Where do you think is the best place to store the documentation?
Unfortunately, readthedocs.ie is not an option. It has issue with C libs that python-ldap depends on. http://docs.readthedocs.io/en/latest/faq.html#i-get-import-errors-on-libraries-that-depend-on-c-modules To make it work we need:
If such libraries are installed via setup.py, you also will need to remove all the C-dependent libraries from your install_requires in the RTD environment.
And it's not an option I think.
Manually, the documentation can be generated with the next command:
$ cd src/lib389/doc/ || make html
And then we can upload the directory 'src/lib389/doc/build/html' somewhere on a web server. Probably, we can use Openshift instance for auto-generation per commit.
What do you think?
ack on the patch
@mreynolds, hi. Where do you think is the best place to store the documentation?
Eventually the admin guide :-)
It can be generated with the next command from 'src/lib389/doc/': $ make html And then we can upload the directory 'src/lib389/doc/build/html' somewhere. Probably, we can use Openshift instance for it. Maybe you have something else in your mind?
The wiki would be good for now (after we get the migration done).
Well, I'd love all our docs in an rpm, then we can just "install the rpm" and host on openshift ...
So that would be a good place to get to I think?
I am okay with either of your ideas. Ok, lets put it first to wiki.
commit 5493b8686de9c88975aed07ec6ca9a85f6cd1b76 Author: Simon Pichugin spichugi@redhat.com Date: Fri Nov 3 16:43:32 2017 +0100
Metadata Update from @spichugi: - Assignee reset
Login to comment on this ticket.