#8116 Pylint parallel execution with custom plugin
Closed: fixed 2 years ago by abbra. Opened 2 years ago by slev.

According to Pylint docs:
http://pylint.pycqa.org/en/latest/user_guide/run.html#parallel-execution

There are some limitations in running checks in parallel in the current implementation. It is not possible to use custom plugins (i.e. --load-plugins option)...

That is the IPA case:
1) jobs count is set in pylintrc as:

jobs=0

2) Pylint is triggered by the tox task or make targets :

--load-plugins pylint_plugins

Since Pylint-2.4.3 I have unstable linting results for the IPA code on system having many (for example, 32/64/128) CPU cores, for example:

************* Module ipalib.request                                             
ipalib/request.py:78: [E1101(no-member), destroy_context] Instance of 'int' has no 'disconnect' member)
ipalib/request.py:78: [E1101(no-member), destroy_context] Instance of 'bool' has no 'disconnect' member)
ipalib/request.py:78: [E1101(no-member), destroy_context] Instance of 'dict' has no 'disconnect' member)

'Unstable' means that one time - PASS, another - FAIL, different errors...
While on systems with 2 cores (for example, Azure Pipelines) errors are rare, but happen.
The previous Pylints somehow 'work' in all clauses.

Next, I've checked Pylint 2.3.1 and 2.4.3 in one process mode.
Here are new R ((R) refactor, for bad code smell) warnings:

************* Module ipalib.install.kinit
ipalib/install/kinit.py:1: [R0401(cyclic-import), ] Cyclic import (ipaclient.frontend -> ipalib -> ipaclient.remote_plugins -> ipaclient.remote_plugins.schema))
ipalib/install/kinit.py:1: [R0401(cyclic-import), ] Cyclic import (ipaclient.plugins.rpcclient -> ipalib.rpc -> ipalib -> ipaclient.remote_plugins))
ipalib/install/kinit.py:1: [R0401(cyclic-import), ] Cyclic import (ipaclient.frontend -> ipalib -> ipaclient.remote_plugins -> ipaclient.remote_plugins.compat))
ipalib/install/kinit.py:1: [R0401(cyclic-import), ] Cyclic import (ipatests.pytest_ipa.integration.config -> ipatests.pytest_ipa.integration.env_config))
ipalib/install/kinit.py:1: [R0401(cyclic-import), ] Cyclic import (ipatests.pytest_ipa.integration.config -> ipatests.pytest_ipa.integration.host -> ipatests.pytest_ipa.integration.env_config))

Obviously, that we should run Pylint in 1 process mode.
Of course, this leads to more longer linting, for comparison:
1 process - 14:41 min
2 processes - 7:48 min
4 processes - 4:19 min


Current status.
1) The single process mode's issue which was tracked at https://github.com/PyCQA/pylint/issues/3245 was fixed at https://github.com/PyCQA/astroid/pull/733 (merged)
2) request.context issue exposed as

************* Module ipalib.request                                             
ipalib/request.py:78: [E1101(no-member), destroy_context] Instance of 'int' has no 'disconnect' member)
ipalib/request.py:78: [E1101(no-member), destroy_context] Instance of 'bool' has no 'disconnect' member)
ipalib/request.py:78: [E1101(no-member), destroy_context] Instance of 'dict' has no 'disconnect' member)

Fix proposed in https://github.com/freeipa/freeipa/pull/3783.

3) Last known issue is pylint#3232
Need to investigate deeply, but there are strong arguments:

https://github.com/PyCQA/pylint/blob/master/pylintrc
https://github.com/PyCQA/astroid/blob/master/pylintrc

Both projects use themselves with jobs=1...

master:

  • 292d686 pytest: Migrate xunit-style setups to Pytest fixtures
  • fec6694 pytest: Migrate unittest/nose to Pytest fixtures
  • 8c7447f pytest: Warn about unittest/nose/xunit tests
  • 43a9708 Update Azure Pipelines to use Fedora 31
  • 1946278 ipatests: Properly kill gpg-agent
  • 92b440a pylint: Teach Pylint how to handle request.context
  • e128e7d pylint: Synchronize pylint plugin to ipatests code
  • a309de6 pylint: Clean up comment
  • ba12165 lint: Make Pylint-2.4 happy again

Metadata Update from @abbra:
- Issue close_status updated to: fixed
- Issue status updated to: Closed (was: Open)

2 years ago

ipa-4-8:

  • 87bc314 pytest: Migrate xunit-style setups to Pytest fixtures
  • 356f907 pytest: Migrate unittest/nose to Pytest fixtures
  • 3659b46 pytest: Warn about unittest/nose/xunit tests
  • f4e2acd Update Azure Pipelines to use Fedora 31
  • 294694a ipatests: Properly kill gpg-agent
  • 5939c90 pylint: Teach Pylint how to handle request.context
  • 3460db4 pylint: Synchronize pylint plugin to ipatests code
  • 6f48848 pylint: Clean up comment
  • 44a59ff lint: Make Pylint-2.4 happy again

Login to comment on this ticket.

Metadata