Currently, unit tests execution time within Azure Pipelines is not balanced: Base tests take ~13 minutes, while XMLRPC tests ~28 minutes. The grouping is:
Base tests
XMLRPC tests
testsToRun: test_cmdline test_install test_ipaclient test_ipalib test_ipaplatform test_ipapython
and
testsToRun: test_xmlrpc
The same problem exists for Travis, where the grouping is
TESTS_TO_RUN=" test_cmdline test_install test_ipaclient test_ipalib test_ipaplatform test_ipapython test_ipaserver test_xmlrpc/test_[l-z]*.py"
TESTS_TO_RUN="test_xmlrpc/test_[a-k]*.py"
task1 ~ 3 min task2 ~ 9 min
Some sort of autobalancer is required.
Azure Pipelines supports slicing out of the box: https://docs.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml#slicing https://docs.microsoft.com/ru-ru/azure/devops/pipelines/test/parallel-testing-any-test-runner?view=azure-devops&viewFallbackFrom=vsts
For the record, I highlight the main:
An agent job can be used to run a suite of tests in parallel. For example, you can run a large suite of 1000 tests on a single agent. Or, you can use two agents and run 500 tests on each one in parallel. To leverage slicing, the tasks in the job should be smart enough to understand the slice they belong to. The step that runs the tests in a job needs to know which test slice should be run. The variables System.JobPositionInPhase and System.TotalJobsInPhase can be used for this purpose: System.TotalJobsInPhase indicates the total number of slices (you can think of this as "totalSlices") System.JobPositionInPhase identifies a particular slice (you can think of this as "sliceNum")
An agent job can be used to run a suite of tests in parallel. For example, you can run a large suite of 1000 tests on a single agent. Or, you can use two agents and run 500 tests on each one in parallel. To leverage slicing, the tasks in the job should be smart enough to understand the slice they belong to.
The step that runs the tests in a job needs to know which test slice should be run. The variables System.JobPositionInPhase and System.TotalJobsInPhase can be used for this purpose:
Two problems remain unsolved: - pytest doesn't understand slicing - the IPA tests haven't to be splitted just using dividing by the number of jobs (there are tests which depend on each other within test module)
I've prepared pytest plugin which solves the slicing.
master:
Metadata Update from @abbra: - Issue set to the milestone: FreeIPA 4.8.1
ipa-4-8:
Metadata Update from @abbra: - Issue close_status updated to: fixed - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.