#8008 Azure Pipeline slicing
Closed: fixed 4 years ago by abbra. Opened 4 years ago by slev.

Currently, unit tests execution time within Azure Pipelines is not balanced:
Base tests take ~13 minutes, while XMLRPC tests ~28 minutes.
The grouping is:

testsToRun:
test_cmdline
test_install
test_ipaclient
test_ipalib
test_ipaplatform
test_ipapython

and

testsToRun:
test_xmlrpc

The same problem exists for Travis, where the grouping is

TESTS_TO_RUN="
test_cmdline 
test_install  
test_ipaclient 
test_ipalib      
test_ipaplatform  
test_ipapython
test_ipaserver 
test_xmlrpc/test_[l-z]*.py"

and

TESTS_TO_RUN="test_xmlrpc/test_[a-k]*.py"

task1 ~ 3 min
task2 ~ 9 min

Some sort of autobalancer is required.

Azure Pipelines supports slicing out of the box:
https://docs.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml#slicing
https://docs.microsoft.com/ru-ru/azure/devops/pipelines/test/parallel-testing-any-test-runner?view=azure-devops&viewFallbackFrom=vsts

For the record, I highlight the main:

An agent job can be used to run a suite of tests in parallel. For example, you can run a large suite of 1000 tests on a single agent. Or, you can use two agents and run 500 tests on each one in parallel. To leverage slicing, the tasks in the job should be smart enough to understand the slice they belong to.

The step that runs the tests in a job needs to know which test slice should be run. The variables System.JobPositionInPhase and System.TotalJobsInPhase can be used for this purpose:

  • System.TotalJobsInPhase indicates the total number of slices (you can think of this as "totalSlices")
  • System.JobPositionInPhase identifies a particular slice (you can think of this as "sliceNum")

Two problems remain unsolved:
- pytest doesn't understand slicing
- the IPA tests haven't to be splitted just using dividing by the number of jobs
(there are tests which depend on each other within test module)

I've prepared pytest plugin which solves the slicing.


master:

  • 2312b38 Simplify ipa-run-tests script
  • a2d4e2a Make use of Azure Pipeline slicing
  • c750022 Avoid use of '/tmp' for pip operations

Metadata Update from @abbra:
- Issue set to the milestone: FreeIPA 4.8.1

4 years ago

ipa-4-8:

  • 8aa3ef0 Simplify ipa-run-tests script
  • 5780d6a Make use of Azure Pipeline slicing
  • 0ec908b Avoid use of '/tmp' for pip operations

Metadata Update from @abbra:
- Issue close_status updated to: fixed
- Issue status updated to: Closed (was: Open)

4 years ago

Login to comment on this ticket.

Metadata