PR#1380: Run parts of the unit tests Jenkins job in parallel - fm-orchestrator

fm-orchestrator

#1380 Run parts of the unit tests Jenkins job in parallel

Merged 4 years ago by csomh. Opened 4 years ago by mprahl.

parallel-unit-tests into master

Make the timestamps consistent in import_mmd

mprahl • 4 years ago

a7099d8

Run the unit tests in parallel in the Jenkins job

mprahl • 4 years ago

f7654f7

Pull the container images in parallel in the Jenkins job

mprahl • 4 years ago

10d36d3

.cico-pr.pipeline

file modified

+19 -5

		`@@ -59,15 +59,29 @@`
		`}`

		`stage('Pull Container Images') {`
		`- onmyduffynode 'docker pull quay.io/factory2/mbs-test-centos'`
		`- onmyduffynode 'docker pull quay.io/factory2/mbs-test-fedora'`
		`+ parallel {`
		`+ stage('Pull quay.io/factory2/mbs-test-centos') {`
		`+ onmyduffynode 'docker pull quay.io/factory2/mbs-test-centos'`
		`+ }`
		`+ stage('Pull quay.io/factory2/mbs-test-fedora') {`
		`+ onmyduffynode 'docker pull quay.io/factory2/mbs-test-fedora'`
		`+ }`
		`+ }`
		`}`

		`stage('Run Test Suites') {`
		`timeout(30) {`
		`- onmyduffynode '~/fm-orchestrator/contrib/run-unittests.sh --no-tty'`
		`- onmyduffynode '~/fm-orchestrator/contrib/run-unittests.sh --py3 --no-tty'`
		`- onmyduffynode '~/fm-orchestrator/contrib/run-unittests.sh --py3 --with-pgsql --no-tty'`
		`+ parallel {`
		`+ stage('Test with Python 2 & SQLite') {`
		`+ onmyduffynode '~/fm-orchestrator/contrib/run-unittests.sh --no-tty'`
		`+ }`
		`+ stage('Test with Python 3 & SQLite') {`
		`+ onmyduffynode '~/fm-orchestrator/contrib/run-unittests.sh --py3 --no-tty'`
		`+ }`
		`+ stage('Test with Python 3 & Postgresql') {`
		`+ onmyduffynode '~/fm-orchestrator/contrib/run-unittests.sh --py3 --with-pgsql --no-tty'`
		`+ }`
		`+ }`
cqi commented 4 years ago Just an idea, is it possible to request more resources in order to run these kinds of tests separately in a dedicated machine?
		`}`
		`}`

module_build_service/utils/general.py

file modified

+4 -3

		`@@ -525,9 +525,10 @@`
		`build.context = context`
		`build.owner = "mbs_import"`
		`build.rebuild_strategy = "all"`
		`- build.time_submitted = datetime.utcnow()`
		`- build.time_modified = datetime.utcnow()`
		`- build.time_completed = datetime.utcnow()`
		`+ now = datetime.utcnow()`
		`+ build.time_submitted = now`
		`+ build.time_modified = now`
		`+ build.time_completed = now`
		`if build.name in conf.base_module_names:`
		`build.stream_version = models.ModuleBuild.get_stream_version(stream)`

no initial comment

1 new commit added

Make the timestamps consistent in import_mmd

4 years ago

mprahl commented 4 years ago

@csomh could you please take a look? I don't know how to actually test the Jenkins job change, but I'd like to know if you think this will work.

This is to take advantage of the change in #1378.

mikeb commented 4 years ago

Build #345 failed (commit: e92ae7dc7f4da274b18266c86a5819730c6c2494).
Rebase or make new commits to rebuild.

3 new commits added

Make the timestamps consistent in import_mmd
Run the unit tests in parallel in the Jenkins job
Pull the container images in parallel in the Jenkins job

4 years ago

cqi commented on line 32 of .cico-pr.pipeline 4 years ago

Just an idea, is it possible to request more resources in order to run these kinds of tests separately in a dedicated machine?

cqi commented 4 years ago

:thumbsup:

jkaluza commented 4 years ago

Not sure if @csomh needs to review this before I can merge. I will wait a bit :).

csomh commented 4 years ago

@mprahl I see that the last job that run on this failed, but it was too long ago and the logs are gone. Can you please do a force push in order to be able to see what was wrong?

Just by reading this, it should work.

rebased onto 10d36d3

4 years ago

mprahl commented 4 years ago

@csomh I just rebased and pushed. I don't think the Jenkinsfile from this PR is being used though. I think CentOS CI uses the one from the master branch. So I'm not sure if there's a way to test this without merging.

csomh commented 4 years ago