greenwave

Clone

#280 Jenkins: run unit tests and archive coverage report
Merged 5 years ago by lholecek. Opened 5 years ago by csomh.
csomh/greenwave jenkins-enable-cov  into  master

file modified
+3 -2 
@@ -4,10 +4,11 @@ 
 

      greenwave/*
 

  

  [report]
 

- fail_under = 55
 

+ fail_under = 50
 

  exclude_lines =
 

      pragma: no cover
 

      if __name__ == .__main__.:
 

  omit =
 

      greenwave/tests/*
 

- 

+     greenwave/logger.py
 

+     greenwave/wsgi.py

file modified
+26 -2 
@@ -14,8 +14,11 @@ 
 

  timestamps {
 

  node('fedora-27') {
 

      checkout scm
 

-     sh 'sudo dnf -y builddep greenwave.spec'
 

-     sh 'sudo dnf -y install python3-flake8 python3-pylint python3-sphinx python3-sphinxcontrib-httpdomain'
 

+     sh '''
 

+     sudo dnf -y builddep greenwave.spec
 

+     sudo dnf -y install python3-flake8 python3-pylint python3-sphinx \
 

+         python3-sphinxcontrib-httpdomain python3-pytest-cov
 

+     '''
 

      /* Needed to get the latest /etc/mock/fedora-28-x86_64.cfg */
 

      sh 'sudo dnf -y update mock-core-configs'
 

      stage('Invoke Flake8') {
 
@@ -24,6 +27,27 @@ 
 

      stage('Invoke Pylint') {
 

          sh 'pylint-3 --reports=n greenwave'
 

      }
 

+     stage('Run unit tests') {

This can be merged to stage 'Perform functional tests', the coverage would be much better.

csomh commented 5 years ago

Hm... this would work until the functional-tests are run like today; that is, greenwave is not deployed as an app in the test env but it's running on the slave.

An another reason, I would not do this, is that the functional tests should aim to cover functionality and treat the application as a black box, while unit tests should be the ones aiming for code coverage. Hope this makes sense :)

 
+     // Yes, this is also done while building the RPM, but we need coverage
 

+         sh '''
 

+         rm -rf htmlcov coverage.xml
 

+         pytest-3 greenwave/tests/ \
 

+             --cov-config .coveragerc --cov=greenwave \
 

+             --cov-report term --cov-report xml --cov-report html
 

+         '''
 

+         archiveArtifacts artifacts: 'htmlcov/**,coverage.xml'
 

+         step([
 

+             $class: 'CoberturaPublisher',
 

+             autoUpdateHealth: false,
 

+             autoUpdateStability: false,
 

+             coberturaReportFile: 'coverage.xml',
 

+             failUnhealthy: false,
 

+             failUnstable: false,
 

+             maxNumberOfBuilds: 0,
 

+             onlyStable: false,
 

+             zoomCoverageChart: false
 

+         ])

Cool, look like this works.

csomh commented 5 years ago

Yep. Let's keep the HTML report, too, if you don't mind. The Cobertura report does not show a line-per-line breakdown (as coverage.xml does not include the lines).

Let's keep the HTML report

Yes, definitely (I didn't notice the HTML being used before).

 
+     }
 

      stage('Build Docs') {
 

          sh 'DEV=true GREENWAVE_CONFIG=$(pwd)/conf/settings.py.example make -C docs html'
 

          archiveArtifacts artifacts: 'docs/_build/html/**'

Enable coverage reporting during PR testing.

This makes the Jenkins fail... But beside that.. +1

Still working on this...

Realized that @ralph was right in lowering the fail_under value to something that can be reached.

I've excluded a few boilerplate files from coverage measurements. As soon as pagure takes the push, Jenkins should pass with a 53% coverage.

1 new commit added

  • Exclude boilerplate code from coverage measurement
5 years ago

1 new commit added

  • Try cobertura reporting
5 years ago

3 new commits added

  • Try cobertura reporting
  • Exclude boilerplate code from coverage measurement
  • Jenkins: run unit tests and archive coverage report
5 years ago

This file shouldn't be ignored, it could contain some important code in future.

3 new commits added

  • Try cobertura reporting
  • Exclude boilerplate code from coverage measurement
  • Jenkins: run unit tests and archive coverage report
5 years ago

Currently it only has __version__, which is more like metadata.

The idea was to omit all the files for which currently there is no point expecting some unit tests to be implemented. That is, I wouldn't file an issue saying 'write tests for init.py' right now.

In case these files are changed in the future, so that unit testing makes sense for them, they could be removed from the omit list.

rebased onto db6ef6a66235917c641eab6b1f28b8d5d1a801ac
5 years ago

I agree, we can add it again in the future.

Any reason to have coverage printed in log and HTML generated?

@lholecek the HTML report is archived and offers the possibility to see results for individual files.

I was trying to use coverage.xml with the Cobertura plugin from Jenkins, to have nicer reporting, but no luck with that yet.

I'll give some more tries, though. If I manage to make it work, and offers the same details as the HTML format, I'll remove the later.

The coverage for this file is currently 100%. Ignoring it now won't increase coverage.

1 new commit added

  • Try cobertura again
5 years ago

My bad, you are absolutely right @lholecek.

I've amended the commit to remove it from the omit list. Thanks for noticing it!

3 new commits added

  • Try cobertura again
  • Exclude boilerplate code from coverage measurement
  • Jenkins: run unit tests and archive coverage report
5 years ago

Cool, look like this works.

Yep. Let's keep the HTML report, too, if you don't mind. The Cobertura report does not show a line-per-line breakdown (as coverage.xml does not include the lines).

This can be merged to stage 'Perform functional tests', the coverage would be much better.

rebased onto 62c9ddc
5 years ago

Let's keep the HTML report

Yes, definitely (I didn't notice the HTML being used before).

Hm... this would work until the functional-tests are run like today; that is, greenwave is not deployed as an app in the test env but it's running on the slave.

An another reason, I would not do this, is that the functional tests should aim to cover functionality and treat the application as a black box, while unit tests should be the ones aiming for code coverage. Hope this makes sense :)

the functional tests should aim to cover functionality and treat the application as a black box, while unit tests should be the ones aiming for code coverage

OK, makes sense.

+1

Pull-Request has been merged by lholecek
5 years ago
Metadata
None
No Tags
Flags
Jenkins
success (100%)
Build passed.
5 years ago
Changes Summary 2
+3 -2
file changed
.coveragerc
+26 -2
file changed
Jenkinsfile
Powered by Pagure 5.13.3
Documentation • File an Issue • About • SSH Hostkey/Fingerprint
© Red Hat, Inc. and others.