PR#511: [WIP] jenkins job definition added - rpkg

rpkg

#511 [WIP] jenkins job definition added

Opened 3 years ago by lrossett. Modified 3 years ago

lrossett/rpkg jenkinsfile2 into master

jenkins job definition added

lrossett • 3 years ago

6a2b1de

deploy/jenkins/README.md

file added

+27

		`@@ -0,0 +1,27 @@`
		`+ # jenkins job builder`
		`+`
		`+ ## Setup`
		`+`
		+ Setup your credentials in `jenkins-job.ini` file, with the proper username (userid) and password (api token).
		`+`
		`+ ## Plugin List`
		`+`
		`+ The following command retrieves the list of installed plugins:`
		`+`
		+ ```sh
		`+ jenkins-jobs --conf jenkins-job.ini get-plugins-info -o plugins_info.yaml`
		+ ```
		`+`
		`+ ## Testing Definitions`
		`+`
		`+ The following command will test all job definitions without updating jenkins:`
		`+`
		+ ```sh
		`+ jenkins-jobs --conf jenkins-job.ini test ./defs`
		+ ```
		`+`
		`+ ## Updating Jobs`
		`+`
		+ ```sh
		`+ jenkins-jobs --conf jenkins-job.ini update ./defs`
		+ ```
		`\ No newline at end of file`

deploy/jenkins/defs/jobs/tests/test-rpkg/Jenkinsfile

file added

+63

		`@@ -0,0 +1,63 @@`
		`+ #!groovy`
		`+`
		`+ def fedora_versions = params.FEDORA_VERSIONS.tokenize(',')`
		`+ def fedora_image = params.FEDORA_IMAGE`
		`+ def jnlp_image = params.JENKINS_JNLP_IMAGE`
		`+ def stages = [:]`
		`+`
		`+ def fedora_containers = [`
		`+ containerTemplate(name: 'jnlp',`
		`+ image: "${jnlp_image}",`
		`+ alwaysPullImage: true,`
		`+ ttyEnabled: false,`
		`+ args: '${computer.jnlpmac} ${computer.name}',`
		`+ workingDir: "/workdir"`
		`+ )`
		`+ ]`
		`+`
		`+ fedora_versions.each { version ->`
		`+ //add a containver per fedora version`
		`+ fedora_containers.add(containerTemplate`
		`+ (`
		`+ name: "${version}",`
		`+ image: "${fedora_image}:${version}",`
		`+ ttyEnabled: true,`
		`+ alwaysPullImage: true,`
		`+ command: "/uid_entrypoint sleep infinity",`
		`+ envVars: [`
		`+ envVar(key: 'HOME', value: '/tmp')`
		`+ ]`
		`+ )`
		`+ )`
		`+ //add a stage for each fedora version`
		`+ stages["tox-py3-${version}"] = {`
		`+ container("${version}") {`
		`+ withEnv(['HOME=/workdir/workspace']) {`
		`+ sh "cp -al ./ ../${version}/"`
		`+ dir( "../${version}" ) {`
		`+ ansiColor('xterm') {`
		`+ sh 'rm -rf .tox'`
		`+ sh 'tox -e flake8,py36,py37,py38,py39 --parallel=auto'`
		`+ }`
		`+ }`
		`+ }`
		`+ }`
		`+ }`
		`+ }`
		`+`
		`+ podTemplate(`
		`+ name: 'fedora-tox',`
		`+ label: 'fedora-tox',`
		`+ cloud: 'openshift',`
		`+ containers: fedora_containers`
		`+ ){`
		`+ node('fedora-tox') {`
		`+ stage('checkout') {`
		`+ ansiColor('xterm'){`
		`+ checkout scm`
		`+ }`
		`+ }`
		`+`
		`+ parallel stages`
		`+ }`
		`+ }`

deploy/jenkins/defs/jobs/tests/test-rpkg/test-rpkg.yaml

file added

+26

		`@@ -0,0 +1,26 @@`
		`+ - job:`
		`+ name: test-rpkg`
		`+ project-type: pipeline`
		`+ display-name: test-rpkg`
		`+ pipeline-scm:`
		`+ scm:`
		`+ - git:`
		`+ url: https://pagure.io/rpkg.git`
		`+ branches:`
		`+ - master`
		`+ clean:`
		`+ after: true`
		`+ script-path: deploy/jenkins/defs/jobs/tests/test-rpkg/Jenkinsfile`
		`+ parameters:`
		`+ - string:`
		`+ name: FEDORA_VERSIONS`
		`+ description: fedora versions to test against (image tags)`
		`+ default: f31,f32,latest`
		`+ - string:`
		`+ name: JENKINS_JNLP_IMAGE`
		`+ description: jenkins jnlp image to use`
		`+ default: jenkins/jnlp-slave:4.0.1-1`
		`+ - string:`
		`+ name: FEDORA_IMAGE`
		`+ description: fedora image url to use (no tag)`
		`+ default: quay.io/centosci/python-tox`
		`\ No newline at end of file`

deploy/jenkins/defs/views.yaml

file added

+12

		`@@ -0,0 +1,12 @@`
		`+ - view:`
		`+ name: Tests`
		`+ view-type: list`
		`+ job-name:`
		`+ - test-rpkg`
		`+ columns:`
		`+ - status`
		`+ - weather`
		`+ - job`
		`+ - last-success`
		`+ - last-failure`
		`+ - last-duration`
		`\ No newline at end of file`

deploy/jenkins/jenkins-job.ini

file added

+14

		`@@ -0,0 +1,14 @@`
		`+ [job_builder]`
		`+ ignore_cache=True`
		`+ keep_descriptions=False`
		`+ include_path=defs`
		`+ recursive=True`
		`+ exclude=.*:manual:./development`
		`+ allow_duplicates=False`
		`+ update=all`
		`+`
		`+ [jenkins]`
		`+ user=fillme`
		`+ password=fillme`
		`+ url=fillme`
		`+ query_plugins_info=False`

deploy/jenkins/plugins_info.yaml

file added

+366

		`@@ -0,0 +1,366 @@`
		`+ - longName: 'Pipeline: REST API Plugin'`
		`+ shortName: pipeline-rest-api`
		`+ version: '2.10'`
		`+ - longName: 'Pipeline: Declarative'`
		`+ shortName: pipeline-model-definition`
		`+ version: 1.6.0`
		`+ - longName: 'JavaScript GUI Lib: ACE Editor bundle plugin'`
		`+ shortName: ace-editor`
		`+ version: 1.0.1`
		`+ - longName: SSH Agent Plugin`
		`+ shortName: ssh-agent`
		`+ version: '1.20'`
		`+ - longName: Pipeline Graph Analysis Plugin`
		`+ shortName: pipeline-graph-analysis`
		`+ version: '1.7'`
		`+ - longName: Pipeline Utility Steps`
		`+ shortName: pipeline-utility-steps`
		`+ version: 2.5.0`
		`+ - longName: 'Pipeline: Groovy'`
		`+ shortName: workflow-cps`
		`+ version: '2.80'`
		`+ - longName: 'Pipeline: Milestone Step'`
		`+ shortName: pipeline-milestone-step`
		`+ version: 1.3.1`
		`+ - longName: REST API for Blue Ocean`
		`+ shortName: blueocean-rest`
		`+ version: 1.10.2`
		`+ - longName: MapDB API Plugin`
		`+ shortName: mapdb-api`
		`+ version: 1.0.9.0`
		`+ - longName: Trilead API Plugin`
		`+ shortName: trilead-api`
		`+ version: 1.0.5`
		`+ - longName: Handy Uri Templates 2.x API Plugin`
		`+ shortName: handy-uri-templates-2-api`
		`+ version: 2.1.6-1.0`
		`+ - longName: OAuth Credentials plugin`
		`+ shortName: oauth-credentials`
		`+ version: '0.3'`
		`+ - longName: AnsiColor`
		`+ shortName: ansicolor`
		`+ version: 0.7.2`
		`+ - longName: Jenkins Parameterized Trigger plugin`
		`+ shortName: parameterized-trigger`
		`+ version: 2.35.2`
		`+ - longName: Windows Slaves Plugin`
		`+ shortName: windows-slaves`
		`+ version: '1.0'`
		`+ - longName: LDAP Plugin`
		`+ shortName: ldap`
		`+ version: '1.11'`
		`+ - longName: Web for Blue Ocean`
		`+ shortName: blueocean-web`
		`+ version: 1.10.2`
		`+ - longName: Jenkins Subversion Plug-in`
		`+ shortName: subversion`
		`+ version: 2.13.1`
		`+ - longName: Config File Provider Plugin`
		`+ shortName: config-file-provider`
		`+ version: '3.5'`
		`+ - longName: Jenkins GIT server Plugin`
		`+ shortName: git-server`
		`+ version: '1.7'`
		`+ - longName: 'Pipeline: Basic Steps'`
		`+ shortName: workflow-basic-steps`
		`+ version: '2.18'`
		`+ - longName: Blue Ocean Core JS`
		`+ shortName: blueocean-core-js`
		`+ version: 1.10.2`
		`+ - longName: Kubernetes Client API Plugin`
		`+ shortName: kubernetes-client-api`
		`+ version: 4.8.0-1`
		`+ - longName: OpenShift Login Plugin`
		`+ shortName: openshift-login`
		`+ version: 1.0.23`
		`+ - longName: 'Pipeline: Model API'`
		`+ shortName: pipeline-model-api`
		`+ version: 1.6.0`
		`+ - longName: Authentication Tokens API Plugin`
		`+ shortName: authentication-tokens`
		`+ version: '1.3'`
		`+ - longName: 'Pipeline: Stage Tags Metadata'`
		`+ shortName: pipeline-stage-tags-metadata`
		`+ version: 1.6.0`
		`+ - longName: Pipeline`
		`+ shortName: workflow-aggregator`
		`+ version: '2.6'`
		`+ - longName: Script Security Plugin`
		`+ shortName: script-security`
		`+ version: '1.71'`
		`+ - longName: Blue Ocean`
		`+ shortName: blueocean`
		`+ version: 1.10.2`
		`+ - longName: 'Pipeline: Build Step'`
		`+ shortName: pipeline-build-step`
		`+ version: '2.12'`
		`+ - longName: GitHub plugin`
		`+ shortName: github`
		`+ version: 1.29.2`
		`+ - longName: Display URL API`
		`+ shortName: display-url-api`
		`+ version: 2.3.1`
		`+ - longName: 'Pipeline: Nodes and Processes'`
		`+ shortName: workflow-durable-task-step`
		`+ version: '2.31'`
		`+ - longName: Durable Task Plugin`
		`+ shortName: durable-task`
		`+ version: '1.33'`
		`+ - longName: Bitbucket Branch Source Plugin`
		`+ shortName: cloudbees-bitbucket-branch-source`
		`+ version: 2.2.15`
		`+ - longName: 'Pipeline: Job'`
		`+ shortName: workflow-job`
		`+ version: '2.33'`
		`+ - longName: Jenkins JIRA plugin`
		`+ shortName: jira`
		`+ version: 3.0.5`
		`+ - longName: Docker Pipeline`
		`+ shortName: docker-workflow`
		`+ version: '1.22'`
		`+ - longName: Docker Commons Plugin`
		`+ shortName: docker-commons`
		`+ version: '1.16'`
		`+ - longName: Jenkins Pipeline Remote Loader Plugin`
		`+ shortName: workflow-remote-loader`
		`+ version: '1.5'`
		`+ - longName: 'Pipeline: Supporting APIs'`
		`+ shortName: workflow-support`
		`+ version: '3.3'`
		`+ - longName: Events API for Blue Ocean`
		`+ shortName: blueocean-events`
		`+ version: 1.10.2`
		`+ - longName: 'Pipeline: Stage View Plugin'`
		`+ shortName: pipeline-stage-view`
		`+ version: '2.10'`
		`+ - longName: Git Pipeline for Blue Ocean`
		`+ shortName: blueocean-git-pipeline`
		`+ version: 1.10.2`
		`+ - longName: External Monitor Job Type Plugin`
		`+ shortName: external-monitor-job`
		`+ version: '1.4'`
		`+ - longName: JDK Tool Plugin`
		`+ shortName: jdk-tool`
		`+ version: '1.0'`
		`+ - longName: GitHub API Plugin`
		`+ shortName: github-api`
		`+ version: '1.90'`
		`+ - longName: SCM API Plugin`
		`+ shortName: scm-api`
		`+ version: 2.6.3`
		`+ - longName: Jenkins Git plugin`
		`+ shortName: git`
		`+ version: 4.2.2`
		`+ - longName: Plain Credentials Plugin`
		`+ shortName: plain-credentials`
		`+ version: '1.6'`
		`+ - longName: Google OAuth Credentials plugin`
		`+ shortName: google-oauth-plugin`
		`+ version: 1.0.0`
		`+ - longName: Matrix Project Plugin`
		`+ shortName: matrix-project`
		`+ version: '1.14'`
		`+ - longName: GitHub Branch Source Plugin`
		`+ shortName: github-branch-source`
		`+ version: 2.3.6`
		`+ - longName: Autofavorite for Blue Ocean`
		`+ shortName: blueocean-autofavorite`
		`+ version: 1.2.2`
		`+ - longName: GitHub Pipeline for Blue Ocean`
		`+ shortName: blueocean-github-pipeline`
		`+ version: 1.10.2`
		`+ - longName: Jenkins Pub-Sub "light" Bus`
		`+ shortName: pubsub-light`
		`+ version: '1.12'`
		`+ - longName: Credentials Plugin`
		`+ shortName: credentials`
		`+ version: 2.3.5`
		`+ - longName: bouncycastle API Plugin`
		`+ shortName: bouncycastle-api`
		`+ version: '2.17'`
		`+ - longName: Javadoc Plugin`
		`+ shortName: javadoc`
		`+ version: '1.1'`
		`+ - longName: Bitbucket Pipeline for Blue Ocean`
		`+ shortName: blueocean-bitbucket-pipeline`
		`+ version: 1.10.2`
		`+ - longName: Display URL for Blue Ocean`
		`+ shortName: blueocean-display-url`
		`+ version: 2.2.0`
		`+ - longName: REST Implementation for Blue Ocean`
		`+ shortName: blueocean-rest-impl`
		`+ version: 1.10.2`
		`+ - longName: 'JavaScript GUI Lib: Handlebars bundle plugin'`
		`+ shortName: handlebars`
		`+ version: '1.1'`
		`+ - longName: 'Pipeline: Input Step'`
		`+ shortName: pipeline-input-step`
		`+ version: '2.11'`
		`+ - longName: 'Pipeline: Declarative Extension Points API'`
		`+ shortName: pipeline-model-extensions`
		`+ version: 1.6.0`
		`+ - longName: Config API for Blue Ocean`
		`+ shortName: blueocean-config`
		`+ version: 1.10.2`
		`+ - longName: 'Pipeline: GitHub Groovy Libraries'`
		`+ shortName: pipeline-github-lib`
		`+ version: '1.0'`
		`+ - longName: Folders Plugin`
		`+ shortName: cloudbees-folder`
		`+ version: 6.10.1`
		`+ - longName: Server Sent Events (SSE) Gateway Plugin`
		`+ shortName: sse-gateway`
		`+ version: '1.16'`
		`+ - longName: Matrix Authorization Strategy Plugin`
		`+ shortName: matrix-auth`
		`+ version: '2.2'`
		`+ - longName: Blue Ocean Pipeline Editor`
		`+ shortName: blueocean-pipeline-editor`
		`+ version: 1.10.2`
		`+ - longName: 'Pipeline: Multibranch'`
		`+ shortName: workflow-multibranch`
		`+ version: '2.20'`
		`+ - longName: Command Agent Launcher Plugin`
		`+ shortName: command-launcher`
		`+ version: '1.2'`
		`+ - longName: conditional-buildstep`
		`+ shortName: conditional-buildstep`
		`+ version: 1.3.1`
		`+ - longName: Favorite`
		`+ shortName: favorite`
		`+ version: 2.3.1`
		`+ - longName: Variant Plugin`
		`+ shortName: variant`
		`+ version: '1.3'`
		`+ - longName: Jackson 2 API Plugin`
		`+ shortName: jackson2-api`
		`+ version: 2.10.2`
		`+ - longName: Jenkins Git client plugin`
		`+ shortName: git-client`
		`+ version: 3.2.1`
		`+ - longName: 'Pipeline: SCM Step'`
		`+ shortName: workflow-scm-step`
		`+ version: '2.9'`
		`+ - longName: 'Pipeline: API'`
		`+ shortName: workflow-api`
		`+ version: '2.40'`
		`+ - longName: GitHub Organization Folder Plugin`
		`+ shortName: github-organization-folder`
		`+ version: '1.6'`
		`+ - longName: Token Macro Plugin`
		`+ shortName: token-macro`
		`+ version: '2.12'`
		`+ - longName: 'Pipeline: Step API'`
		`+ shortName: workflow-step-api`
		`+ version: '2.22'`
		`+ - longName: Job DSL`
		`+ shortName: job-dsl`
		`+ version: '1.72'`
		`+ - longName: Jenkins JSch dependency plugin`
		`+ shortName: jsch`
		`+ version: 0.1.55.1`
		`+ - longName: HTML Publisher plugin`
		`+ shortName: htmlpublisher`
		`+ version: '1.21'`
		`+ - longName: Jenkins Mailer Plugin`
		`+ shortName: mailer`
		`+ version: '1.30'`
		`+ - longName: Run Condition Plugin`
		`+ shortName: run-condition`
		`+ version: '0.10'`
		`+ - longName: SSH Credentials Plugin`
		`+ shortName: ssh-credentials`
		`+ version: 1.18.1`
		`+ - longName: 'JavaScript GUI Lib: jQuery bundles (jQuery and jQuery UI) plugin'`
		`+ shortName: jquery-detached`
		`+ version: 1.2.1`
		`+ - longName: Lockable Resources plugin`
		`+ shortName: lockable-resources`
		`+ version: '2.5'`
		`+ - longName: Ant Plugin`
		`+ shortName: ant`
		`+ version: '1.10'`
		`+ - longName: Metrics Plugin`
		`+ shortName: metrics`
		`+ version: 4.0.2.5`
		`+ - longName: Branch API Plugin`
		`+ shortName: branch-api`
		`+ version: 2.0.20`
		`+ - longName: Pipeline SCM API for Blue Ocean`
		`+ shortName: blueocean-pipeline-scm-api`
		`+ version: 1.10.2`
		`+ - longName: PAM Authentication plugin`
		`+ shortName: pam-auth`
		`+ version: '1.6'`
		`+ - longName: Prometheus metrics plugin`
		`+ shortName: prometheus`
		`+ version: 2.0.0`
		`+ - longName: Kubernetes Credentials Plugin`
		`+ shortName: kubernetes-credentials`
		`+ version: 0.6.2`
		`+ - longName: JAXB plugin`
		`+ shortName: jaxb`
		`+ version: 2.3.0`
		`+ - longName: JIRA Integration for Blue Ocean`
		`+ shortName: blueocean-jira`
		`+ version: 1.10.2`
		`+ - longName: Structs Plugin`
		`+ shortName: structs`
		`+ version: '1.20'`
		`+ - longName: i18n for Blue Ocean`
		`+ shortName: blueocean-i18n`
		`+ version: 1.10.2`
		`+ - longName: Dashboard for Blue Ocean`
		`+ shortName: blueocean-dashboard`
		`+ version: 1.10.2`
		`+ - longName: JWT for Blue Ocean`
		`+ shortName: blueocean-jwt`
		`+ version: 1.10.2`
		`+ - longName: Jenkins Design Language`
		`+ shortName: jenkins-design-language`
		`+ version: 1.10.2`
		`+ - longName: 'Pipeline: Shared Groovy Libraries'`
		`+ shortName: workflow-cps-global-lib`
		`+ version: '2.15'`
		`+ - longName: Pipeline implementation for Blue Ocean`
		`+ shortName: blueocean-pipeline-api-impl`
		`+ version: 1.10.2`
		`+ - longName: Jenkins Mercurial plugin`
		`+ shortName: mercurial`
		`+ version: '2.3'`
		`+ - longName: OpenShift Sync`
		`+ shortName: openshift-sync`
		`+ version: 1.0.45`
		`+ - longName: Jenkins Apache HttpComponents Client 4.x API Plugin`
		`+ shortName: apache-httpcomponents-client-4-api`
		`+ version: 4.5.10-2.0`
		`+ - longName: Personalization for Blue Ocean`
		`+ shortName: blueocean-personalization`
		`+ version: 1.10.2`
		`+ - longName: Credentials Binding Plugin`
		`+ shortName: credentials-binding`
		`+ version: '1.19'`
		`+ - longName: Common API for Blue Ocean`
		`+ shortName: blueocean-commons`
		`+ version: 1.10.2`
		`+ - longName: OpenShift Client Jenkins Plugin`
		`+ shortName: openshift-client`
		`+ version: 1.0.32`
		`+ - longName: OWASP Markup Formatter Plugin`
		`+ shortName: antisamy-markup-formatter`
		`+ version: '1.1'`
		`+ - longName: 'Pipeline: Stage Step'`
		`+ shortName: pipeline-stage-step`
		`+ version: '2.3'`
		`+ - longName: JUnit Plugin`
		`+ shortName: junit`
		`+ version: 1.26.1`
		`+ - longName: Kubernetes plugin`
		`+ shortName: kubernetes`
		`+ version: 1.25.2`
		`+ - longName: 'JavaScript GUI Lib: Moment.js bundle plugin'`
		`+ shortName: momentjs`
		`+ version: '1.1'`
		`+ - longName: 'Pipeline: Declarative Agent API'`
		`+ shortName: pipeline-model-declarative-agent`
		`+ version: 1.1.1`

tox.ini

file modified

		`@@ -9,6 +9,8 @@`
		`py27: {env:TOXPYTHON:python2.7}`
		`py36: {env:TOXPYTHON:python3.6}`
		`py37: {env:TOXPYTHON:python3.7}`
		`+ py38: {env:TOXPYTHON:python3.8}`
		`+ py39: {env:TOXPYTHON:python3.9}`
		`flake8: {env:TOXPYTHON:python3.7}`
		`flake8python2: {env:TOXPYTHON:python2.7}`

lrossett commented 3 years ago

Signed-off-by: lrossett lrossett@redhat.com

Background

The current job is manually defined in the Jenkins UI which can make it a bit tricky to keep it up to date.

Proposal

Adds jobs definitions using the jenkins-job-builder CLI tool;
Job definition stays in a local Jenkinsfile which gives the dev team more control on what needs to be done in those tests

Status

The reasons for it to be flagged as WIP are:

acceptance of the idea/proprosal;
fedora images needed some changes to make it work, and those still need to be merged (only rawhide got merged): https://github.com/centosci/images/pull/9 and https://github.com/centosci/images/pull/8;
epel7 for python 2.7 testing is failing to run tox (I am working on it);
either a jenkins config change or move to a new Jenkins instance (there is one new jenkins instance already provisioned which was used during development that could be used for prod);