#35 Can't create privileged container on https://console-openshift-console.apps.ocp.ci.centos.org/k8s/ns/fedora-ci-jenkins-prod
Closed: Fixed 3 years ago by pingou. Opened 3 years ago by bgoncalv.

I'm trying to create a privileged container as we need to use kvm module to start a VM.

The problem is I can't create a provileged container:

Error in provisioning; agent=KubernetesSlave name: dist-git-pipeline-3-x3ljm-l45p4-l1pmd, template=PodTemplate{, name='dist-git-pipeline_3-x3ljm-l45p4', namespace='fedora-ci-jenkins-prod', label='dist-git-pipeline_3-x3ljm', nodeUsageMode=EXCLUSIVE, workspaceVolume=EmptyDirWorkspaceVolume [memory=false], annotations=[org.csanchez.jenkins.plugins.kubernetes.PodAnnotation@aab9c821, org.csanchez.jenkins.plugins.kubernetes.PodAnnotation@c92c82e4]}
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://kubernetes.default/api/v1/namespaces/fedora-ci-jenkins-prod/pods. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods "dist-git-pipeline-3-x3ljm-l45p4-l1pmd" is forbidden: unable to validate against any security context constraint: [spec.containers[0].securityContext.privileged: Invalid value: true: Privileged containers are not allowed].

Metadata Update from @dkirwan:
- Issue tagged with: centos-ci-infra

3 years ago

Metadata Update from @dkirwan:
- Issue tagged with: medium-gain, medium-trouble, need-more-info

3 years ago

Hi @bgoncalv , you don't have permissions to create a privileged container yes that is expected.

Can you provide more information about the full work flow of what you are attempting to do, and where this VM is you're trying to start?

Metadata Update from @dkirwan:
- Issue priority set to: Waiting on Reporter (was: Needs Review)

3 years ago

Metadata Update from @dkirwan:
- Issue assigned to dkirwan

3 years ago

We plan to move the current dist-git CI pipeline [1] that is running on old cluster to this new cluster.

The pipeline needs to run on privileged container, because in the container we start a VM.

Hi @bgoncalv I'm unfamiliar with the setup on the older 3.6 cluster, I had to speak with some of the others to get a better understanding of how the dist-git CI pipeline was working. It seems we tagged some of the OCP nodes for /dev/kvm access, so workloads would only run on the nodes which had access to /dev/kvm, the Jenkins service account had privileged container access to interact with kvm directly to control/run VMs.

On OCP4, the architecture is completely different, I'm not sure if you can continue to use the same method of interacting with kvm via privileged containers, but I have been given approval to provide privileged container access to the service account for Jenkins in the fedora-ci-jenkins-prod namespace.

I'll update the ticket when this work is complete, and have you retest to see if that is enough to unblock you.

Ideally though, it would be good if the dist-git CI pipeline can be modified/upgraded to interact with Kubevirt operator via the new mechanism [1]. On OCP4, we have a fully supported way to achieve this workflow via an operator which is installed on the cluster: https://github.com/kubevirt/hyperconverged-cluster-operator

We interact with this operator via VirtualMachine kubernetes API resources, which then interacts with kvm on your behalf.

Metadata Update from @dkirwan:
- Issue priority set to: Waiting on Assignee (was: Waiting on Reporter)

3 years ago

@bgoncalv that change has been made to the service account, can you retest to see if you are unblocked now please.

 oc adm policy add-scc-to-user privileged -n fedora-ci-jenkins-prod -z osci-jenkins-2
securitycontextconstraints.security.openshift.io/privileged added to: ["system:serviceaccount:fedora-ci-jenkins-prod:osci-jenkins-2"]

Metadata Update from @dkirwan:
- Issue priority set to: Waiting on Reporter (was: Waiting on Assignee)

3 years ago

Metadata Update from @dkirwan:
- Issue untagged with: need-more-info

3 years ago

Thank you, I was able to create a privileged container, now I'm blocked on https://pagure.io/centos-infra/issue/47

Great, past one hurdle, I'll close this ticket and start working on your blocker #47

Metadata Update from @dkirwan:
- Issue status updated to: Closed (was: Open)

3 years ago

Issue status updated to: Open (was: Closed)

3 years ago

Metadata Update from @dkirwan:
- Issue status updated to: Closed (was: Open)

3 years ago

Issue status updated to: Open (was: Closed)

3 years ago

Issue status updated to: Closed (was: Open)
Issue close_status updated to: Fixed

3 years ago

Login to comment on this ticket.

Metadata
Boards 1
CentOS CI Infra Status: Done