I'm trying to create a privileged container as we need to use kvm module to start a VM.
The problem is I can't create a provileged container:
Error in provisioning; agent=KubernetesSlave name: dist-git-pipeline-3-x3ljm-l45p4-l1pmd, template=PodTemplate{, name='dist-git-pipeline_3-x3ljm-l45p4', namespace='fedora-ci-jenkins-prod', label='dist-git-pipeline_3-x3ljm', nodeUsageMode=EXCLUSIVE, workspaceVolume=EmptyDirWorkspaceVolume [memory=false], annotations=[org.csanchez.jenkins.plugins.kubernetes.PodAnnotation@aab9c821, org.csanchez.jenkins.plugins.kubernetes.PodAnnotation@c92c82e4]} io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://kubernetes.default/api/v1/namespaces/fedora-ci-jenkins-prod/pods. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods "dist-git-pipeline-3-x3ljm-l45p4-l1pmd" is forbidden: unable to validate against any security context constraint: [spec.containers[0].securityContext.privileged: Invalid value: true: Privileged containers are not allowed].
Metadata Update from @dkirwan: - Issue tagged with: centos-ci-infra
Metadata Update from @dkirwan: - Issue tagged with: medium-gain, medium-trouble, need-more-info
Hi @bgoncalv , you don't have permissions to create a privileged container yes that is expected.
Can you provide more information about the full work flow of what you are attempting to do, and where this VM is you're trying to start?
Metadata Update from @dkirwan: - Issue priority set to: Waiting on Reporter (was: Needs Review)
Metadata Update from @dkirwan: - Issue assigned to dkirwan
We plan to move the current dist-git CI pipeline [1] that is running on old cluster to this new cluster.
The pipeline needs to run on privileged container, because in the container we start a VM.
Hi @bgoncalv I'm unfamiliar with the setup on the older 3.6 cluster, I had to speak with some of the others to get a better understanding of how the dist-git CI pipeline was working. It seems we tagged some of the OCP nodes for /dev/kvm access, so workloads would only run on the nodes which had access to /dev/kvm, the Jenkins service account had privileged container access to interact with kvm directly to control/run VMs.
On OCP4, the architecture is completely different, I'm not sure if you can continue to use the same method of interacting with kvm via privileged containers, but I have been given approval to provide privileged container access to the service account for Jenkins in the fedora-ci-jenkins-prod namespace.
fedora-ci-jenkins-prod
I'll update the ticket when this work is complete, and have you retest to see if that is enough to unblock you.
Ideally though, it would be good if the dist-git CI pipeline can be modified/upgraded to interact with Kubevirt operator via the new mechanism [1]. On OCP4, we have a fully supported way to achieve this workflow via an operator which is installed on the cluster: https://github.com/kubevirt/hyperconverged-cluster-operator
We interact with this operator via VirtualMachine kubernetes API resources, which then interacts with kvm on your behalf.
VirtualMachine
Metadata Update from @dkirwan: - Issue priority set to: Waiting on Assignee (was: Waiting on Reporter)
@bgoncalv that change has been made to the service account, can you retest to see if you are unblocked now please.
oc adm policy add-scc-to-user privileged -n fedora-ci-jenkins-prod -z osci-jenkins-2 securitycontextconstraints.security.openshift.io/privileged added to: ["system:serviceaccount:fedora-ci-jenkins-prod:osci-jenkins-2"]
Metadata Update from @dkirwan: - Issue priority set to: Waiting on Reporter (was: Waiting on Assignee)
Metadata Update from @dkirwan: - Issue untagged with: need-more-info
Thank you, I was able to create a privileged container, now I'm blocked on https://pagure.io/centos-infra/issue/47
Great, past one hurdle, I'll close this ticket and start working on your blocker #47
Metadata Update from @dkirwan: - Issue status updated to: Closed (was: Open)
Issue status updated to: Open (was: Closed)
Issue status updated to: Closed (was: Open) Issue close_status updated to: Fixed
Login to comment on this ticket.