#150 Changes to pod templates are not persistent across Jenkins pod restarts
Closed: Fixed 3 years ago by mrc0mmand. Opened 3 years ago by mrc0mmand.

Hello,

I already discussed this with @siddharthvipul1 on IRC, but filing a full report for completeness.

For some strange reason changes in the Manage Jenkins -> Manage Nodes and Clouds -> Configure Clouds -> Kubernetes -> Pod Templates don't survive the OCP Jenkins pod restart. I can confirm this in two separate ways after every pod restart:

  1. In our project (systemd) I always add a LANG=en_US.utf8 env variable to the cico-workspace pod template, but even though it gets properly saved to the configuration XML (which is stored on a PV), after pod restart it gets overwritten by the default pod template configuration

  2. After every pod restart I get this warning in the Jenkins administration:
    https://sumsal.cz/s/Screen_jen3gjTh.png (the Pagure image upload doesn't work for some reason)
    After discarding the unreadable data using the provided button the warning goes away, but after pod restart it reappears.

I have no idea what's going on. I checked the Jenkins config several times before restarting the pod and all changes I made were there, but disappeared after restart. Curiously enough, this applies only to the Pod Templates section, making changes to the Kubernetes Cloud Details section works as expected (all changes there survive pod restart).


@mrc0mmand in which project is this issue occurring?

Metadata Update from @dkirwan:
- Issue priority set to: Waiting on Reporter (was: Needs Review)
- Issue tagged with: centos-ci-infra, medium-gain, medium-trouble

3 years ago

After an investigation with @dkirwan & @siddharthvipul1 the culprit is in the Jenkins pod deploy config, where the cico-workspace template carries the whole pod templates configuration in its payload. So, in order to change the configuration under the Pod Templates section in Jenkins, the changes have to be done in the respective config map in OCP:

OCP -> Workloads -> Config Maps -> cico-workspace -> YAML

The XML blob to edit is under data -> cico-workspace.

Big thanks to both David and Vipul for figuring this out!

Metadata Update from @mrc0mmand:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

3 years ago

Login to comment on this ticket.

Metadata
Boards 1
CentOS CI Infra Status: Done