#165 Add SAR script support for Openshift apps
Merged 3 years ago by pingou. Opened 3 years ago by zlopez.
Unknown source 7924  into  master

@@ -14,3 +14,16 @@

  #

  bodhi_version: 5.4.1

  bodhi_openshift_pods: 2

+ 

+ # GDPR SAR related dictionary

+ sar_openshift:

+   # Name of the app

+   release-monitoring:

+     # Location of the script

+     sar_script: /usr/local/bin/sar.py

+     # Output file on local machine

+     sar_output_file: anitya.json

+     # Openshift namespace where the app runs

+     openshift_namespace: release-monitoring

+     # Name of openshift pod - will be used for label search

+     openshift_pod: release-monitoring-web

@@ -44,6 +44,12 @@

        delegate_to: localhost

        when: "sar_huge is defined and sar_huge"

  

+ - name: Gather SAR data - openshift apps

+   hosts: os_masters[0]

+   tasks:

+     - include_tasks: sar_openshift.yml

+       loop: "{{ lookup('dict', sar_openshift, wantlist=True) }}"

+ 

  - name: Create the archive

    hosts: localhost

    tasks:

@@ -0,0 +1,15 @@

+ # Retrieve GDPR data from single openshift app.

+ # Get the correct pod

+ - name: Retrieve the id of the running container/pod

+   shell: "oc get -n {{ item.value.openshift_namespace }} -o name -l app={{ item.value.openshift_pod }} pods | cut -f 2 -d '/'"

+   register: pod_id

+ # Run the SAR script

+ - name: Run oc exec in container to extract the data

+   command: "oc -n {{ item.value.openshift_namespace }} rsh {{ pod_id.stdout }} bash -c 'SAR_USERNAME={{ sar_fas_user }} SAR_EMAIL={{ sar_email }} {{ item.value.sar_script }}'"

+   register: sar_data

+ # Store the result on disk

+ - name: Copy the output data locally

+   copy:

+     content: "{{ sar_data['stdout'] }}"

+     dest: "{{ hostvars['localhost']['sar_tmp_dir'] }}/{{ item.value.sar_output_file }}"

+   delegate_to: localhost

you could run this from one of the os-masters host so you don't need to login.
something like hosts:os-masters[0]

hum actually this is here that we want to have hosts:os-masters[0] and then you don't need the login task

you can just use oc -n {{ openshift_project }} get/exec ... instead of doing this step here.

hum actually this is here that we want to have hosts:os-masters[0] and then you don't need the login task

If I use hosts: os-masters[0], how can I use the sar_openshift inventory?

you can just use oc -n {{ openshift_project }} get/exec ... instead of doing this step here.

Yeah, I could remove this step.

hum actually this is here that we want to have hosts:os-masters[0] and then you don't need the login task

If I use hosts: os-masters[0], how can I use the sar_openshift inventory?

I don't think you need that inventory.

In this case I need to know where to find variables for the various openshift apps, that need to be checked out.

You can put your variables in the os-masters inventory for example we use that for the bodhi version --> https://pagure.io/fedora-infra/ansible/blob/master/f/inventory/group_vars/os_masters#_15

Thanks @cverna I will look at it :-)

rebased onto d24a05561c5a4b8240a56036a3872fdec1bdb1fe

3 years ago

I rewrote how the ansible playbook works, but I have issue with testing.

Is this the correct way to test the changes:

1) ssh batcave
2) git clone <repo>
3) run the local changes

Because locally I don't have access to most of the machines, which is probably correct.

rebased onto 037313f981eddd130a04c045749364372d1e4dd8

3 years ago

rebased onto 037313f981eddd130a04c045749364372d1e4dd8

3 years ago

rebased onto c6658d4f6d278283d5e42fd92c10c1c8da7268b4

3 years ago

rebased onto c6658d4f6d278283d5e42fd92c10c1c8da7268b4

3 years ago

The ansible-review error is caused by bad parsing of the command, not sure what I can do with it.

rebased onto e14c6b259e17dcb0762a1250e153804086e6386c

3 years ago

I rewrote how the ansible playbook works, but I have issue with testing.
Is this the correct way to test the changes:
1) ssh batcave
2) git clone <repo>
3) run the local changes
Because locally I don't have access to most of the machines, which is probably correct.

I tried to test it today using this method and it looks like I don't have permission to execute it.

rebased onto da3c663f88ac8859f77f9446437878c12da001b2

3 years ago

rebased onto da3c663f88ac8859f77f9446437878c12da001b2

3 years ago

Trying to run this part of the playbook, I got:

fatal: [os-master01.iad2.fedoraproject.org]: FAILED! => {"msg": "An unhandled exception occurred while running the lookup plugin 'dict'. Error was a <class 'ansible.errors.AnsibleError'>, original message: with_dict expects a dict"}

This is strange, I created the sar_openshift dict the same way as in https://docs.ansible.com/ansible/latest/plugins/lookup/dict.html

rebased onto 7ad775f963ea41f1bd6a6e148978698b8925c499

3 years ago

I updated how the values are read from the dictionary, could you try it again?

I'm also not sure if include could be used together with lookup.

Why not naming the tasks directly instead of documenting them here?

turns out it is in /usr/local/bin/sar.py

Proposal:

 - name: Include the SAR data from the different openshift apps
   include_tasks: sar_openshift.yml
   loop: "{{ lookup('dict', sar_openshift, wantlist=True) }}"

Proposal for the entire file:

# Retrieve GDPR data from single openshift app.
# Get the correct pod
- name: Retrieve the id of the running container/pod
  shell: "oc get -n {{ item.value.openshift_namespace }} -o name -l app={{ item.value.openshift_pod }} pods | cut -f 2 -d '/'"
  register: pod_id
# Run the SAR script
- name: Run ox exec in the container with the script specified in the inventory to extract the data
  command: "oc -n {{item.value.openshift_namespace }} rsh {{ pod_id.stdout }} bash -c 'SAR_USERNAME={{ sar_fas_user }} SAR_EMAIL={{ sar_email }} {{ item.value.sar_script }}'"
  register: sar_data
# Store the result on disk
- name: Copy the output data locally
  copy:
    content: "{{ sar_data['stdout'] }}"
    dest: "{{ hostvars['localhost']['sar_tmp_dir'] }}/{{ item.value.sar_output_file }}"
  delegate_to: localhost

rebased onto 872842a70fec1115abecc4bc77b4d86bb11156ad

3 years ago

Metadata Update from @pingou:
- Request assigned

3 years ago

That's not the line I had in my proposal, did you change this in purpose?

I missed this one :-D

rebased onto f962097f14165e33400ff3f300788fc3fe7dc346

3 years ago

rebased onto 04863be

3 years ago

rebased onto 04863be

3 years ago

So this still fails but it fails within the container because of something about my account. So from the ansible point of view, this works!

Thanks for your work @zlopez !!

Pull-Request has been merged by pingou

3 years ago