#36 merge-standard-inventory script to run and merge output from the default standard inventory scripts
Merged 6 years ago by stefw. Opened 6 years ago by merlinm.
Unknown source merge-standard-inventory  into  master

Updates to address review feedback.
Merlin Mathesius • 6 years ago  
Add scripts directory and README file.
Merlin Mathesius • 6 years ago  
file added
+30
@@ -0,0 +1,30 @@

+ # Standard Test Interface Scripts

+ 

+ ## `merge-standard-inventory`

+ 

+ `merge-standard-inventory` is a wrapper script that runs _all_ of the

+ available Standard Testing Interface dynamic inventory scripts from

+ `/usr/share/ansible/inventory` and merges their output. This allows

+ an individual test to provide a single customized dynamic inventory

+ script that sets up certain environment variables before running all

+ the default standard-inventory scripts. To make use of it, simply

+ have `ansible` use the custom inventory script (which makes use of

+ the `merge-standard-inventory` wrapper) in place of the

+ `/usr/share/ansible/inventory` directory.

+ 

+ The following short example custom inventory script sets the

+ `TEST_DOCKER_EXTRA_ARGS` environment variable before running

+ `merge-standard-inventory` with any command line arguments provided

+ by ansible. By doing this, when `merge-standard-inventory` runs the

+ `standard-inventory-docker` script it will launch the docker

+ container with `CAP_SYS_ADMIN` capabilities (if the `TEST_SUBJECTS`

+ environment variable triggers a container to be launched). Other

+ `standard-inventory-*` scripts will simply ignore the

+ `TEST_DOCKER_EXTRA_ARGS` environment variable--but may have their own

+ environment variables set in the inventory script.

+ 

+ ```

+ #!/bin/bash

+ export TEST_DOCKER_EXTRA_ARGS="--cap-add=SYS_ADMIN"

+ exec merge-standard-inventory "$@"

+ ```

@@ -0,0 +1,201 @@

+ #!/usr/bin/python2

+ 

+ import json

+ import os

+ import sys

+ import subprocess

+ import time

+ 

+ 

+ def main(argv):

+     """

+     Run all standard inventory scripts and return their merged output.

+ 

+     Note the standard inventory scripts clean up their spawned hosts

+     when they detect their parent processes go away. To accomodate

+     that behavior, this script forks a child process to run the

+     inventory scripts, send back the merged inventory, and then wait

+     for the parent of this script (ansible) to die before silently

+     exiting. In the mean time, this script outputs the merged

+     inventory gathered by the child and then exits.

+     """

+ 

+     # Keep track of our parent

+     waitpid = os.getppid()

+ 

+     tty = err_to_tty()

+     pipein, pipeout = os.pipe()

+ 

+     childpid = os.fork()

+     if childpid == 0:

+         # this is the child process

+ 

+         # close the inherited input side of the pipe

+         os.close(pipein)

+ 

+         # run and merge output from standard inventory scripts

+         merged_data = merge_standard_inventories(argv[1:])

+ 

+         # send merged data to parent via output pipe

+         os.write(pipeout, merged_data)

+ 

+         # close the pipe so the parent knows we are done

+         os.close(pipeout)

+ 

+         # wait for the grandparent process to exit

+         linger(waitpid, tty)

+ 

+         # exit cleanly

+         sys.exit(0)

+ 

+     # this is the parent process

+ 

+     # close the inherited output side of the pipe

+     os.close(pipeout)

+ 

+     # send eveything from the child to stdout

+     while True:

+         data = os.read(pipein, 999)

+         if not data:

+             os.close(pipein)

+             break

+         sys.stdout.write(data)

+ 

+     return 0

+ 

+ 

+ def merge_standard_inventories(args):

+ 

+     inventory_dir = os.environ.get(

+         "TEST_DYNAMIC_INVENTORY_DIRECTORY", "/usr/share/ansible/inventory")

+     inventory_ignore_extensions = (

+         "~", ".orig", ".bak", ".ini", ".cfg", ".retry", ".pyc", ".pyo")

+ 

+     merged = Inventory()

+ 

+     for i in os.listdir(inventory_dir):

+         ipath = os.path.join(inventory_dir, i)

+         if not i.startswith("standard-inventory-"):

+             continue

+         if i.endswith(inventory_ignore_extensions):

+             continue

+         if not os.access(ipath, os.X_OK):

+             continue

+ 

+         cmd = [ipath] + args

+ 

+         try:

+             inv_out = subprocess.check_output(cmd, stdin=None, close_fds=True)

+         except subprocess.CalledProcessError as ex:

+             raise RuntimeError("Could not run: {0}".format(str(cmd)))

+ 

+         merged.merge(inv_out)

+ 

+     return merged.dumps()

+ 

+ 

+ def err_to_tty():

+     try:

+         tty = os.open("/dev/tty", os.O_WRONLY)

+         os.dup2(tty, 2)

+     except OSError:

+         tty = None

+ 

+     return tty

+ 

+ 

+ def linger(waitpid, tty=None):

+     # Go into daemon mode and watch the process

+ 

+     null = open(os.devnull, 'w')

+ 

+     os.chdir("/")

+     os.setsid()

+     os.umask(0)

+ 

+     if tty is None:

+         tty = null.fileno()

+ 

+     # Duplicate standard input to standard output and standard error.

+     os.dup2(null.fileno(), 0)

+     os.dup2(tty, 1)

+     os.dup2(tty, 2)

+ 

+     # Now wait for the watched process to go away, then return

+     while True:

+         time.sleep(3)

+ 

+         try:

+             os.kill(waitpid, 0)

+         except OSError:

+             break  # The process no longer exists

+ 

+     return

+ 

+ 

+ class Inventory:

+     """

+     Merge JSON data from standard test dynamic inventory scripts.

+ 

+     Note: This class is very specific to the JSON data written by the

+     ansible dynamic inventory scripts that are provided by the

+     standard-test-roles package. In particular, it insists on finding

+     and generating "subjects" and "localhost" members.

+     """

+ 

+     def __init__(self):

+         self.hosts = []

+         self.variables = {}

+ 

+     def merge_files(self, files):

+         for f in files:

+             with open(f) as ifile:

+                 s = ifile.read()

+                 self.merge(s)

+ 

+     def merge(self, s):

+         # parse provided string as JSON

+         inventory = json.loads(s)

+ 

+         if not isinstance(inventory, dict):

+             raise ValueError(

+                 "inventory JSON does not contain the expected top level dictionary")

+ 

+         if "subjects" not in inventory or not isinstance(inventory["subjects"], dict):

+             raise ValueError(

+                 "inventory JSON does not contain the expected [subjects] dictionary")

+         if "hosts" not in inventory["subjects"] or not isinstance(inventory["subjects"]["hosts"], list):

+             raise ValueError(

+                 "inventory JSON does not contain the expected [subjects][hosts] list")

+ 

+         for h in inventory["subjects"]["hosts"]:

+             self.hosts.append(h)

+ 

+         if "_meta" not in inventory or not isinstance(inventory["_meta"], dict):

+             raise ValueError(

+                 "inventory JSON does not contain the expected [_meta] dictionary")

+         if "hostvars" not in inventory["_meta"] or not isinstance(inventory["_meta"]["hostvars"], dict):

+             raise ValueError(

+                 "inventory JSON does not contain the expected [_meta][hostvars] dict")

+ 

+         for h in inventory["_meta"]["hostvars"]:

+             if not isinstance(inventory["_meta"]["hostvars"][h], dict):

+                 raise ValueError(

+                     "inventory JSON does not contain the expected [_meta][hostvars][{0}] dict".format(h))

+ 

+             self.variables[h] = inventory["_meta"]["hostvars"][h]

+ 

+         if "localhost" not in inventory or not isinstance(inventory["localhost"], dict):

+             raise ValueError(

+                 "inventory JSON does not contain the expected [localhost] dictionary")

+         if "hosts" not in inventory["localhost"] or not isinstance(inventory["localhost"]["hosts"], list):

+             raise ValueError(

+                 "inventory JSON does not contain the expected [localhost][hosts] list")

+ 

+     def dumps(self):

+         data = {"subjects": {"hosts": self.hosts, "vars": {}}, "localhost": {

+             "hosts": self.hosts, "vars": {}}, "_meta": {"hostvars": self.variables}}

+         return json.dumps(data, indent=4, separators=(',', ': '))

+ 

+ if __name__ == '__main__':

+     sys.exit(main(sys.argv))

The plan is to install the new script to /usr/bin/merge-standard-inventory so it appears in the default $PATH.

See the README.md file added by this PR for a full description of the script and how to use it.

I would suggest wrapping this line at a column width of 70 characters so it is readable in every text editor. Ditto for the line below.

Again, I would suggest wrapping this at around 70 characters so it is readable in a text editor.

argparse is not used by this script

The intent with Ansible is to treat everything that's marked +x executable as an inventory script in this directory. Do you think we could have the same behavior here? If so, we could remove the ignore extensions stuff above and simplify this.

The intent of the tty stuff here is that any stderr output of the above scripts (inventory scripts, etc) goes to the tty (when present). Otherwise Ansible hides this stuff. So I think this opening of the tty stuff and duping to fd 2 needs to happen before forking. Or am I confused here?

This makes this script only work with the standard roles inventory. That's fine. Maybe worth a comment here that "subjects" and "localhost" are expected from standard-test-roles dynamic inventory scripts.

Inventory scripts are expected to take a --list and --host argument. Should we take those arguments and pass them through to the scripts that we invoke?

The intent with Ansible is to treat everything that's marked +x executable as an inventory script in this directory. Do you think we could have the same behavior here? If so, we could remove the ignore extensions stuff above and simplify this.

Adding an executable check on the scripts is a good idea. But the inventory_ignore_extensions stuff is borrowed straight from ansible (http://docs.ansible.com/ansible/latest/intro_dynamic_inventory.html#using-inventory-directories-and-multiple-inventory-sources) and I really think it should stay.

Inventory scripts are expected to take a --list and --host argument. Should we take those arguments and pass them through to the scripts that we invoke?

Indeed. It's already doing that. It simply passes along the exact arguments it is given to each of the scripts it runs.

The intent of the tty stuff here is that any stderr output of the above scripts (inventory scripts, etc) goes to the tty (when present). Otherwise Ansible hides this stuff. So I think this opening of the tty stuff and duping to fd 2 needs to happen before forking. Or am I confused here?

Hmmm. That might explain some oddness I experienced. I'll look into that.

1 new commit added

  • Updates to address review feedback.
6 years ago

Tested by changing the sed repo like so:

diff --git a/inventory b/inventory
index aeb9a12..b118a5a 100755
--- a/inventory
+++ b/inventory
@@ -1,3 +1,3 @@
 #!/bin/bash
 export TEST_DOCKER_EXTRA_ARGS="--privileged"
-exec ./merge-standard-inventory "$@"
+exec merge-standard-inventory "$@"
diff --git a/merge-standard-inventory b/merge-standard-inventory
deleted file mode 100755
index 3b460dc..0000000
--- a/merge-standard-inventory
+++ /dev/null
...

And then:

# cd /path/to/upstreamfirst/sed
# export TEST_SUBJECTS=docker:docker.io/library/fedora:26
# export ANSIBLE_INVENTORY=$(test -e inventory && echo inventory || echo /usr/share/ansible/inventory)
# echo $ANSIBLE_INVENTORY
inventory
# TEST_DEBUG=1 ansible-playbook -t container tests.yml
...
# docker exec -ti gifted_wing /bin/bash
[root@199c25900bbc /]# ls /dev/
autofs           mapper              stdin   tty36  tty9    uhid
bsg              mcelog              stdout  tty37  ttyS0   uinput
btrfs-control    mei0                tty     tty38  ttyS1   urandom
...

And:

# cd /path/to/upstreamfirst/sed
# export TEST_SUBJECTS=../atomic.qcow2
# TEST_DEBUG=1 ansible-playbook -t atomic tests.yml
# echo $?
0

Pull-Request has been merged by stefw

6 years ago