#402 add support for TEST_HOSTALIASES
Merged 3 years ago by astepano. Opened 3 years ago by rmeggins.
rmeggins/standard-test-roles hostalias-arg  into  master

file modified
+53
@@ -154,6 +154,59 @@ 

  TEST_EXTRA_SSH_ARGS="-o PubkeyAcceptedKeyTypes=+ssh-rsa" TEST_SUBJECTS=/path/to/el6.qcow2 ...

  ```

  

+ ## HOSTALIAS for standard-inventory-qcow2

+ 

+ By default, `standard-inventory-qcow2` will use the full path and filename of

+ the image.qcow2 file as the host key in the inventory.  If you would rather

+ provide a different name for the host that corresponds to the file, you can use

+ either the `--hostalias` cli flag or the `TEST_HOSTALIASES` environment

+ variable.  You can also use the special keyword `BASENAME` if you want the

+ hostalias of the image to use the basename of the image.  For example:

+ ```

+ TEST_SUBJECTS="/path/to/fedora-34.qcow2 /path/to/f-33.qcow2" TEST_HOSTALIASES="sut1 BASENAME" ....

+ ```

+ will result in 2 inventories that look like this:

+ ```yaml

+ all:

+   children:

+     localhost:

+       hosts: &id001

+         sut1:

+           ansible_host: 127.0.0.3

+ ...

+ all:

+   children:

+     localhost:

+       hosts: &id001

+         f-33.qcow2:

+           ansible_host: 127.0.0.3

+ ...

+ ```

+ If you want all of the hostaliases for all images to be the basename, use `--use-basename`

+ or `TEST_USE_BASENAME=true`.  NOTE: If you do this, make sure that the basenames of all of

+ the images are unique.

+ ```

+ TEST_SUBJECTS="/path/to/fedora-34.qcow2 /path/to/f-33.qcow2" TEST_USE_BASENAME=true ....

+ ```

+ will result in 2 inventories that look like this:

+ ```yaml

+ all:

+   children:

+     localhost:

+       hosts: &id001

+         fedora-34.qcow2:

+           ansible_host: 127.0.0.3

+ ...

+ all:

+   children:

+     localhost:

+       hosts: &id001

+         f-33.qcow2:

+           ansible_host: 127.0.0.3

+ ...

+ ```

+ If the number of hostaliases does not match the number of subjects, you will get

+ an error.

  

  [1]: https://fedoraproject.org/wiki/CI/Metadata

  [2]: http://fmf.readthedocs.io/

@@ -209,11 +209,17 @@ 

  def inv_list(opts):

      hosts = []

      variables = {}

-     for subject in opts.subjects:

-         host_vars = inv_host(subject, opts)

+     for ii, subject in enumerate(opts.subjects):

+         if len(opts.hostalias) > ii:

+             hostalias = opts.hostalias[ii]

+         else:

+             hostalias = None

+         hostalias = image_to_alias(subject, hostalias, opts.use_basename)

+         logger.info(f"image {subject} alias {hostalias}")

+         host_vars = inv_host(opts, subject, hostalias)

          if host_vars:

-             hosts.append(subject)

-             variables[subject] = host_vars

+             hosts.append(hostalias)

+             variables[hostalias] = host_vars

      if not hosts:

          return EMPTY_INVENTORY

      return {"localhost": {"hosts": hosts, "vars": {}},
@@ -447,7 +453,17 @@ 

      return qemu_proc, port, log_guest

  

  

- def inv_host(image, opts):

+ def image_to_alias(image, hostalias, use_basename):

+     if not hostalias and not use_basename:

+         return image

+     if isinstance(hostalias, list) and len(hostalias) > 0:

+         hostalias = hostalias[0]

+     if hostalias == "BASENAME" or use_basename:

+         return os.path.basename(image)

+     return hostalias

+ 

+ 

+ def inv_host(opts, image, hostalias):

      if not image.endswith((".qcow2", ".qcow2c")):

          logger.info("Return empty inventory for image: %s.", image)

          return EMPTY_INVENTORY
@@ -492,6 +508,7 @@ 

      ssh_args = "-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no"

      if opts.extra_ssh_args:

          ssh_args += " " + opts.extra_ssh_args

+     hostalias = image_to_alias(image, hostalias, opts.use_basename)

      for _ in range(0, 600):

          try:

              # The variables
@@ -505,7 +522,7 @@ 

              }

              # Write out a handy inventory file, for our use and for debugging

              inventory = os.path.join(directory, "inventory")

-             write_debug_inventory(inventory, {image: variables})

+             write_debug_inventory(inventory, {hostalias: variables})

              # Wait for ssh to come up

              ping = [

                  ansible_bin,
@@ -567,7 +584,7 @@ 

              return None

          variables["ansible_python_interpreter"] = ansible_python_interpreter

          # Update inventory file

-         write_debug_inventory(inventory, {image: variables})

+         write_debug_inventory(inventory, {hostalias: variables})

          return variables

      # Daemonize and watch the processes

      os.chdir("/")
@@ -618,7 +635,26 @@ 

      sys.exit(0)

  

  

- def main(argv):

+ def help_hostalias():

+     return """By default, this script will add each subject to the inventory, which is

+     typically the full path and filename of the disk image to use.  If you

+     cannot use a full path in the inventory, you can use the --hostalias to

+     specify an alias to use for the image in the inventory.  The position of

+     each --hostalias you specify on the command line must correspond exactly to

+     the position of the subject you specify on the command line or in the

+     TEST_SUBJECTS env var.  For example, if you specify

+     TEST_SUBJECTS="/path/to/a.qcow2 /path/to/b.qcow2" ... --hostalias a --hostalias b

+     Then the inventory would use "a" as the alias for /path/to/a.qcow2, and

+     b as the alias for /path/to/b.qcow2.  If you want to use the basename of the file

+     as the hostalias, specify the special value 'BASENAME'.  For example, if you have

+     TEST_SUBJECTS="/path/to/a.qcow2 /path/to/b.qcow2" ... --hostalias BASENAME --hostalias b

+     Then the inventory will use "a.qcow2" as the alias for path /path/to/a.qcow2, and will

+     use "b" as the alias for path /path/to/b.qcow2.

+     If you want all of the subjects to use the basename of the file, use option

+     --use-basename, then you do not have to use --hostalias for every subject.

+     """

+ 

+ def main():

      global logger

      global diagnose

      logger = logging.getLogger(__name__)
@@ -641,9 +677,12 @@ 

      conhandler.setFormatter(formatter)

      logger.addHandler(conhandler)

      parser = argparse.ArgumentParser(description="Inventory for a QCow2 test image")

+     # list is currently unused

      parser.add_argument("--list", action="store_true", help="Verbose output")

-     parser.add_argument('--host', help="Get host variables")

      parser.add_argument('--extra-ssh-args', default=os.environ.get("TEST_EXTRA_SSH_ARGS"), help="Extra arguments to pass to SSH")

+     parser.add_argument('--host', help="Full path to qcow2 disk image")

+     parser.add_argument("--use-basename", default=bool(distutils.util.strtobool(os.environ.get("TEST_USE_BASENAME", "False"))), action="store_true", help=help_hostalias())

+     parser.add_argument("--hostalias", default=shlex.split(os.environ.get("TEST_HOSTALIASES", "")), action="append", help=help_hostalias())

      parser.add_argument("subjects", nargs="*", default=shlex.split(os.environ.get("TEST_SUBJECTS", "")))

      opts = parser.parse_args()

      # Send logs to common logfile for all default provisioners.
@@ -657,6 +696,9 @@ 

      fhandler.setFormatter(logFormatter)

      logger.addHandler(fhandler)

      logger.info("Start provisioner.")

+     if opts.use_basename and opts.hostalias:

+         logger.critical("Use only one of --use-basename or --hostalias")

+         sys.exit(1)

      ansibles = ['ansible', 'ansible-3', None]

      global ansible_bin

      ansible_bin = functools.reduce(which, ansibles)
@@ -664,8 +706,16 @@ 

          raise Exception("Fail to find ansible.")

      logger.info("Path to ansible: %s", ansible_bin)

      if opts.host:

-         data = inv_host(opts.host, opts)

+         data = inv_host(opts, opts.host, opts.hostalias)

      else:

+         if (

+             opts.hostalias

+             and len(opts.hostalias) != len(opts.subjects)

+         ):

+             logger.critical(

+                 "If you are using hostalias, you must specify a --hostalias for each subject."

+             )

+             sys.exit(1)

          data = inv_list(opts)

      # Dump Ansible inventory.

      sys.stdout.write(json.dumps(data, indent=4, separators=(',', ': ')))
@@ -674,7 +724,7 @@ 

  if __name__ == '__main__':

      ret = -1

      try:

-         main(sys.argv)

+         main()

          ret = 0

      except RuntimeError as ex:

          print_bad_inventory()

By default, standard-inventory-qcow2 will use the full path and filename of
the image.qcow2 file as the host key in the inventory. If you would rather
provide a different name for the host that corresponds to the file, you can use
either the --hostalias cli flag or the TEST_HOSTALIASES environment
variable. If you want to use the basename of the path as the alias for all images,
use the --use-basename cli flag.

What happens if I set TEST_HOSTALIASES="sut1 BASENAME"? :) And if I have 5 TEST_SUBJECTS with TEST_HOSTALIASES="sut1 BASENAME", just the first one has "sut1" and the other 4 would have its own test_subject file name?

Never mind. BASENAME has to be a string and one item. I'm happy with it.

I am not sure about this, it looks like the same host-alias under index 0 can be used a few times.

1 new commit added

  • add --use-basename parameter
3 years ago

I am not sure about this, it looks like the same host-alias under index 0 can be used a few times.

ok - in order to make this clearer and simpler
hostaliases must be exactly the same length as subjects
there is a new --use-basename if you want to use the basename for all subjects/images

Ack to merge, looks reasonable.
Please write comment when the PR is ready to be merged.

@nhosoi ok?

Yes, it's clearer now. Thanks, @rmeggins!
lgtm

the PR is ready to be merged

Hello, sorry, I merged https://pagure.io/standard-test-roles/pull-request/403
Now this PR cannot be merged due conflicts.
Please fix.

Question: do you want to have new build of standard-test-roles with these merged PR?

rebased onto c4d7972

3 years ago

Hello, sorry, I merged https://pagure.io/standard-test-roles/pull-request/403
Now this PR cannot be merged due conflicts.
Please fix.

Done.

Question: do you want to have new build of standard-test-roles with these merged PR?

When are you planning to do another build? What is your typical build/release cadence? We don't need this in a build/release urgently - we can continue to grab the file from pagure for the short term.

@rmeggins which environment are you looking primarly? Fedora CI, RHEL CI, BaseOS CI? Something else?

Commit b885eb0 fixes this pull-request

Pull-Request has been merged by astepano

3 years ago

Pull-Request has been merged by astepano

3 years ago

@rmeggins which environment are you looking primarly? Fedora CI, RHEL CI, BaseOS CI? Something else?

The environment is the current downstream testing using qcow2 images, and upstream developer use.

This feature is primarily to support ansible-navigator, which will eventually replace ansible and ansible-playbook command line tools. ansible-navigator currently creates directories for logs, caches, etc. which have the hostname in the path name. This doesn't work if the hostname is the absolute path to a qcow2 image file. So we need this feature in order to use an alias/basename instead of the absolute path.