standard-test-roles

Clone
Source Code
GIT

#219 Add fmf to standard-inventory-qcow2 provisioner.
Merged 5 years ago by astepano. Opened 5 years ago by astepano.
qcow2_fmf  into  master

file modified
+59 -2 
@@ -4,6 +4,7 @@ 
 

  import errno
 

  import json
 

  import os
 

+ import fmf
 

  import shutil
 

  import shlex
 

  import signal
 
@@ -142,6 +143,59 @@ 
 

      return inventory
 

  

  

+ class FmfMetadataTree(object):
 

+     """This is aux class to hold one copy FMF tree. fmf.Tree(path) could be
 

+     very resource consuming when walking through big project with many
 

+     directories.
 

+ 

+     """
 

+     tree = None
 

+     """fmf.Tree() object."""
 

+     path = None
 

+     """Metadata tree is created for this path."""
 

+     def get(self, path="."):
 

+         if self.path != path or self.tree is None:
 

+             FmfMetadataTree.path = path
 

+             FmfMetadataTree.tree = fmf.Tree(path)
 

+         return self.tree
 

+ 

+ 

+ def fmf_get(path, default=None):
 

+     """Return parameter from FMF at desired path or default.
 

+ 

+     Parameters
 

+     ----------
 

+     path: dict
 

+         List of strings. Strings form a path for looking parameter.
 

+     default: str
 

+         Function ignores this parameter.
 

+ 

+     Returns
 

+     -------
 

+     str
 

+         Found parameter in FMF or `default`.
 

+ 

+     """
 

+     tree = FmfMetadataTree().get()
 

+     path.insert(0, 'standard-inventory-qcow2')
 

+     value = default
 

+     for provision in tree.prune(names=[".*/provision$"]):
 

+         value = provision.data
 

+         for node in path:
 

+             try:
 

+                 value = value[node]
 

+             except (KeyError, TypeError):
 

+                 value = default
 

+                 break
 

+     try:
 

+         diagnose = distutils.util.strtobool(os.getenv("TEST_DEBUG", "0"))
 

+     except ValueError:
 

+         diagnose = 0
 

+     if diagnose:
 

+         sys.stderr.write("DIAGNOSE: fmf_get() for {}: {}\n".format(str(path), str(value)))
 

+     return value
 

+ 

+ 

  def start_qemu(image, cloudinit, artifacts, portrange=(2222, 5555)):
 

      for _ in range(10):
 

          port = random.randint(*portrange)
 
@@ -166,12 +220,15 @@ 
 

      log_guest = os.path.join(artifacts, "{0}.guest.log".format(os.path.basename(image)))
 

      # Log from qemu itself.
 

      log_qemu = log_guest.replace(".guest.log", ".qemu.log")
 

+     # Parameters from FMF:
 

+     param_m = fmf_get(['qemu', 'm'], 1024)
 

+     param_net_nic_model = fmf_get(['qemu', 'net_nic', 'model'], 'virtio')
 

      # Use -cpu host and -smp by default.
 

      # virtio-rng-pci: https://wiki.qemu.org/Features/VirtIORNG
 

      qemu_cmd = ["/usr/bin/qemu-system-x86_64",
 

                  "-cpu", "host", "-smp", get_qemu_smp_arg(),
 

-                 "-m", "1024", image, "-enable-kvm", "-snapshot", "-cdrom", cloudinit,
 

-                 "-net", "nic,model=virtio", "-net", "user,hostfwd=tcp:127.0.0.3:{0}-:22".format(port),
 

+                 "-m", param_m, image, "-enable-kvm", "-snapshot", "-cdrom", cloudinit,
 

+                 "-net", "nic,model=%s" % param_net_nic_model, "-net", "user,hostfwd=tcp:127.0.0.3:{0}-:22".format(port),
 

                  "-device", "virtio-rng-pci", "-rtc", "base=utc",
 

                  "-device", "isa-serial,chardev=pts2", "-chardev", "file,id=pts2,path=" + log_guest,
 

                  "-display", "none"]

Signed-off-by: Andrei Stepanov astepano@redhat.com

I would probably suggest to use something like this:

config = fmf.Tree('.').find("/provision").data['standard-inventory-qcow2']

Then you can easily access all configuration values directly:

config['qemu']['net_nic']['model']

Hope this helps.

@psss 

cat provision.fmf 
---

# qemu options: https://qemu.weilnetz.de/doc/qemu-doc.html

standard-inventory-qcow2:
  qemu:
    # RAM size in megabytes. Optionally, a suffix of “M” or “G”.
    m: 3G
    net_nic:
      # Use qemu-system-x86_64 -net nic,model=help for a list of available devices.
      model: e1000

standard-inventory-docker:
  dumb_option: dumb_parameter

# vim:ft=yaml: ts=2 sts=2 sw=2 expandtab

Of course we can use : config['qemu']['net_nic']['model']
With above approach we must always run in try/catch + specify backup code to set default value.

If we want to use function, as I did, that we need represent ['qemu']['net_nic']['model'] as a parameter to function fmf_get(). How to do this ? I use list.

Think about fmf_get() as the same as dict.get(key[, default]).

@psss I have a question:

fmf.Tree('.') - does it scans all work dir each time?

If I want to get many parameters, how is to safe to call fmf.Tree('.') each time?

Thank you for your time.
Edited 5 years ago by astepano

rebased onto 016263398d388930688c2a625eae8dc13417fd98
5 years ago

I see. Thanks for clarification. Make sense to handle default values in the functin. Regarding the scanning: Yes, when construction a new tree the directory is scanned again. It might make sense to implement something like this:

class Config(object):
    tree = None
    def get(self):
        if self.tree is None:
            Config.tree = fmf.Tree(".")
        return self.tree

If this is more common use case we could also consider support for simple caching directly in FMF. BTW, the extended .get() functionality might be an interesting future feature as well.

@psss I was trying to use .find(), but it is not easy as it would seem:

for node in tree.climb(): print node.name
empty/3/output-document-file
empty/3/output-document-pipe
empty/provision

works:

tree.find("empty/provision")
Out[25]: <fmf.base.Tree at 0x248dcd0>

but doesn't work:

tree.find("/provision")

Nothing

rebased onto fc31b6a2cdd3b3cfdaf075491be4b3f9b22b9e45
5 years ago

Yes, Tree.find() expects a full name of the node to be found. If you need to search among multiple nodes you can use Tree.prune(names=[".*/provision"]) which supports regular expressions. The FmfMetadataTree class looks fine.

rebased onto 7e98dc554011a820f5830b96faa0147235f2fad6
5 years ago

@psss Thank you for the Tree.prune() hint. I updated PR.

pretty please pagure-ci rebuild

rebased onto dad29c5e36a81291e221aa19aecbcce98b1590eb
5 years ago

pretty please pagure-ci rebuild

rebased onto bf3a444451eb1a9c6f73d6c913af45adb34e5884
5 years ago

pretty please pagure-ci rebuild

rebased onto 43d5594
5 years ago

pretty please pagure-ci rebuild

Commit 9e216f6 fixes this pull-request

Pull-Request has been merged by astepano
5 years ago

Pull-Request has been merged by astepano
5 years ago

FYI, as this seems to be quite useful feature (and possible common use case) I've implemented the deep dictionary attribute retrieval directly in fmf:

https://github.com/psss/fmf/commit/b537c59

@psss nice, when it is in stable RPM branch we can update this code.
Thank you.
Edited 5 years ago by astepano

It would be nice if you could provide some example how to use this.

I've added basic documentation including a simple example here:
https://fedoraproject.org/wiki/CI/Metadata#Provision
Changes Summary 1
+59 -2
file changed
inventory/standard-inventory-qcow2
Powered by Pagure 5.13.3
Documentation • File an Issue • About • SSH Hostkey/Fingerprint
© Red Hat, Inc. and others.