#52 UX problem: Tedious boilerplate when defining pipeline tests
Opened 4 years ago by churchyard. Modified 3 years ago

This is one thing that crossed my mind recently.

I need to define an ansible playbook to have tests.

The yaml already has (arguably reasonable amount of) boilerplate:

- hosts: localhost  # why do I need this?
  - role: standard-test-basic  # why is this not the default?
    - classic  # dtto

Yet in order to provision more RAM, I need to create another yaml with:

    m: 3G  # Amount of VM memory

Why can't I simply state that in the first YAML?
Why do I need to say "standard-inventory-qcow2"? Those are implementation details. What I really need to say is: "I need 3 GiB RAM."

Yet that YAML doesn't work alone, I need to install a custom tool, to run fmf init that creates another file, tests/.fmf/version - and that contains just this:


Why cannot I just put this the FMF version in that one YAML ?

And now if I want to gate, I need yet another YAML file for that.

For comparsion, please take a look at major public CI services, such as Travis CI. There is exactly 1 YAML file. It has all I need, it lets me define:

  • what to test (run)
  • what do I need (operating system, language runtime version, packages, etc...)
  • what happens when the test success/fail (notifications)

It doesn't bother me with:

  • where do I run this
  • leaky abstractions
  • custom tools installation

Yes, you are right in many points. That's why we are looking for a better solution for test metadata and ci metadata in long-term. Some of the main ideas are already described here:

For example, the memory use case you've mentioned should be very simple, as you say, without any unnecessary implementation details. Please, have a look at the Provision section here:

Does that make sense to you? We would be glad for any feedback.

I'll read that, thanks. The Provision section looks awesome.

It would be much easier to grasp if there was a complete example. I don't really understand the code snippets in there - are those yams? tree dir views? where do i put this...

You can find an initial bunch of examples at the top of the L2 page. For example workflow.fmf shows how individual steps would be configured in a single file.

The whole idea is based on fmf so basically yaml with a few enhancements such as inheritance and elasticity which prevent unnecessary duplication and minimize maintenance. Elasticity allows to store everything in a single file ci.fmf for small project as well as distribute metadata into separate files or even directories when the project grows.

Regarding the location: Single ci.fmf file could be stored directly in the dist git rpms namespace root. Another option is to have all L2 metadata files stored under a ci directory. Whatever is more suitable for particular project.

Ie just seen the gating.yaml example and that is even worse. Not only it has tedious boilerplate, but it has some custom YAML tags that make it unparsable by humans (me) and machines (PyYAML) alike.

Completely agreed. Here's the proposed alternative using fmf and L2 metadata:

        - merge-pull-request

Does it look better? See also this pull request with more detailed documentation.

It does look better indeed!

@psss @churchyard can we consider this solved and close it out?

This is not solved at all. From the Fedora CI user perspective, nothing has changed since I've open this ticket. I understand everything discussed here is "future feature". Or have I misunderstood this and I can use the new fmf L2 thing today to run package tests?

Login to comment on this ticket.