Brainstorming how a simple metadata file testing.fmf could look like:

components:
- python26
- python34
- python35
- python36
- python-virtualenv
- python-setuptools
- python-pip
releases:
- f28
- rawhide

Based on this settings CI jobs would be scheduled for listed
python components on Fedora 28 and Rawhide.

Thanks, I would say defining in a metadata file makes sense, to cover the default state. I wonder if we are already adding a new configuration file, how much sense would it take to standardize the name of the file (it actually defines the testing expected to be done from a CI system) and the content, so people in the future have only one format they need to use (thinking about the future need similar config for CI in dist-git in rpms/* space)

I'd be satisfied if I define the tests for the repo in tests namespace in a similar way we do it in the rpm namespace. No need to track down the consumers and run tests for them, simply execute our own tests.

If, however, you decide to implement this is proposed by @psss, consider if rebuilding all the consumer packages is necessary (is there away to skip the koji scratch build and use the version from the repo?).

Let's start with the basic support for STI as suggested by @churchyard, that is without inventing new config file format or detecting all relevant components. Here's what would be needed:

Commit

• New pipeline watching commits in the tests namespace
• Job for each new commit added to the master branch
• Koji build step can be skipped (nothing to build)
• Execute tests as defined by the Standard Test Interface

Pull Request

• New pipeline watching pull requests in the tests namespace
• Job should be created for each new pull request
• Koji build step can be skipped here as well
• Fetch tests + merge the new pull request content
• Execute tests as defined by the Standard Test Interface

Is that all we need? Anything missing? @bgoncalv? @mvadkert?

It looks good, I'm just not convinced we should support commit. I'd prefer any change to the test be via PRs, so we can know if the patch will break something before it gets committed.

Yeah, pull requests are more important. However I see benefit in both (you get feedback for pull requests but also for direct work on branches). But it probably makes sense to start just with pull requests and implement commit pipeline later if desired.

Yeah, I would start with pull requests. Commit testing is after a change was introduced, and while can provide a value, but is something we should be really focusing on.

OK, the consensus seems to be clear: Let's enable new pipeline for pull requests. One more thing: We will need message format for sending updates about PR testing. The dist-git-pr messages haven't been open-sourced yet. Seems it is the right time now, @mvadkert?

The test repo can have it's own tests and when a pull requests is created, the test are run.

Works for me, thank you!

NIce work @psss and @bgoncalv !

Thanks.