#149 Confirm (or specify) acceptance criteria for modules in Fedora
Opened 2 months ago by lruzicka. Modified 2 months ago

TLDR: As a tester, I need exact acceptance criteria for created modules that appear in Fedora, or I cannot be sure if my testing is enough.

Confirm (or specify) acceptance criteria for modules

Why do we need this?

I cannot find any modular criteria in the documentation at https://docs.pagure.org/modularity/. At the same time, they are important for our acceptance testing, otherwise the testing process is incomplete, which can affect user experience with modularity.

Current acceptance criteria

During several discussions with @psabata and @sgallagh , I was able to identify what I think should be the acceptance criteria for testing modules in Fedora. Please correct them if necessary:

  1. If the default stream of the module is defined, the module can be installed using dnf module install <module>. Since setting a default stream causes the module
    to override the non-modular content (FESCo approval), this behaviour is not strictly required.
  2. The module must install using dnf module install <module>:<stream>.
  3. The module has to offer at least one or more streams.
  4. Each stream has to offer at least one or more profiles.
  5. Each stream must have a default profile set.
  6. Each module must have a corresponding yaml file in the
    https://pagure.io/releng/fedora-module-defaults repository.

Fedora 31 Modularity Status

Recently, I have been testing the modules using the above mentioned criteria and I found this:

  1. In Fedora 31, there are 64 modules.
  2. 19 modules do not have the default profiles set.
  3. 15 modules do not have the defaults yaml file.

Modules without default profiles

  • cri-o
  • hub
  • libgit2
  • librealsense
  • lizardfs
  • mysql
  • openmpi
  • perl
  • pki
  • rpm
  • rust
  • setools
  • skychart
  • standard-test-roles
  • sway
  • testmodule
  • tomcat
  • varnish

Module without the YAML file.

  • cri-o
  • hub
  • librealsense
  • lizardfs
  • openmpi
  • perl-bootstrap
  • pki
  • rpm
  • rust
  • setools
  • skychart
  • standard-test-roles
  • sway
  • testmodule
  • tomcat

Modules without profiles

There are 3 modules that do not have any profile.

  • libgit2
  • rpm
  • perl-bootstrap

I think that these are the modules that only serve the dependency purposes, but as far as I know, we have been thinking about a default empty profile for these modules, so that QE could make sure, the profiles are empty on purpose and not by mistake. Can you confirm?

Suggestions for improvements

  • The criteria must be finally stated for now and ever. We can go with the above mentioned criteria.
  • The requirements must be forced and if they are not met, the module should not be let into Fedora. This is a good task for gating.
  • If such a module already is in Fedora, it should be fixed as soon as possible or removed from the system -> could we block on such modules?
  • For non-installable modules, there should be a special empty profile, with a specific name, so that we can easily check that the module is correctly created this way.
  • Everything must be fully documented.

Thank you for all responses.


mostly I agree with you: this stuff seems to be under-defined and needs to be defined. it is difficult for us to confidently test modules when we aren't sure what the requirements should be.

This stuff really all needs to be clearly nailed down for us to be able to reason...reasonably...about module behaviour in scenarios like upgrades. The fact that it isn't at present is clearly hurting us, as situations like the libgit2 one show.

Current acceptance criteria
...
If the default stream of the module is defined, the module can be installed using dnf module install <module>. Since setting a default stream causes the module

Addendum: it must also be possible to do dnf install <package_from_the_default_stream> without explicitly enabling the stream first.

to override the non-modular content (FESCo approval), this behaviour is not strictly required.

Rephrase that to be more clear that it means that modules are not required to have a default stream.

The module must install using dnf module install <module>:<stream>.
The module has to offer at least one or more streams.

"at least one or more" is redundant.

Each stream has to offer at least one or more profiles.

This is not actually required (see below).

Each stream must have a default profile set.

This is not entirely true. Each stream must set the entry for default profile in the fedora-module-defaults repository, but this setting may be an empty list (i.e. streamname: [ ]). In this case, the module author is indicating that no subset of the packages in the module always make sense to install. (For example, a hypothetical "CPAN" module might contain thousands of Perl packages, but no specific ones should always be installed from it.)

Each module must have a corresponding yaml file in the
https://pagure.io/releng/fedora-module-defaults repository.

Yes

Fedora 31 Modularity Status
Recently, I have been testing the modules using the above mentioned criteria and I found this:

In Fedora 31, there are 64 modules.
19 modules do not have the default profiles set.
15 modules do not have the defaults yaml file.
...
I think that these are the modules that only serve the dependency purposes, but as far as I know, we have been thinking about a default empty profile for these modules, so that QE could make sure, the profiles are empty on purpose and not by mistake. Can you confirm?

Yes, see my comments above.

Suggestions for improvements

The criteria must be finally stated for now and ever. We can go with the above mentioned criteria.

Once we finish this discussion, I agree.

The requirements must be forced and if they are not met, the module should not be let into Fedora. This is a good task for gating.
I concur.

If such a module already is in Fedora, it should be fixed as soon as possible or removed from the system -> could we block on such modules?

I'm not sure it's worth blocking on. Certainly not for F31 at least; I'm open to considering it as a Final criterion for F32+.

For non-installable modules, there should be a special empty profile, with a specific name, so that we can easily check that the module is correctly created this way.

I don't think that really makes sense. If it's supposed to be installable and isn't, that'll get reported as a bug quickly.

Everything must be fully documented.

Thanks for volunteering! 👊

Thank you for all responses.

Thank you, @sgallagh, for answers.

This is not entirely true. Each stream must set the entry for default profile in the fedora-module-defaults repository, but this setting may be an empty list (i.e. streamname: [ ]). In this case, the module author is indicating that no subset of the packages in the module always make sense to install. (For example, a hypothetical "CPAN" module might contain thousands of Perl packages, but no specific ones should always be installed from it.)

I am not sure I understand this. So, theoretically, if there is such a statement in that hypothetical cpan.yaml file, how does DNF show the status of the module? Do those modules also look like not having a default profile? Don't you plan something in the UX to make it clear that the stream's list of profiles is empty on purpose?

I'm not sure it's worth blocking on. Certainly not for F31 at least; I'm open to considering it as a Final criterion for F32+.

We haven't any plans for blocking this for F31, but we should definitely discuss it for upcoming releases, because when modules become more numerous and important, things need to be solved properly.

Thanks for volunteering! 👊

Not a problem, where do you keep the documentation source files?

Thank you, @sgallagh, for answers.

This is not entirely true. Each stream must set the entry for default profile in the fedora-module-defaults repository, but this setting may be an empty list (i.e. streamname: [ ]). In this case, the module author is indicating that no subset of the packages in the module always make sense to install. (For example, a hypothetical "CPAN" module might contain thousands of Perl packages, but no specific ones should always be installed from it.)

I am not sure I understand this. So, theoretically, if there is such a statement in that hypothetical cpan.yaml file, how does DNF show the status of the module? Do those modules also look like not having a default profile? Don't you plan something in the UX to make it clear that the stream's list of profiles is empty on purpose?

That's probably worth filing as a UX enhancement. I can't speak to whether that exists in the plan currently. Maybe @jmracek would know.

I'm not sure it's worth blocking on. Certainly not for F31 at least; I'm open to considering it as a Final criterion for F32+.

We haven't any plans for blocking this for F31, but we should definitely discuss it for upcoming releases, because when modules become more numerous and important, things need to be solved properly.

I am open to being convinced. I wouldn't block on it before Final Freeze though.

Thanks for volunteering! 👊

Not a problem, where do you keep the documentation source files?

https://pagure.io/fedora-docs/modularity

Login to comment on this ticket.

Metadata