@langdon @ralph @psabata For both fedmod's automated modulemd generation, as well as for ease-of-maintenance in general, it would be helpful to be able to specify "Just give me the default for the target system" option in modulemd files.

Actually adding that will require coordinated changes to both the modulemd spec (to nominate the special value) and to fm-orchestrator (to handle it properly), so should I start a fedora-devel thread about it?

I think this is wrong approach because we should be building module against all possible platforms and use context field to distinguish them.

There's no reason to build the "f27" branch of a platform-version-linked module across all versions of the platform module - if there was, the module's versioning scheme would be different in order to indicate that it wasn't tightly coupled to a particular version of the operating system.

Instead, you just want a one-to-one correspondence, whereby your f27 branch gets built against platform:f27, your master branch gets built against platform:master, etc.

Now, you could design the substitution scheme to emphasise the matching branch names, so you'd write something like this instead:

data:
    dependencies:
        requires:
            platform: <matching-branch>

And then have the build fail if there was no matching branch.

I moved from that to the "only build for the target system's default stream" because the latter seemed a bit more flexible, but I can also see how it doesn't really fit the conceptual model for how modular builds are managed.

What is "the target system"?

If you submit a module for building there is nothing like target. MBS reads all buildeps from the modulemd file, find their latest builds and tag them into the build root. (E.g. you can pretty easily put platform:master and platform:f27 at the same time into the build root. This is probably a bug if we assume we cannot enable more streams of the same module.)

What is "system profile's default stream"?

Is this a default platform stream defined somewhere in an installation image so that dnf knows that this system is Fedora 27 and not Fedora 28? I think MBS does not have access to this setting.

In my opinion there should be a new syntax for streams in dependencies in the modulemd file. (A whitelist or a blacklist or an all-available-streams keyword.) MBS would then search for all available streams matching the criterion (with recursive check for run requires to prvent from mixing multiple streams of the same module name) and started builds for all of them. If any of them failed, the whole task would fail (particular subtask could be left available as finished for free usage and caching).

The question is how to differentiate the resulting modules. MBS could generate the stream of the resulting module as concatenation of the dist-git branch and run-required module streams. The disadvantage would be the streams became long and hard to understand.

Or one could add another layer of identifiers and keep the stream equaled to the dist-git branch. (The context field? I don't know what it is it used for exactly.)

Also one need to pay attention that list of build and run requires can be different. E.g. you build from bootstrap but run-requires platform and perl. (This is a bad practice and should not be used but it happens.) Or you really build requires platform only but run requires platform and perl because your code can use any perl.

Yeah, the default streams are currently defined as part of assembling the system compose, so you're right that that information won't be available at module build time.

I think that leaves three main cases:

1. "Build separately for all streams". For example, the Python 3.6 stream should be built against all platform streams, not just one of them.
2. Build against the stream that matches the current module's dist-git branch. This is the desired option for any module that derives its branching layout from another module (e.g. those tracking the Fedora version number)
3. Build against the matching branch for another wildcard stream. For example, if building against all platform streams, you'd want a way to say "the stream in this module dependency should match the platform module's stream"

"Build separately for all streams". For example, the Python 3.6 stream should be built against all platform streams, not just one of them.

This is the plan. Better to ask @ralph when he will get time to implement this.

I think that leaves three main cases:

The second option is only a special case. Not all modules have Fedora-like branches. People sometimes use it just only because they cannot figure out any better stream identifier. And then the meaning is contains-the-same-packages-as-that-fedora. We still need to support the generic case. The last option becomes complicated if you realize that you can have multiple wildcard-stream dependencies and their dependencies alike. And it's not only about platform module. Now we have two perls and pythons.

I agree with ignatenkobrain that we need to build against all streams.

Non-conflicting combinations should be resolved by MBS and packager should be able to restrain of some of them (bacause we know they are broken, we do not want to support etc.).

My idea about the wildcard stream syntax is this: Keep the structure, redefine the meaning of the values:

• One word string -- this literal stream as it works now.
• Special value "any" (or empty string) -- build against all available
• White space separated words "a b c" -- build only against streams "a", "b", and "c".
• White space separated words string prefixed with an exclamation mark (or minus) "!a b c" -- build against all available except "a", "b", and "c".

I think anything that treats stream names purely as a set of unrelated strings, rather than as an ordered sequence, is going to lead to usability problems.

To provide a more concrete example, let's consider Python 3 application modules that want to opt in to being built for all actively maintained Python 3 stacks. Conceptually, that would look like:

platform: [inherit this from the python3 build stream somehow]
python3: [inherit this from the python3-ecosystem build stream somehow]
python3-ecosystem: [build for all currently active streams]

(Note: we should problem aim to use neutral terminology like "active" and "inactive" streams at the build definition layer)

Right now, "python3-ecosystem" only defines a single stream ("master"), but I expect we're eventually going to have to switch that to the CalVer naming scheme (i.e. "python3-ecosystem:2018.0", "python3-ecosystem:2019.0", etc) to cope with components that decide to change problematic default settings, or deprecate and remove problematic APIs.

Once that happens, then the feature that Python application modules need isn't "specify an exact set of python3-ecosystem streams to build for", it's "specify the minimum python3-ecosystem version needed". Applying an upper limit would then only happen as a workaround for cases where an ecosystem level update actually did break that particular application (until either the application was updated to handle it, or the ecosystem level change was identified as an accidental regression and fixed).

In other words you want the stream to have a version semantics. I.e. to be comparable, to have defined ordering. As far as I know stream is now a plain word without any other requirements. If you want this supported in the dependencies specification, we first need to change the stream specification.

What's the status of this, do we still need this with stream name expansion ?

@karsten As far as I know, it's still needed, since the f28 branch for python3-ecosystem is still depending on the master branches of platform and libtom, while the f27 branch has been manually edited to depend on the f27 branches. If python3-ecosystem ends up needing a branch for every version of Fedora, then the gains relative to the old branch-every-srpm model seem relatively limited (especially for as long as the SRPMs still need their dedicated branches to support traditional monolithic distribution).

Reading https://fedoraproject.org/wiki/Infrastructure/Factory2/Focus/StreamExpansion, the current stance still seems to be "Programming language ecosystems that integrate with the base platform can all just be dismissed as a weird edge case that we don't need to worry about, so we're going to ignore this problem". It's not even just Fedora's own Python (Ruby, Perl, et al) bindings that are going to need this, you can also bet that things like Ansible modules are going to want this capability.

So from my point of view, if modularity can't express "When building this module for version X of the Fedora platform, use version Y of Python" (or, even better, "When building this module for version X of the Fedora platform, consult the metadata for version X of the Fedora platform to determine which version of Python to use"), the potential benefits of it are going to be incredibly limited, as if I'm going to be cutting myself off from the base platform entirely, I can just run in a from scratch or Alpine Linux container, and ignore Fedora level packaging mechanisms entirely.

(Also cc'ing @cstratak and @pviktori)

If you wanted to do this in a generic way that didn't assume access to anything like Fedora's system profile mechanism, then you could structure this as a "stream map" feature at the module metadata level, so you might declare a dependency as something like:

    requires:
        platform: [master, f27, f28]
        libtom: [master, f27, f28]
        python:
            3.7: ["platform/master"]
            3.6: ["platform/f27", "platform/f28"]

That declaration would mean "Use the Python 3.7 stream when building for the platform master stream, but use the Python 3.6 stream when building for F27 or F28".

The exact spelling of the conditional stream dependencies would be completely up in the air, that's just the simplest possible approach that occurred to me, where you explicitly list the criteria for depending on a particular stream as "other_module/stream" entries. (I don't think the Python use case necessarily needs the ability to make dependencies conditional on multiple other modules, but if a use case for that was found, it could be expressed as sublists within the stream criteria list).

Other Python modules could then declare a dependency on python3-ecosystem, and inherit the appropriate module build matrix from there.

The second modulemd format https://github.com/fedora-modularity/libmodulemd/blob/master/spec.v2.yaml brings a list of dependency matrices:

- requires:
        platform: [master]
        libtom: [master]
        python: [3.7]
- requires:
        platform: [f27, f28]
        libtom: [f27, f28]
        python: [3.6]

I also think MSE and the new format solve this issue. The fact that there are still some modules that haven't been migrated is another thing.

We updated all modules that were added to the F28 compose or were included in the updates. Updating modules that aren't currently in the distro, as well as including them, is up to their maintainers.