#1175 SCLs -- FPC questions on backwards compat
Closed None Opened 5 years ago by toshio.

FPC has a question about a non-technical SCL assumption that properly belongs to FESCo. It also wants to give a heads up about another policy question that will arrive at FESCo's doorstep in due course:

=== Use cases and backwards compat ===
What are the use cases for SCLs? FPC has been assembling a [https://fedoraproject.org/wiki/User:Toshio/SCL_Guidelines_Q%26A#SCL.27s_Big_Picture_Goals_within_Fedora list here]. As part of that, the request for 100% backwards compat has become an issue. This seems to be contrary to the Fedora Foundations of "Features" and "First". Is this a purposeful deviation? Does FESCo agree with the desire for SCLs to promise complete backwards compatibility within an SCL for the life of the SCL ("multiple" Fedora releases)?

FPC is also concerned that SCLs don't provide anything useful for backwards compat without introducing unacceptable maintenance pain:

Backwards compat: "thin" scenario: SCLs should have the bare minimum to implement a platform. Other platforms would be built on top of it using SCL dependencies (ruby1.9 is one SCL. rails3 is a separate SCL that depends on it) -- Counter argument: Don't we already have ways to do this? compat packages, parallel installable python stacks, etc? What value do SCLs add if we follow this scenario?

Backwards compat: "thick" scenario: SCLs would be able to ship a platform and the stack that runs on top of it [ruby1.9 + rails3 + mintest, etc]. -- Counter argument: since the SCL has to maintain 100% backwards compat, any package included in an SCL cannot be changed. Therefore over time we will end up with a base package (ruby1.9) that we'd want to continue to support but some of the packages included on top of it would be unmaintained (and potentially unmaintainable where upstream has long since moved on). Where we have stacks built on stacks we also run the risk of having duplication: ruby1.9 and rails3 in one SCL, ruby1.9 and rails4 is in a different SCL, ruby2.0 and rails3 is in a third, etc (there is a desire to avoid this combinatorial explosion but so far no criteria to avoid it has been proposed)

=== SCL Process ===

This can't be answered yet because it depends on other questions (including the backwards compat question). It is something looming in the future though:

SCLs will need to be approved in some manner. Proposals so far are to model it after the package review process or the Fedora Changes process. If strict backwards compat is affirmed by fesco as a goal we need to control both what SCLs are allowed and what packages are in each of those SCLs (as every package added to an SCL adds to the API that must remain compatible for the life of the SCL).


Replying to [ticket:1175 toshio]:

=== Use cases and backwards compat ===
What are the use cases for SCLs? FPC has been assembling a [https://fedoraproject.org/wiki/User:Toshio/SCL_Guidelines_Q%26A#SCL.27s_Big_Picture_Goals_within_Fedora list here]. As part of that, the request for 100% backwards compat has become an issue. This seems to be contrary to the Fedora Foundations of "Features" and "First". Is this a purposeful deviation? Does FESCo agree with the desire for SCLs to promise complete backwards compatibility within an SCL for the life of the SCL ("multiple" Fedora releases)?

I would say yes. The idea of a SCL is that it is a platform to build on top of - if it's intended to have a same version, it should remain compatible. It's a distinction of having a fast-moving system base vs. a platform for layered apps that might remain more stable.

Backwards compat: "thin" scenario: SCLs should have the bare minimum to implement a platform. Other platforms would be built on top of it using SCL dependencies (ruby1.9 is one SCL. rails3 is a separate SCL that depends on it) -- Counter argument: Don't we already have ways to do this? compat packages, parallel installable python stacks, etc? What value do SCLs add if we follow this scenario?

compat/parallel/etc - the gain of SCL for the complication is that it i a single interface to handle this, as opposed to potentially different infrastructures for every different stack.

Replying to [comment:1 notting]:

I would say yes [complete backwards compat]. The idea of a SCL is that it is a platform to build on top of - if it's intended to have a same version, it should remain compatible. It's a distinction of having a fast-moving system base vs. a platform for layered apps that might remain more stable.

Well, that's one of the possible uses of SCLs. The linked section lists several things that SCLs allow us to do. They're not mutually exclusive but they're also not dependent on each other. I'm verifying that FESCo considers that to be The Important Goal of SCLs that we '''must''' enable (rather than some of the other things like allow us to package for Fedora software that currently depends on other versions of software).

Also note that we've been talking about whether having SCLs in different pieces of Fedora (Fedora Commons vs the Outer Rings) following different policies would be a good idea. That could allow us to target the stable api for developers for the Outer Rings and allowing packaging of software that needs older versions to the Fedora Commons scl use case. We haven't decided if that's a good idea yet -- it's another possibility being discussed.

Backwards compat: "thin" scenario: [...] What value do SCLs add if we follow this scenario?

compat/parallel/etc - the gain of SCL for the complication is that it i a single interface to handle this, as opposed to potentially different infrastructures for every different stack.

This isn't the only value, is it? Because we're also considering the last bullet point here: https://fedoraproject.org/wiki/User:Toshio/SCL_Guidelines_Q%26A#Collections_and_backwards_compat

  • Should a collection be used only for things that cannot be parallel packaged (e.g. python 3.3 can be packaged natively so it's not a candidate for an SCL but ruby1.9 cannot and thus it is) (Note -- I need to verify with slavek or someone who knows that this is in fact the case for ruby).

which would seem to detract from this particular plus.

So, I'd like to propose the following statement:

The Fedora Project's mission is to lead the advancement of free and open source software and content as a collaborative community. The values of "features" and "first" underpin this goal of advancement, and historically we have concentrated on packaging the newest possible versions of software with an explicit non-concern for backwards compatibility. However, this does not mean that we shun efforts to provide that compatibility. To the contrary, giving our users a compatibility path provides additional freedom to move quickly without disruption. If Fedora subprojects or packagers are interested in providing and maintaining software stacks which provide backwards or forwards compatibility and can commit the resources required to do so, such effort is very welcome.

does that help? Is it sufficient?

It helps but I don't think it's sufficient. there are several different people on the fpc with different angles on what troubles them about the backwards compat guarantees. I think the statement addresses the question about being a purposeful deviation from the fedora foundations. we still have to address:

  • whether we desire to force all scls to make this backwards compat guarantee
  • what the advantage of scls is over other methods of creating a backwards compat environment (cc'ing slavek because that might be a question he can help answer)

Replying to [comment:4 toshio]:

It helps but I don't think it's sufficient. there are several different people on the fpc with different angles on what troubles them about the backwards compat guarantees. I think the statement addresses the question about being a purposeful deviation from the fedora foundations. we still have to address:

  • whether we desire to force all scls to make this backwards compat guarantee

I think we do want to make backwards compat guarantee, to a certain point. But there is one use-case of SCLs that should probably be taken out of the backward compat promise. Let's consider this:

Puppet/OpenShift/OpenStack are complicated pieces of software and have large dependency stacks - tens to hundreds of deps, usually pinned to a specific version. This usually makes them very hard to impossible to package with system dependencies. So I've already heard some people from OpenStack talking about just creating an "openstack" SCL that would carry all OpenStack dependencies. In this situation, I don't think we should enforce backwards compat, because the collection is there just to provide OpenStack dependencies in specific versions. If these dependencies change, so should the collection, regardless of backwards compat.

This leads me to conclusion that dividing collections into "thin" and "thick" is not exactly what we want to do. Rather we should distinguish "generic" collections (e.g. ruby19-rails3, guaranteeing backwards compat) and "support" (anyone has a better name?) collections (e.g. openstack, guaranteeing that they will carry whatever OpenStack needs).

  • what the advantage of scls is over other methods of creating a backwards compat environment (cc'ing slavek because that might be a question he can help answer)
  • The fact that you can run the same command (e.g. "gem") in different environment and you don't have to find out whether it is "gem-18", "gem-1.8", "ruby18-gem" or whatever.
  • The fact that SCLs can easily act as a stable stack of dependencies, when built on different systems. Yes, you can do that even with standard methods, but then you start running into problems like the one above - binary names change, paths change, etc. With SCL, everything is always there in the same form and you can rely on it.
  • If you convert your specfile properly for SCL compatibility, you don't have to do any directory renaming and tinkering with paths, you just use the spec to build SCL with older/newer version. So you have almost the same spec for any version (even if you want to create bazillion SCLs with different versions of this package). This, among other things, makes merging patches to dist-git work much easier.

As a sidenote, I'd like to mention that "First" is also a use case for SCLs. Just yesterday, I was talking with a guy who works with v8 on Fedora and he was complaining that we have the oldest v8 version of all major distributions, except maybe Debian stable (supposedly, this is because of Chromium compatibility, but I didn't check). More generally, there are things, bleeding edge technologies or whatever we may call them, that we don't always want to put into Fedora, because they're too bleeding edge in the time of Fedora release, possibly at Alpha or Beta. But we may want users to have access to this technology First. I agree that this usecase may not be so often needed for Fedora as backward compat SCLs, but I definitely wouldn't say that SCLs have nothing to offer in this regard.

Replying to [comment:5 bkabrda]:

I think we do want to make backwards compat guarantee, to a certain point. But there is one use-case of SCLs that should probably be taken out of the backward compat promise. Let's consider this:

Puppet/OpenShift/OpenStack are complicated pieces of software and have large dependency stacks - [...] If these dependencies change, so should the collection, regardless of backwards compat.

<nod> So actually, this gets right to the reason for my question. This is one of the other use cases listed on https://fedoraproject.org/wiki/User:Toshio/SCL_Guidelines_Q%26A#SCL.27s_Big_Picture_Goals_within_Fedora : "Backwards Compat: (toshio) Collections could allow Fedora to package software that depends on older and not natively parallelizable versions of a software stack." In my estimation, if we said that SCLs must be 100% backwards compatible this use case would be possible but much harder (to the point where I don't think a large dep tree like openstack would be willing to package in scl format).

This leads me to conclusion that dividing collections into "thin" and "thick" is not exactly what we want to do. Rather we should distinguish "generic" collections (e.g. ruby19-rails3, guaranteeing backwards compat) and "support" (anyone has a better name?) collections (e.g. openstack, guaranteeing that they will carry whatever OpenStack needs).

In https://fedoraproject.org/wiki/User:Toshio/SCL_Guidelines_Q%26A#Collections_and_backwards_compat there's two other proposals that also try to address this. One is very similar to what you're saying: "[...] SCLs in The Outer Rings should have a strict backwards compat section but SCLs in Fedora Commons are more relaxed [...] Fedora Commons SCLs are primarily for enabling different versions of dependencies for software that we want to ship in Fedora Commons; Outer Ring SCLs are primarily for enabling developers [...]" I split this on Outer Rings vs Fedora Commons because separate repos seem like the best way to differentiate to end users but there might be other ways to achieve this.

The other proposal is: "[...] mark some pieces of an SCL as having guaranteed backwards compat while others are only implementation detail. If you had an SCL that shipped ruby1.9 but it needed a newer version of OpenSSL you might ship that inside the SCL at first. But you wouldn't want people to rely on that particular version being available in the SCL in the future (once the base OS was updated to include that version.)" To use the openstack example, you'd ship an Openstack1.0 SCL where the openstack api was marked for backwards compat but the dependencies were not. The technical question is whether there is some way we could mark the pieces of the package like this (maybe something in the metapackage? Maybe with special virtual Provides?)

  • what the advantage of scls is over other methods of creating a backwards compat environment (cc'ing slavek because that might be a question he can help answer)

[snipped reasons]

Thanks. Added to the summary page: https://fedoraproject.org/wiki/User:Toshio/SCL_Guidelines_Q%26A#SCL.27s_Big_Picture_Goals_within_Fedora

I'd like to mention that "First" is also a use case for SCLs. [...]

For the v8 example, I'd much rather see the newer v8 in Fedora and chromium depend on a backwards compat v8-in-SCL. (James would likely say that he'd rather see the v8 in Fedora updated and a v8-compat type package created for chromium which might work too.) This plays well with Fedora's goal of going right up to the bleeding edge. Going further could happen but we have to be careful that maintainers don't ship stuff in SCLs and then devote their time to fixing the SCL rather than the older packages that were in the distro itself. We would also have to throw out mmaslano's proposed criteria: "that SCLs must be known stable, well tested, and working well with other packages in its SCL. Ways to evaluate these objectively would need to be defined" -- https://fedoraproject.org/wiki/User:Toshio/SCL_Guidelines_Q%26A#What_is_the_logical_limit_of_the_.22size.22_of_that_collection.3F

Replying to [comment:6 toshio]:

Replying to [comment:5 bkabrda]:

I'd like to mention that "First" is also a use case for SCLs. [...]

For the v8 example, I'd much rather see the newer v8 in Fedora and chromium depend on a backwards compat v8-in-SCL. (James would likely say that he'd rather see the v8 in Fedora updated and a v8-compat type package created for chromium which might work too.) This plays well with Fedora's goal of going right up to the bleeding edge. Going further could happen but we have to be careful that maintainers don't ship stuff in SCLs and then devote their time to fixing the SCL rather than the older packages that were in the distro itself. We would also have to throw out mmaslano's proposed criteria: "that SCLs must be known stable, well tested, and working well with other packages in its SCL. Ways to evaluate these objectively would need to be defined" -- https://fedoraproject.org/wiki/User:Toshio/SCL_Guidelines_Q%26A#What_is_the_logical_limit_of_the_.22size.22_of_that_collection.3F

For the record, the reason for v8 being locked to a specific version actually has nothing to do with !Chromium/Chrome at all. I coordinated with Tom Callaway where we agreed that Fedora would carry whatever version of v8 was required to support Node.js and not Chromium (because there's little-to-no chance of Chromium ever getting into Fedora at all). Node.js was the only other major effort relying on v8 at that time, so we decided to carry the version they depend on.

v8 is ''very'' version-specific, because Google pretty much treats it as a copylib. It does ''not'' maintain backards-compatibility even between minor version releases, so upgrading to the latest v8 would guarantee that all Node.js software would stop functioning.

Essentially, we're treating the v8/Node.js pair as a fundamental language stack (similar to Python 2.7) for Fedora.

Now, this ''is'' a place where SCLs could potentially help, as we could move the entire Node.js SIG to an SCL-based approach. That seems a bit heavy-handed, though, given the literally hundreds of packages in Fedora that depend on the Node.js runtime. So I'd rather see v8-<version> SCLs for newer major/minor releases and keep the stable version that supports the current Node.js release as the standard system copy.

Replying to [comment:7 sgallagh]:

Replying to [comment:6 toshio]:

Replying to [comment:5 bkabrda]:

I'd like to mention that "First" is also a use case for SCLs. [...]

For the v8 example, I'd much rather see the newer v8 in Fedora and chromium depend on a backwards compat v8-in-SCL. (James would likely say that he'd rather see the v8 in Fedora updated and a v8-compat type package created for chromium which might work too.) This plays well with Fedora's goal of going right up to the bleeding edge. Going further could happen but we have to be careful that maintainers don't ship stuff in SCLs and then devote their time to fixing the SCL rather than the older packages that were in the distro itself. We would also have to throw out mmaslano's proposed criteria: "that SCLs must be known stable, well tested, and working well with other packages in its SCL. Ways to evaluate these objectively would need to be defined" -- https://fedoraproject.org/wiki/User:Toshio/SCL_Guidelines_Q%26A#What_is_the_logical_limit_of_the_.22size.22_of_that_collection.3F

For the record, the reason for v8 being locked to a specific version actually has nothing to do with !Chromium/Chrome at all. I coordinated with Tom Callaway where we agreed that Fedora would carry whatever version of v8 was required to support Node.js and not Chromium (because there's little-to-no chance of Chromium ever getting into Fedora at all). Node.js was the only other major effort relying on v8 at that time, so we decided to carry the version they depend on.

v8 is ''very'' version-specific, because Google pretty much treats it as a copylib. It does ''not'' maintain backards-compatibility even between minor version releases, so upgrading to the latest v8 would guarantee that all Node.js software would stop functioning.

Essentially, we're treating the v8/Node.js pair as a fundamental language stack (similar to Python 2.7) for Fedora.

Now, this ''is'' a place where SCLs could potentially help, as we could move the entire Node.js SIG to an SCL-based approach. That seems a bit heavy-handed, though, given the literally hundreds of packages in Fedora that depend on the Node.js runtime. So I'd rather see v8-<version> SCLs for newer major/minor releases and keep the stable version that supports the current Node.js release as the standard system copy.

Interesting approach, but it make sense. We were more thinking about shipping v8-exeperimental branch as scl, which is a lot used for development. Stable releases are used with projects like node.js, but quick development seems to be happening on experimental. But with v8 there are many possibilities. It was just one use-case of SCL.

These proposals were agreed upon by yesterday's FESCo:

  • AGREED: it is okay for people maintaining an SCL to guarantee
    backwards compatibility, or to make a separate SCL for backwards (or
    forwards) compatibility. (+9, -0, 0) (t8m, 18:40:21)
  • AGREED: A SCL when proposed must document it's purpose, including
    what API/ABI promises to provide to packages outside of the SCL; it
    is then expected to maintain that promise [and that compatibility
    promise only] (+8, -0, 0) (t8m, 18:58:28)

Login to comment on this ticket.

Metadata