Is releasing it as Child of Boltron II: The Servering a possibility?

I edited the post to use a numbered list so we can discuss the pros and cons via shorthand. As in, "Con 5 really concerns me based on user feedback we've seen so far".

I edited the post to use a numbered list so we can discuss the pros and cons via shorthand. As in, "Con 5 really concerns me based on user feedback we've seen so far".

ohh.. good thought

Is releasing it as Child of Boltron II: The Servering a possibility?

Certainly a possibility. However, it raises the issue of being "still off to the side" and may not generate the feedback we want.

However, I think we should make it Child of Boltron: True Unification

I think it'd be helpful to know how the Modularity and Server Working Groups are leaning on this. It'd be useful to have QA input too, I think.

My take as a member of the Server Working Group. Don't release it with the Server name attached to it. Fedora 27 Modular Preview (or similar Bikeshed name) and make sure that people are not confused about the lack of tools or the changes in workflow. At this time, the cons to me outweigh the pros.

I wish we had gone with the approach @smooge stated to begin with.

I think we still composed the regular server edition, so we could just run it through its paces and then release that as Fedora 27 Server Edition, couldn't we?

I think we still composed the regular server edition, so we could just run it through its paces and then release that as Fedora 27 Server Edition, couldn't we?

Yes. In fact, that's why we did it this way. Or, at least, we think so.

I think it'd be helpful to know how the Modularity and Server Working Groups are leaning on this. It'd be useful to have QA input too, I think.

I have tried to get individuals to attend the council meeting. I don't think the groups have unified opinions.

Additionally, with F28 we intend for the modules to be entirely opt-in, rather than mandatory from the user’s perspective. We will default to including everything available in the Fedora package collection and only modularize those pieces that make sense (things that we want to provide parallel availability for or otherwise use the module metadata for informational purposes).

I don't want to derail this ticket, but I would like to see what the future solution is aiming for.
Do you have documents on what problems that will solve, or features that will provide? Also, to clarify, parallel availability does not necessarily mean parallel installation, correct?

As a Modularity contributor, I think "Modular Preview" or "Child of Boltron" would be a more apt name than claiming it as a usable Fedora Server release.

I'm not sure we have a doc anywhere, but the essence of the problem with our current approach is:

1. Modules are currently required to depend solely on content that is also available via a module (including for their build deps)
2. Disentangling build deps for build deps can get "exciting"
3. Even managing low level runtime deps can get complicated for complex services like FreeIPA and Samba
4. None of that low level work is technically necessary to reach the goal of allowing selective upgrades and downgrades of particular things that run on the base platform without necessarily being in the base platform (language runtimes, databases, application servers, etc).

For Boltron, we dealt with this via the "bootstrap" pseudo-module. For F27, we tried to eliminate the reliance on "bootstrap", and that's the aspect that hasn't really worked.

So my interpretation of Langdon's proposal (which I like) is that for F28 we go the other way: instead of trying to eliminate the "bootstrap" pseudo-module, we instead figure out how to build it in a maintainable way, and call it "platform" instead.

"Make the base platform smaller" can then be a long term effort, as folks identify subsections of it that can be carved out as true meaningful modules where being able to upgrade and downgrade them independently of the rest of the system is actually useful.

I am not that technically involved, but from an outside POV I would prefer a clear solution, not a something like Modular but still developing as F28 will change many things release.
If we are able to ship Modular Server (not other names) and to manage upgrades, we should do that, if not or if we have too many doubts we have to wait for F28 to get a tested Modular Server release.

Personally I don't see so many problems having an F26 release for a year, instead of 6 months. Server users won't change their OS so often and upgrading to F27 is still possible.

I believe we should release the F27 traditional server build as soon as it has gotten through QA. I believe we should capitalize on the non-lockstep release by respinning it to be fresher, if appropriate, and we should trumpet its updates. This is an opportunity to improve the press/marketing visibility of server - let's take it. Depending on results, I could see leaving server on a non-lockstep release cycle.

Echoing @jwboyer a bit I believe modularity needs to clearly articulate the problem being solved. I think they need to do so in a market oriented way. Specifically, we need to answer the question, "how is this better than containers?" That stumbling block hits every modularity conversation I have. If modules are opt-in, exactly what is a module getting you? How does this differ from a dnf group?

I also believe that we need to understand how F28's goal, as outlined above, actually gets us to the goal. It sounds like we built something, realized it was too hard (turtles all the way down) so now we are building what is essentially packages of rpms that are versioned. While I am not a distro builder, it seems like this doesn't require much in the way of new tech as I am told we could release multiple versions of RPMs today but don't by choice (this may be incorrect?). Also, if we don't get parallel installation just locked versioning, this doesn't seem to be adding much other than an administrative interface, assuming the above statement about versions is true.

I am not opposed to experimenting, even when you get negative results. However, I think it is important that we get the experiment into a usable condition before we call it an edition. I am ok with the experimentation continuing and in fact I want to encourage more, but I would like to see a more docs-driven development approach taken.

For visual types, like me, I've visualized the difference of the current model and the one @langdon proposes: https://asamalik.fedorapeople.org/modularity-hybrid.png

This would allow packagers to modularize their stuff only when they want to. It would create less presure at packages because they could decide whether to opt-in or not. Modularity would still give us the ability to:

1. Make more streams of software available, such as "latest" and "LTS".
2. Build and use multiple versions of dependencies.
3. Build containers with multiple versions of software thanks to having the above available in Fedora. This is why parallel availability is important. We don't want to re-invent containers by implementing our own parallel installability.

@bex I think modularity is not better than containers, it is just different thing. But I'm huge +1 for modularity to describe which problems they are trying to solve. But I would prefer it not in marketing way like "we want to provide multiple versions of same package and let people to choose" because it is simply not enough. We already have mechanisms to do this while modularity (at this point, with current implementation) creates a lot of burden instead of simply solving problem. So what I would like to see is some accumulated post of every problem modularity tries to solve + technical details why it was done this way.

Just FYI, Stratis Storage (which is a change for F28) has such kind of paper I would like to see for modularity: https://stratis-storage.github.io/StratisSoftwareDesign.pdf

Also to clear @bex part about multiple versions of RPMs. Since we switched to dnf, we can explore possibility of doing much better things using libsolv to solve dependencies without introducing totally new concept of building distro. We can do install multiple versions in parallel (although this is not what modularity is providing).

@asamalik

Build and use multiple versions of dependencies.

But how? I need to build app X which needs LLVM5 and app Y which needs LLVM4. So modularity is proposing to use 2 streams for LLVM (aka no compat packages), but you would not be able to install them in parallel while you can do this today and fedora and you actually need to.

llvm4.0-libs-4.0.1-2.fc28.x86_64
llvm-libs-5.0.0-5.fc28.x86_64

@adamwill How does QA feel about @bex's suggestion of spinning a new Fedora 27 Server compose with updates rolled in?

(That is, a non-modular regular-old Fedora Server)

Also, @sgallagh and @mohanboddu — because, let's be upfront, asking for an updated, revalidated Server Classic release is asking specific people to do a lot of work next month.

(Some bigger picture stuff here, re @ncoghlan):

"Make the base platform smaller" can then be a long term effort, as folks identify subsections of it that can be carved out as true meaningful modules where being able to upgrade and downgrade them independently of the rest of the system is actually useful.

Can we — maybe for F29 — split this into two repositories, the base/platform and an everything-else-non-modular? Even without using modularity to do this, it would give us some obvious lines that we can then color in different ways.

I'd also like to find an easy way for small leaf packages to opt into stream-based development (rather than release-based), without the overhead of the packager needing to create and care about a whole module or users to need to think about that package as a module now. For a very concrete example, I maintain the package calc, which has two upstream streams which they call "stable" and "unstable". The rules for this are complicated and weird and have nothing to do with the version numbers, but they're clearly labeled by upstream (see here). Right now, I tend to just ignore the unstable stream, except sometimes I don't, and I update in rawhide and let things trickle in.

I'd like to instead have a stable and unstable branch in dist-git and that's all, and have those built for all releases (including EL) without me having to do anything extra except maybe a one-time configuration. And I'd like users to be able to just yum install calc and get the stable branch without any previous enablement or selection, with the option of enabling the unstable branch instead.

@mattdm, you can do this even right now. Create packages called calc and calc-unstable, add Provides: calc and Conflicts: calc to both. That's it.

dnf install calc will install calc, but you can easily swap it with calc-unstable. And it is even doable with dnf to show you choices.

@mattdm, you can do this even right now. Create packages called calc and calc-unstable, add Provides: calc and Conflicts: calc to both. That's it.
dnf install calc will install calc, but you can easily swap it with calc-unstable. And it is even doable with dnf to show you choices.

You can't though, because you need branches for every single release where you might just be doing "git merge master" for both stable/unstable packages, giving you anything up to a dozen branches.

What would be useful is havign a single calc package, and two branches, and specify which branches are used on which releases, something "fedpkg set-branch f27 stable" and "fedpkg set-branch f28 unstable" or "fedpkg set-branch el6 stable-lts".

Then being able to build the stable branch automatically against all current releases with that branch enabled. So if there's a new stable release to fix CVEs a 'fedpkg build' would essentially do a build for all stable/enabled releases so you don't need to do numerous "fedpkg switch-branch f26; git merge master; git push;fedpkg build" IE make it easy for the maintainer by using automation.

Also, @sgallagh and @mohanboddu — because, let's be upfront, asking for an updated, revalidated Server Classic release is asking specific people to do a lot of work next month.

To be clear, I'm not asking for this. I think that if we make this decision, my personal stance is that we're acknowledging that F27 does not have a Server Edition. With all that entails and implies.

@sgallagh Yeah, it was kind of the reverse question: from your point of view as one of the people who does the actual labor, is it reasonable for the Council to ask for a Classic F27-with-updates Server Edition, if we decide that "does not have a Server Edition" is really bad?

(I'm not saying we will decide that, just making sure we're clear on what the actually-available choices are. It may be that we decide that not having a Server Edition is really bad, but... oh well.)

@sgallagh Yeah, it was kind of the reverse question: from your point of view as one of the people who does the actual labor, is it reasonable for the Council to ask for a Classic F27-with-updates Server Edition, if we decide that "does not have a Server Edition" is really bad?
(I'm not saying we will decide that, just making sure we're clear on what the actually-available choices are. It may be that we decide that not having a Server Edition is really bad, but... oh well.)

It's reasonable to ask, but I'm not committing to doing the work. I'm over-committed as it is. If you can find volunteers, I might be able to oversee it.

@ignatenkobrain yes, I can, but I'm not going to, because that's literally double the work, rather than being less work. As @pbrobinson says.

Let me go even bigger picture. Here's what I want:

1. To provide users with software stacks in different versions, where those stack versions span multiple Fedora and EL releases. I had high hopes for SCLs as a solution here, but that both had technical problems and got stuck in... non-technical details. Basically, SCLs.next.
2. To test those stacks as a unit in integration testing, and provide reasonable assurance and guidance about API changes within the lifespan of a given version of a stack.
3. To reduce packager effort by automating builds of applications ("leaf packages") against multiple bases, as in my calc example. This can either be done by leveraging Arbitrary Branching (as modularity does) or by some kind of automated cherry-picking bot. I don't care about the implementation really, except that it needs to be really, really simple for packagers, not another layer of complexity.
4. To replace EPEL by automatically providing all Fedora stacks and leaf application packages to EL users (but without committing to an EL lifecycle) — possibly providing containers is adequate in many cases here, but we need some way to make that transparent and manage the container lifecycles that I think we just don't have (short of everyone running OpenShift on everything).
5. To have the option of having Fedora Server or Fedora IoT have a 2-3 year maintenance cycle instead of 13 months, without needing to extend that lifespan to every single package in the Fedora package collection.

Does modularity as it stands, or as it is proposed here for F28, get us these things?

@ignatenkobrain yes, I can, but I'm not going to, because that's literally double the work, rather than being less work. As @pbrobinson says.
Let me go even bigger picture. Here's what I want:

To provide users with software stacks in different versions, where those stack versions span multiple Fedora and EL releases. I had high hopes for SCLs as a solution here, but that both had technical problems and got stuck in... non-technical details. Basically, SCLs.next.
To test those stacks as a unit in integration testing, and provide reasonable assurance and guidance about API changes within the lifespan of a given version of a stack.
To reduce packager effort by automating builds of applications ("leaf packages") against multiple bases, as in my calc example. This can either be done by leveraging Arbitrary Branching (as modularity does) or by some kind of automated cherry-picking bot. I don't care about the implementation really, except that it needs to be really, really simple for packagers, not another layer of complexity.
To replace EPEL by automatically providing all Fedora stacks and leaf application packages to EL users (but without committing to an EL lifecycle) — possibly providing containers is adequate in many cases here, but we need some way to make that transparent and manage the container lifecycles that I think we just don't have (short of everyone running OpenShift on everything).
To have the option of having Fedora Server or Fedora IoT have a 2-3 year maintenance cycle instead of 13 months, without needing to extend that lifespan to every single package in the Fedora package collection.

Does modularity as it stands, or as it is proposed here for F28, get us these things?

I don't think modularity provides any of those things, at least not in a simple straight forward way, I personally believe that most of what I mentioned can be provided with a little dist-git work, plus some tooling/logic, whether that be fedpkg/fedmsg or some other tool, i don't believe any of it is really changes in the distro, but rather the tools/CI etc that are around dist-git/fedpkg.

@mattdm,

To provide users with software stacks in different versions, where those stack versions span multiple Fedora and EL releases. I had high hopes for SCLs as a solution here, but that both had technical problems and got stuck in... non-technical details. Basically, SCLs.next.

Are you interested in parallel-installability here?

To reduce packager effort by automating builds of applications ("leaf packages") against multiple bases, as in my calc example. This can either be done by leveraging Arbitrary Branching (as modularity does) or by some kind of automated cherry-picking bot. I don't care about the implementation really, except that it needs to be really, really simple for packagers, not another layer of complexity.

That is not about modularity in current state, but I definitely agree here. As a packager I am really hoping for this for last 5 years. That's actually could be simply solved by using OBS instead of Koji. But this comes with a lot of other implications, so we could improve koji to do things.

To have the option of having Fedora Server or Fedora IoT have a 2-3 year maintenance cycle instead of 13 months, without needing to extend that lifespan to every single package in the Fedora package collection.

How could you imagine doing this? So packagers are committed to support packages more or...?

But at this point, as @pbrobinson says, existing modularity doesn't provide solution to any of those problems.

That is not about modularity in current state, but I definitely agree here. As a packager I am really hoping for this for last 5 years. That's actually could be simply solved by using OBS instead of Koji. But this comes with a lot of other implications, so we could improve koji to do things.

Actually i don't see koji as the problem here, koji just builds what it's told to, if you look at most git hashes for packages that have been "git merge master" they are pretty much the same across branches, it comes down to policy and implementation details. Most of that could be implemented with some tooling work around dist-git/fedpkg and friends. Not really related to modularity discussion other than modularity doesn't fix any of that.

@mattdm,

To provide users with software stacks in different versions, where those stack versions span multiple Fedora and EL releases. I had high hopes for SCLs as a solution here, but that both had technical problems and got stuck in... non-technical details. Basically, SCLs.next.

Are you interested in parallel-installability here?

Not particularly, although that would be nice. I'm happy with "run on different VMs or in different machines" as the general approach. I don't think we need to revisit that now.

To have the option of having Fedora Server or Fedora IoT have a 2-3 year maintenance cycle instead of 13 months, without needing to extend that lifespan to every single package in the Fedora package collection.

How could you imagine doing this? So packagers are committed to support packages more or...?

For this to work, someone would need to commit to the maintenance, yes. That's clearly not feasible with the current state of n-thousand packages in an undifferentiated repository, but it might be with a reduced subset.

@langdon, where did the outputs of https://fedoraproject.org/wiki/Objectives/Fedora_Modularization,_Requirements_Phase end up? I don't see them at https://docs.pagure.org/modularity/docs.html anywhere. I think that would really help at this point.

Are there things other than the wishlist I gave above that are Modularity goals? Are any of my items definitely out of scope?

To have the option of having Fedora Server or Fedora IoT have a 2-3 year maintenance cycle instead of 13 months, without needing to extend that lifespan to every single package in the Fedora package collection.

From the IoT base image PoV I fully intend it to be Atomic based and just roll from one Fedora release to the next using stable/test/devel style atomic branches (or what ever they're called) so I don't see how modularity assists with that. From an application layer, say Node-RED, having different versions available in different containers to run on top might be a modularity thing, but I thing it should be achievable with either different naming noderedXX or the different branches matt mentions above

@pbrobinson koji is a problem because it makes available into buildroot only one (latest built) version of package. And doesn't really expect to work with multiple versions of same package.

Have we analyzed the nix package manager for parallel installability?

https://en.wikipedia.org/wiki/Nix_package_manager

@bowlofeggs Fundamentally, nix does this by just putting them in different places on the filesystem. We can do that with RPMs just as well, too.

As an aside, we also have the fun ability (which we don't really use) to make Relocatable RPMs. That capability could be leveraged to make parallel installable things in a flexible manner, but we should talk to the RPM guys more about that.

While I appreciate the technical discussion on wants and possibilities, capturing it in this ticket isn't really the right venue. This kind of stuff should be done on the Fedora devel list and we should restrict ourselves to solving the question that @langdon has asked in the initial comment.

(I realize I probably opened this can of worms with my question above, so I'll take the blame for the divergence. Let's get back on focus here though.)

We shouldn't be solving technical things here, but I think having "why we're doing this" clear is important for making the right choice on the question at hand.

We shouldn't be solving technical things here, but I think having "why we're doing this" clear is important for making the right choice on the question at hand.

Yes, I agree with that part.

So, yeah, anyway — I think Langdon's pros and cons look pretty comprehensive. Does anyone have others to add?

Here's my thoughts on each of them:

• P1. We said we’d ship it.
  • Eh. We did, but I think it's more important to be responsive.
• P2. It’s pretty close to GA status.
  • This is good. But, maybe with the other things leans towards "Child of Boltron"
• P3. The user UI has no known major changes needed before F28.
  • How about unknown ones? How certain are we that we've got it right now? If it's unclear, that also weighs towards Boltron II: This Time, It Boots on Hardware
• P4. We can technically push updates via Bodhi.
  • This seems... super fragile.
• P5. This is Fedora: freedom/friends/features/first … quality is important but on what to judge quality?
  • I think this is more a question than a pro. :)

• C1. We know we’ll be changing the underlying architecture and repo layout “significantly” for F28.
  • Eeep. This is scary. We definitely need system upgrades to work once we've released, and going from regular F26 to modular F28 is going to be a big enough change that I don't want also worry simultaneously about F27->F28.
• C2. Fedora, as a brand, might be impacted.
  • Yeah. This too leans towards presenting this as After Boltron: The Sequel.
• C3. The packager experience will change from F27 to F28.
  • This might be my biggest worry. We really need packagers to feel comfortable and on board and we don't have a lot of trust to burn in this area.
• C4. Doing updates for F27 will take resources from F28 work needed to change everything.
  • But, we need to get updates smoothed out for F28 anyway, so I think going through the process for one cycle is actually a good thing.
• C5. It is missing an enormous amount of software desired by server admins.
  • This is my alternate contender for biggest worry. We're not likely to get admins to give this a second look if the first experience is bad.

I had a short discussion about this with @mattdm on IRC. I told him that my opinion was that it's not really worth releasing as Boltron 2: Electric Boogaloo because it would really only encourage people to try out (and try packaging for) something that would ultimately be considerably different from what we eventually ship as an official offering.

I'd personally prefer that we just be honest to our users and the tech media and say "Hey, we tried something new and discovered that there were some real problems with it. Rather than fall into the sunk-cost fallacy and drive it to an underwhelming completion, we're going to take a step back and try a different approach. Details on the new design to follow soon."

I had a short discussion about this with @mattdm on IRC. I told him that my opinion was that it's not really worth releasing as Boltron 2: Electric Boogaloo because it would really only encourage people to try out (and try packaging for) something that would ultimately be considerably different from what we eventually ship as an official offering.
I'd personally prefer that we just be honest to our users and the tech media and say "Hey, we tried something new and discovered that there were some real problems with it. Rather than fall into the sunk-cost fallacy and drive it to an underwhelming completion, we're going to take a step back and try a different approach. Details on the new design to follow soon."

I'm leaning this way myself. However, that still leaves the question @mattdm posed around traditional Server.

I had a short discussion about this with @mattdm on IRC. I told him that my opinion was that it's not really worth releasing as Boltron 2: Electric Boogaloo because it would really only encourage people to try out (and try packaging for) something that would ultimately be considerably different from what we eventually ship as an official offering.
I'd personally prefer that we just be honest to our users and the tech media and say "Hey, we tried something new and discovered that there were some real problems with it. Rather than fall into the sunk-cost fallacy and drive it to an underwhelming completion, we're going to take a step back and try a different approach. Details on the new design to follow soon."

I'm leaning this way myself.

me too, even if it wasn't clear from my coment above.

However, that still leaves the question @mattdm posed around traditional Server.

A colleague who runs Fedora Server on several machines today said they had been doing a dnf update from F26 to F27. They were not 100% sure, when pressed this was the case and that may have done F25 to F26. When we released F27 did we include an upgrade path for F26 traditional server users to something with F27 bits? If so, this pushes me very firmly into the "ship an ISO/release" camp as we have a partial base that got something.

A colleague who runs Fedora Server on several machines today said they had been doing a dnf update from F26 to F27. They were not 100% sure, when pressed this was the case and that may have done F25 to F26. When we released F27 did we include an upgrade path for F26 traditional server users to something with F27 bits? If so, this pushes me very firmly into the "ship an ISO/release" camp as we have a partial base that got something.

Well, we more-or-less validated that upgrades to F27 worked from F26 if people went that way (and I updated a fairly critical system today from F26 Server to F27 by way of dnf system-upgrade with no ill effects).

I'll bring it up at the Server SIG meeting in 30 minutes; if QA decides they want to try to validate the Final GA in terms of Server Edition, we could attempt it, I suppose. But if it fails, I'm not sure it's worth the effort to revive that release train; I'm not even sure if rel-eng would be able to do so without tripping over the earlier release.

So I guess that's my proposal: if the Server DVD that matches the 27 GA release already passes validation, release it. If not, I'd say it's probably more effort than it's worth to do a fresh install disk.

A colleague who runs Fedora Server on several machines today said they had been doing a dnf update from F26 to F27. They were not 100% sure, when pressed this was the case and that may have done F25 to F26. When we released F27 did we include an upgrade path for F26 traditional server users to something with F27 bits? If so, this pushes me very firmly into the "ship an ISO/release" camp as we have a partial base that got something.

Well, we more-or-less validated that upgrades to F27 worked from F26 if people went that way (and I updated a fairly critical system today from F26 Server to F27 by way of dnf system-upgrade with no ill effects).

The upgrade path does make it easier to not ship an installation DVD. That still allows existing users to get f27 content.

I'll bring it up at the Server SIG meeting in 30 minutes; if QA decides they want to try to validate the Final GA in terms of Server Edition, we could attempt it, I suppose. But if it fails, I'm not sure it's worth the effort to revive that release train; I'm not even sure if rel-eng would be able to do so without tripping over the earlier release.
So I guess that's my proposal: if the Server DVD that matches the 27 GA release already passes validation, release it. If not, I'd say it's probably more effort than it's worth to do a fresh install disk.

I believe that seems reasonable in light of upgrades functioning.

@mattdm Hum, let's see. Practically speaking we could probably make it happen. From QA's perspective I'll say we might need to fiddle around and do a few things manually, but we probably can do it.

From a wider perspective It brings up a lot of sort of interesting theoretical questions and odd corner cases, though, as we've literally never done it before AFAIK, and lots of our processes kinda assume quite heavily that we release one compose. Doing a new compose and then releasing some part of it as being somehow "part" of the Fedora 27 release (which was compose Fedora-27-20171105.0 , no ifs or buts about it) is a pretty radical departure from our existing practice and would be a bit of a leap into the dark. It would be much less radical to just update getfedora to list the Server images from the 27-20171105.0 compose, just as if we'd shipped a perfectly normal Fedora 27 Server release all along. The images are already on the mirrors, I believe.

The only way in which the Fedora-27-20171105.0 compose fails release validation for Server, that I'm aware of, is https://bugzilla.redhat.com/show_bug.cgi?id=1503321 . But we don't really need a media respin to fix that, as it's an upgrade issue and upgrades use the repos 99.99% of the time. We could ship the 20171105.0 Server images and fix the upgrade issue in the repos, or we could even choose to just document it on CommonBugs as there is a relatively easy workaround.

It would be much less radical to just update getfedora to list the Server images from the 27-20171105.0 compose, just as if we'd shipped a perfectly normal Fedora 27 Server release all along. The images are already on the mirrors, I believe.

This is exactly what I was proposing above.

The only way in which the Fedora-27-20171105.0 compose fails release validation for Server, that I'm aware of, is https://bugzilla.redhat.com/show_bug.cgi?id=1503321 . But we don't really need a media respin to fix that, as it's an upgrade issue and upgrades use the repos 99.99% of the time. We could ship the 20171105.0 Server images and fix the upgrade issue in the repos, or we could even choose to just document it on CommonBugs as there is a relatively easy workaround.

That seems like it might be worth a Go/No-Go meeting to discuss, but I think I'm happy with that suggestion.

"This is exactly what I was proposing above."

Sorry, I short cut to "add new comment" after reading matt's proposal :)

Yeah, it may be worth holding a special go/no-go meeting to make sure we don't miss anything.

At today's Server SIG meeting, we made the following decisions:

• Server SIG does not want to ship a formal GA of Fedora 27 Modular Server
• Since the GA RC compose passed the Server tests, Server WG would like to just release the Server Edition content from that compose as the official Fedora 27 Server Edition

and the following recommendations:

• The Server WG recommendation is not to bother with a Boltron 2 release, but this is ultimately up to the Council and the Modularity WG to decide.

The full meeting minutes can be viewed here.

@pbrobinson koji is a problem because it makes available into buildroot only one (latest built) version of package. And doesn't really expect to work with multiple versions of same package.

koji can actually build custom repos with any list of NVRs in it you like via API, the default is to build the latest one. The koji-shadow which use to be used to build secondary architectures used this exact functionality to build the NVR with the exact same packages.

C1. We know we’ll be changing the underlying architecture and repo layout “significantly” for F28.
Eeep. This is scary. We definitely need system upgrades to work once we've released, and going from regular F26 to modular F28 is going to be a big enough change that I don't want also worry simultaneously about F27->F28.

C2. Fedora, as a brand, might be impacted.
Yeah. This too leans towards presenting this as After Boltron: The Sequel.

C3. The packager experience will change from F27 to F28.
This might be my biggest worry. We really need packagers to feel comfortable and on board and we don't have a lot of trust to burn in this area.

C4. Doing updates for F27 will take resources from F28 work needed to change everything.
But, we need to get updates smoothed out for F28 anyway, so I think going through the process for one cycle is actually a good thing.

C5. It is missing an enormous amount of software desired by server admins.
This is my alternate contender for biggest worry. We're not likely to get admins to give this a second look if the first experience is bad.

C6. Workload and extra requirements on rel-eng, infrastructure, QA, Alt-Arches and other teams. This particularly visible in the F-27 cycle where already extremely loaded people were getting more pressure to review/deliver stuff on top of all their other responsibilities. This was often without notice or planning.
C7. Infrastructure requirements.
C8. Impact on other parts of the project and perceived priotities
C9. Community and other contributors. I spoke to numerous companies and contributors in the Alt Architecture community, where in a lot of cases Server is a key or their only consumable (see Power64/s390x as an example, Aarch64 to a lesser degree). This left confusion and in some cases I believe we've lots contributors and users over this confusion.

In today's council meeting, it was agreed that:

Fedora will release the GA spin of F27 server as the F27 server release; give modularity WG permission to continue to release boltron derivatives for demo/testing as needed

This ticket is being closed. If there are server challenges, Server WG will open a new ticket.<zodbot> Meeting ended Wed Nov 29 15:04:32 2017 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot .
<zodbot> Minutes: https://meetbot.fedoraproject.org/fedora-meeting/2017-11-29/council.2017-11-29-14.00.html
<zodbot> Minutes (text): https://meetbot.fedoraproject.org/fedora-meeting/2017-11-29/council.2017-11-29-14.00.txt
<zodbot> Log: https://meetbot.fedoraproject.org/fedora-meeting/2017-11-29/council.2017-11-29-14.00.log.html