#1878 Please change "Everything" directory to something less inaccurately comprehensive

Created 4 months ago by mattdm
Modified 3 months ago

The Fedora mirror tree has a directory structure like this:

AtomicHost/  
AtomicWorkstation/  
Cloud/  
Container/  
Everything/  
Modular/  
Server/  
Spins/  
Workstation/

It used to be that the directories other than Everything contained subsets of Everything, but that's increasingly not the case — notably, there's content in Modular, and then, also, the Atomic things are different. And we have things like Copr, which provides Fedora Project bits, and for that matter there's also EPEL.

This isn't just aesthetic, or "merely" confusing. It's a barrier to thinking about things that aren't part of the traditional Fedora package collection as Real Fedora.

I suggest "Collection" as a possible alternative, but I'm open to anything that allows the existence of Everything Else.

"Collection" or "Components" or "Content" all all fine with me. +1 this request

IMHO the new name should described it's content better than "something" and and "Collection", "Components" and "Content" do not really add anything to understand what to expect there for me. It just leads to the question "Collection of what?". My initial idea would be to use something like "Tradtional" or "Classic", which at least suggests that it contains the bits about how we used to do things. But this misses the information that stuff from there is also included in other artifacts. This lead me to realize that in the past Fedora was basically "Everything" and then we added new things with new names but we newer got a name for the stuff that used to be Fedora. IMHO once we have this, it should make it easier to name the directory. For example if we would call it "Fedora Classic", it could also be "ClassicEverything" and it would also make sense to rename "Spins" to "ClassicSpins" and "Workstation" to "ClassicWorkstation" and so on.

I guess the new name for old stuff needs to be sanctioned by the Council and then the technical details about what needs to be changed is part of rel-eng.

I guess the new name for old stuff needs to be sanctioned by the Council and then the technical details about what needs to be changed is part of rel-eng.

That logic doesn't make sense to me. It's not "old stuff". The packages in Everything are used to build the rest of the Editions and spins. The addition of content in Fedora that isn't from Everything is relatively new. It certainly doesn't make the packages in Everything "old" or "classic". Both of those have a negative connotation that they are going away or somehow obsolete. That's not the case. The packages there will still be used as the foundation for almost everything else.

Perhaps "Foundation" or "Cornerstone" would work?

This lead me to realize that in the past Fedora was basically "Everything" and then we added new things with new names but we newer got a name for the stuff that used to be Fedora. IMHO once we have this, it should make it easier to name the directory.

Yes! The question is how do we call this "old" (in the sense of "having existed for a long time", not "tired" or "outdated") Fedora. I tried to think what I would say if somebody asked me what flavour I have installed on my laptop, and "traditional" is the word that comes to mind.

I'd vote for "traditional", even though it does't convey that other directories contain subsets of this. "Foundation" is also good.

Maybe we should collect idea here and open this up to a poll and just use whatever name people like the most?

"Collection" is still my preferred bikeshed color, but I'm open to other things.

I would suggest that "Foundation" is confusing simply because of Fedora's lack of a "Fedora Foundation" in our legal structure. But it would be less confusing than "Everything".

I would suggest that "Foundation" is confusing simply because of Fedora's lack of a "Fedora Foundation" in our legal structure. But it would be less confusing than "Everything".

Not to put too fine a point on it, but being worried about confusion with a non-existent thing seems... odd.

just to note here, this is not a trivial change, be aware we will need to change:

  • pungi needs code changes
  • mirrormanager will need changes
  • sync scripts in infra
  • websites
  • probibly docs

and probibly other places I am not sure of.

Are you saying that it's not worth the trouble unless we have a clearly superior name?

Maybe we should consider just adding a README file that says explains the historical naming. httpd listings will put the contents of README file in the directory listing automatically, so anybody browsing through will see it.

The mirrormanager change is literally s/Everything/Newthing/ — it's a change, but not a particularly large amount of work.

I don't know what's needed in pungi. I assume infra sync script changes are similar to mirrormanager.

I can handle docs and website changes.

Overall, sure, it's not trivial, but it's also not, like, at the level of a project. Let's not get crushed under the weight of all the stuff we do for legacy reasons.

Not to put too fine a point on it, but being worried about confusion with a non-existent thing seems... odd.

The concern is that people looking for such a thing will find this in search results, because now it would be an existent thing — just not one that they're looking for.

I would suggest "Base", except that seems kind of disingenuous when it's twenty thousand packages, many of them leaf applications.

4 months ago

Metadata Update from @zbyszek:
- Issue tagged with: meeting

I guess the new name for old stuff needs to be sanctioned by the Council and then the technical details about what needs to be changed is part of rel-eng.

That logic doesn't make sense to me. It's not "old stuff". The packages in Everything are used to build the rest of the Editions and spins. The addition of content in Fedora that isn't from Everything is relatively new. It certainly doesn't make the packages in Everything "old" or "classic". Both of those have a negative connotation that they are going away or somehow obsolete. That's not the case. The packages there will still be used as the foundation for almost everything else.

So let's rename it to AlmostEverything ;-) - I did not mean to use "old" as being worse but compared to Modules and OSTree (which is the new stuff), it is the old stuff. Also it is not the content that is described here but the method the content is prepared. So if we do not want a name based on the time it was invented, maybe we can fit the method in the name, such as "BaseRPMs" or "RPMCollection".

This was discussed in the FESCo meeting today:
- We'll return to this after F28 is released

4 months ago

Metadata Update from @zbyszek:
- Issue untagged with: meeting

I suggest that "Default" be used as the name. it is the default set of content you have available from a fedora install. It is quite a bit of work in the background, but mostly because there is likely places that have been forgotten have baked in information. on the releng side pungi should not need any code changes, the pungi configs will need changing and the syncing will need changing. while the syncying has been kept as simple as possible, it is a little bit fragile.

I can confirm that Pungi does not care about the name. Only configuration would need be updated, but code can remain as is.

fedora-repos packages needs to be updated too.

I prefer not "default", because a lot of the stuff in there isn't the default for anything, and many things in the future may have defaults which aren't there.

I prefer not "default", because a lot of the stuff in there isn't the default for anything, and many things in the future may have defaults which aren't there.

sure. it depends on the lens you use to look at it. What is the story you are trying to sell? without really knowing that it is hard to make ideal suggestions. I do not like Collection as they are all collections of things, I think we need to paint it with something more specific

Is it really worth bikeshedding this? The name has been around for a while and may be baked into lots of stuff. Well, I mean, I know it's baked into lots of stuff. The whole release validation process bakes these names in, for instance; 'Everything' becomes part of the 'flavor' identifier in openQA, so we'd have to change that, and so far as openQA was concerned, this would be a different image - it wouldn't in any way 'know' that the new flavor is really just a rename of the old flavor.

It's baked into fedfind as the 'generic' tree name in various functions (that are then used by fedfind consumers); I'd have to change all that again, in fact, I'd have to conditionalize it (more than it's conditionalized already) because fedfind works with old things too, so it would need to know that the 'generic' tree was called Everything from Fedora 21 to Fedora 29 (or whatever) but called something else after that...

It's in the release validation wiki pages, and this overlaps with the openQA stuff, because for reporting openQA results to the wiki we parse the image identifier out of the openQA 'flavor' value and use that to locate the row to edit in the result validation pages...

that's just stuff I can think up / find off the top of my head. These identifiers get built around, it's really inconvenient whenever people want to change them.

I think I agree with @adamwill here - changing this name will definitely break a lot of things and may not gain Fedora that much in the end. I don't think the risk/cost to reward ratio looks favourable.

Bodhi is also on the list of things that expects to see "Everything".

I think it's worth changing. If we avoided all changes because they touched lots of things or they took a lot of work, we'd never do anything.

As a mitigation, I suggest we phase this in. It shouldn't be difficult to create the new name and symlink Everything to it. Then we can migrate things without having a massive flag day where everything breaks.

We cannot use symlinks on the master mirror. All our mirrors don't support them.

We could do a hardlink of contents, but then if anyone pulled without hardlinks they would get a lot more content.

I think it's worth changing because it is so baked in to … everything. That means that the implication (that this is everything, or at least everything that's Real Fedora) gets spread to more places than just the mirror directory names.

@mattdm would Generic, Standard or Foundation work better than Default?

I don't think this is a trivial amount of work, but it does seem worthwhile from these perspectives: (1) not confusing users/least surprise (with novice eyes, I look at Everything in the current tree and expect everything else at the same level is included there too); (2) how we think about Fedora (a mountain of RPM packages is not only not everything, but over time it'll be increasingly less of everything). We should do a simple accounting of what the actual effort comprises. Then figure out whether/when/how to exert that effort and weigh against other priorities.

"It shouldn't be difficult to create the new name and symlink Everything to it."

Actually it is, because this name is not just a random directory name. It's the name of a variant in the Pungi configuration:

https://pagure.io/pungi-fedora/blob/master/f/variants-fedora.xml#_30

The only sensible way to change the name of the directory is to change the name of the variant, but that does much more than just change the name of the directory this ticket is talking about it. It becomes part of the compose metadata in lots of other places. It also becomes part of the filename of deliverables and stuff. Basically, in:

https://kojipkgs.fedoraproject.org/compose/branched/Fedora-28-20180424.n.0/compose/Everything/x86_64/iso/Fedora-Everything-netinst-x86_64-28-20180424.n.0.iso

That's not just a filename and a path which both have 'Everything' in it which were constructed by artisanal bash scripts or something, it all comes out of the Pungi config. You can see it in the images metadata. You can see it in the package metadata and the composeinfo metadata. It's not just a directory name you can change and create a symlink to and call it good.

edit: well, I mean you can, but the point is, it wouldn't do much good. All the tooling we're concerned about isn't just expecting to see a directory called Everything, so changing the name and creating a symlink wouldn't make it all keep working, because we wouldn't have 'symlinks' in the metadata or the image filenames or anything like that.

Edited 4 months ago by adamwill

I don't think the risk/cost to reward ratio looks favourable.

Yes. If we had a name that was obviously better, than the reward side would be bigger. But despite all the attempts, I'd say we don't, so that side is small. And the cost side is certainly non-negligible, and comes at a time when our infrastructure is undergoing changes related to modularity and new deliverables.

I think "Components" is good — it's stuff that can be used to build deliverables, but it doesn't imply that it's the only source of such components. It even works for the "generic rpm-based netinstall source" variant — that's a deliverable meant for people who want to build up something custom, and "Components" is as good as "Everything" for that.

On 04/24/2018 03:21 PM, Josh Boyer wrote:

I think it's worth changing. If we avoided all changes because they touched lots of things or they took a lot of work, we'd never do anything.

The argument isn't simply that they touch a lot of things or take a lot
of work, it's that they have those qualities and don't have a payoff
significant enough to justify them.

@bowlofeggs I'm reading back and I don't see any points arguing that there is not a significant "payoff".

@ausil "What's the story you're trying to sell" is a great question. It's:

One way to think of Fedora is "It's a bunch of rpms in a unified repo produced in a certain way through a rigorous process". But, that's very limiting and does not position us for future success. Instead, "Sure, that's an important part of what Fedora does, but it's not what Fedora is — it's not Everything Fedora at all. Fedora is a big project made up of lots of people doing lots of different things, and we have plenty of things going on right now that are bigger than that one area of focus, and we have plenty of room to do even more in the future."

@mattdm You are describing 'putting one's best foot forward' -- that is, building a 'Showcase' [to suggest a new term to describe 'Everything' better]

... not that I think this is needed, but ...

@mattdm OK, then here's one: how often is anyone even exposed to this name? It's not easily found on any of our public-facing pages, it's not on getfedora.org unless you click all the way through to alternative downloads, which is not very heavily highlighted at all.

If you're worried about the occurrence of the name there, why not just change the name there (or add some text disclaiming it) without actually changing the variant name?

Edited 4 months ago by adamwill

@adamwill Savvy users definitely see it in the directory structure, and when for example doing a network install. But the primary audience for the story isn't end users — it's:

  • potential new contributors and collaborators (who will come across the various things you've highlighted very quickly)
  • existing contributors, including all of us (what we call things has consequences for how we treat them, no matter how much text-disclaiming we do).

On 04/26/2018 10:00 AM, Matthew Miller wrote:

@bowlofeggs I'm reading back and I don't see any points arguing that there is not a significant "payoff".

I made the argument two days ago:

"changing this name will definitely break a lot of things and may not
gain Fedora that much in the end. I don't think the risk/cost to reward
ratio looks favourable."

I think the burden of proof rests on those who want to make the change
that there is significant gain.

@adamwill

why not just change the name there

This is asking for confusion under the principle of having a Single Point Of Truth. ISO images get passed around without or separated from external documentation.

If something is called an: Everything in the Image Label, and yet is really a: NewName on the website, doing 'half the job' and just updating a website just engenders confusion

'mkisofs' [genisoimage] has ways to reduce confusion

-appid
-biblio
-copyright
-publisher
-p 
-preparer
-V
-volset

provide any number of places to insert labeling --- and so to provide a future user encountering an image to figure out what in the world it is -- encouraging a discrepancy between a mutable web page, and a piece of media is not a good approach at all

This request is entirely about optics and future positioning. The payoff is being more flexible in how we think about things coming from Fedora. Can we do that without changing Everything? Yes. However it makes it harder to do so. People naturally remain ingrained in whatever mental map existed before the change if there are no external indicators otherwise.

If we don't want to make the changes because we think they are wrong, that's one thing. However, I don't really see anyone arguing against the reasons for the requests. I only see arguments against it because it's work. Put another way, if we were to take the state of the project today and try and accurately reflect what it produces, would we call it Everything? I don't think we would.

It's unnecessary work that takes away from working on other things.

I've got a backlog of requests for new openQA tests as long as your arm. I've got a project to try and unify CI messaging between Fedora and RH internal. I've got a dozen other important things to do. If this gets changed, I'm going to have to drop all of those for at least the time it takes to make all the necessary changes to fedfind to make it handle both the new and old names properly, or else various things in the release validation process which people expect to work will not work. And I'll probably wind up being pulled into helping to figure out all the inevitable unexpected consequences for the compose process too, because fixing issues in the compose process tends to fall on releng and QA together. All the time that's going on, I won't be doing more productive work on actually helping us test more stuff and so on. (I imagine the situation is exactly the same for releng: I know @dustymabe and @mohanboddu have a boatload of improvements they'd like to make to the release process and other things; all the time they spend implementing this rename and chasing its consequences through the compose scripts is time they're not spending on, say, actually improving the compose scripts or automating things which need automating, etc etc etc).

I am especially sensitive to this because I've already been through this circus more than once in recent releases. releng keeps inventing new composes and renaming them. We had that whole mess with the 'Modular' composes showing up, deciding to call what was basically their Rawhide something else ('Bikeshed') for no very good reason, then going away again. I lost like two months of time I could've spent on something more productive chasing after those changes. From my perspective, this apparent constant desire to rename things and create new ones with very little consideration of the amount of running-to-stand-still work it produces is getting extremely frustrating.

Edited 4 months ago by adamwill

On 04/26/2018 12:48 PM, Josh Boyer wrote:

However, I don't really see anyone arguing against the reasons for the requests. I only see arguments against it because it's work. Put another way, if we were to take the state of the project today and try and accurately reflect what it produces, would we call it Everything? I don't think we would.

I argue that it's work and that the impact of the name Everything is
negligible in comparison to the size of the effort. If we expend our
efforts on renaming Everything, that means we didn't do something else,
and I believe the list of other things we could have done instead that
would have a greater payout is lengthy (so, opportunity cost).

Randy, if it were negligible, I wouldn't have filed this ticket. Even though it may sound like fun, I don't actually go around just trying to make work for people for no reason. You keep stating that the benefits are low, but you're not actually engaging with the arguments for the importance that we're giving. Just repeating that you don't think much of it, without any substance, doesn't add anything.

Randy, if it were negligible, I wouldn't have filed this ticket. Even though it may sound like fun, I don't actually go around just trying to make work for people for no reason. You keep stating that the benefits are low, but you're not actually engaging with the arguments for the importance that we're giving. Just repeating that you don't think much of it, without any substance, doesn't add anything.

Right. Both of the recent replies boil down to "we have lots of other work already". Everyone on this ticket does. So let's think creatively if we can. Perhaps there is a group of people that would be willing to dedicate the time to make this change so work can progress in other areas?

@mattdm well, I mean, whether someone considers the benefits that have been outlined as substantial is fundamentally a subjective thing, isn't it? @bowlofeggs and I are saying that our opinion is they're not significant enough to be worth the effort required. I don't know what kind of "substance" you're expecting us to provide, extensive user surveys? But the onus to provide "substance" should be on those proposing change, really.

@jwboyer "Perhaps there is a group of people that would be willing to dedicate the time to make this change so work can progress in other areas?"

practically speaking, even if someone else does the work, it's going to require extensive handholding from those actually familiar with the processes that use variant/subvariant names to ensure they're changing all the necessary things and changing them in the right way. It'd probably be more work for QA and releng to review pull requests from folks unfamiliar with the processes than it would be just to make the changes themselves. No, there are not perfect test suites in place such that we can just merge pull requests without human review; this is the kind of thing it's very difficult to cover with test suites. Test suites have to make assumptions at some point, like..."the variant which has all the stuff in it is called Everything, not something else". That's an assumption a test suite would probably make. Not to mention that it's not really practical to build a test suite that covers the entire chain of producing composes and testing them. The only mechanism we have for testing that is "make the change, run a compose, see what blows up".

Edited 4 months ago by adamwill

On 04/26/2018 01:42 PM, Matthew Miller wrote:

You keep stating that the benefits are low, but you're not actually engaging with the arguments for the importance that we're giving. Just repeating that you don't think much of it, without any substance, doesn't add anything.

I can argue that your arguments are also not substantial and arrive at
the same conclusion. As I stated before, the burden of proof (i.e., the
burden to provide substance) fairly relies on those proposing the
change. It's my opinion that the arguments made for this change have not
been substantial in that no supporting evidence has been supplied.
Without evidence, I'm left with my own experiences as a Fedora
contributor, and I've not found the string "Everything" to be at all
bothersome or even prominent in any way. Since there is no evidence
provided by the proposal I have only my own experience to go on, which
is why I believe the benefit is negligible. The word "Everything" being
used in a path that I frankly never look at really has negligible impact
on my thoughts and attitudes about RPMs vs. containers vs. ostrees vs.
whatever else. Due to my experience, I have a hard time believing that
others' experience around this path that is rarely encountered is
significantly problematic. Sure, it's an opinion, but no facts have been
brought to the table to sway my opinion, thus so far it's just opinions
vs. opinions, anecdotes vs. anecdotes.

The fact we do know is that the effort to make the change is
substantial, but no data have been provided in support of the benefit
being at least approximately equally substantial. The burden of
providing those facts lies with those who want to make the change.
Without facts, we are left with opinions and I personally just don't see
the benefits justifying the costs (especially considering the
opportunity costs).

BTW, it's not that I don't want to do the work and am being lazy. I would personally not be very affected by this change.

Proposal:
Add a README file at the level where CloudImages/Docker/Everything/Server/Spins/Workstation/WorkstationOstree/COMPOSE_ID appears:

-----------8<----------------------------------------------------------------------------------------------
# README
Various subdirectories contain different variants of the Fedora distribution. They are all based on the same set of RPMs, but not always the same set. The way that packages are installed varies too.

CloudImages/ — File system images suitable for virtual machines.
Docker — File system images suitable for use as docker containers.
Everything — This is the most traditional way to install Fedora. This directory contains the full set of RPMs from everything else is built, and installation images suitable for use with USBs and CDs.
Server — This directory contains the set of RPMs used in Fedora Server, and installation images suitable for use with USBs and CDs.
Spins — This directory contains installation images suitable for use with USBs and CDs to install Fedora KDE, Fedora Xfce, Fedora Cinnamon, Fedora LXDE, Fedora LXQt, Fedora MATE, Fedora SoaS.
Workstation — This directory contains the set of RPMs used in Fedora Workstation, and installation images suitable for use with USBs and CDs.
WorkstationOstree —This directory contains installation images suitable for use with USBs and CDs to install Fedora Atomic Workstation
.composeinfo — This file contains information about the subdirectories in text format
COMPOSE_ID — This file contains a short identifier like "Fedora-27-20171105.0"
----------->8----------------------------------------------------------------------------------------------

(corrections are welcome, please drop a comment and I'll update the text in the comment.)

... and let's discuss this today.

4 months ago

Metadata Update from @zbyszek:
- Issue tagged with: meeting

It doesn't seem like you did, from the minutes...

This is on the agenda for Friday's FESCo meeting.

AGREED: FESCo does not feel that the benefits outweigh the implementation cost and recommends that nomenclature be changed in documentation first. (+7, 0, -0)

https://meetbot.fedoraproject.org/teams/fesco/fesco.2018-05-04-15.00.html

3 months ago

Metadata Update from @bowlofeggs:
- Issue untagged with: meeting
- Issue status updated to: Closed (was: Open)

AGREED: FESCo does not feel that the benefits outweigh the implementation cost and recommends that nomenclature be changed in documentation first. (+7, 0, -0)
https://meetbot.fedoraproject.org/teams/fesco/fesco.2018-05-04-15.00.html

For the record, I am -1 to this resolution. The official count is (+7, 0, -1).

For the record, my personal position is more nuanced than the FESCo decision - I do not oppose changing the Everything directory if the groups involved in doing the work decide to do so. What I am opposed to is FESCo mandating that those groups do said work.

Login to comment on this ticket.