#244 šŸ“šŸ†• docs(project): Initial commit, "Upstream First" (closes #84)
Merged 2 months ago by jflory7. Opened 4 months ago by jflory7.
Fedora-Council/ jflory7/council-docs add/upstream-first  into  main

@@ -1,4 +1,5 @@ 

  * xref:index.adoc[Mission and Foundations]

+ * xref:upstream-first.adoc[Upstream First]

  * xref:leadership.adoc[Leadership]

  * xref:orgchart.adoc[High-Level Organization]

  * xref:brand.adoc[Brand]

@@ -0,0 +1,119 @@ 

+ = Upstream First: A Core Fedora Principle

+ 

+ The concept of "upstream first" is a fundamental principle within the Fedora Project.

+ It shapes our history, culture, and approach to contributing to the open source ecosystem.

+ Understanding this principle is crucial for anyone looking to contribute to Fedora and the broader Linux distribution ecosystem.

+ 

+ 

+ [[upstream-why]]

+ == Fedora's Commitment to Upstream First

+ 

+ Fedora, as a Linux distribution, plays a unique role as an _integrator_ of countless software components.

+ While Fedora develops some of its own software, its primary function is to package and deliver a cohesive operating system experience built upon the work of numerous upstream projects.

+ 

+ From its inception, Fedora has championed the principle of _upstream first_.

+ This isn't a policy or hard rule; it's a core value woven into the fabric of the Fedora community.

+ Fedora contributors believe that changes and improvements to open source software should, whenever possible, be shared back with the upstream projects.

+ This ensures that *all* users of that software, not just Fedora users, can benefit.

+ Ultimately, Fedora's final objective is the benefit of the user, and Fedora contributors will do what they think is best for the user, even if upstream disagrees.

+ 

+ Upstream first is also a pragmatic engineering principle.

+ By contributing changes upstream, Fedora reduces the long-term maintenance burden of carrying downstream-specific patches.

+ Maintaining external patch sets can become increasingly difficult as upstream projects evolve.

+ Embracing upstream first helps ensure the sustainability of Fedora and its contributions.

+ 

+ This approach has a ripple effect throughout the open source ecosystem.

+ Changes landed in Fedora often impact numerous downstream projects that use Fedora as a foundation.

+ Therefore, contributing to Fedora is a highly effective way to influence and improve the broader open source landscape, particularly within the RPM/Enterprise Linux ecosystem.

+ 

+ By prioritizing upstream contributions, Fedora aligns with its vision of a world where everyone benefits from free and open source software built by inclusive and welcoming communities.

+ This commitment extends to *all* open source software, not just Fedora itself.

+ 

+ 

+ [[upstream-downstream]]

+ == Understanding Upstream and Downstream

+ 

+ In the world of open source, projects often have interconnected relationships described as "upstream" and "downstream."

+ The _upstream_ project is the original source of the software—the foundation upon which other projects are built.

+ _Downstream_ projects, in turn, are those that utilize and often modify the upstream software.

+ Think of it like a river: the upstream is the source, and downstream projects are further along the flow, receiving and potentially altering the water.

+ 

+ This metaphor is essential for understanding how open source projects depend on and interact with each other.

+ Different development models encourage varying types of upstream/downstream relationships.

+ 

+ [[upstream-communication]]

+ === Open Communication with Upstream

+ 

+ Fedora recognizes the importance of clear and open communication with upstream projects.

+ We believe in fostering strong relationships with upstream developers and communities, and actively seek their input and feedback.

+ 

+ Fedora is always open to hearing from upstream projects about how we can improve our collaboration and integration processes.

+ We understand that Fedora's downstream usage can sometimes create challenges or friction for upstream projects.

+ We encourage upstream maintainers to reach out to us if they encounter any issues or have suggestions for improvement.

+ 

+ Our goal is to work together constructively to find solutions that benefit both Fedora and the upstream projects we rely on.

+ While we cannot always accommodate every upstream request, we are committed to listening, learning, and adapting our practices to minimize any negative impact on upstream communities.

+ 

+ The philosophy of working upstream first doesn't end with just development.

+ It also encompasses a productive, positive, and respectful relationship with our upstream projects.

+ Our communities overlap and we want to extend our Fedora values to their project as much as we would within Fedora.

+ To that end we always want to deal with challenges between projects together and have clear lines of communication.

+ 

+ [[upstream-exceptions]]

+ === When Downstream Changes Happen

+ 

+ While Fedora prioritizes upstream contributions, there are situations where downstream-specific changes are necessary.

+ These exceptions are not contradictions of the upstream-first principle, but rather acknowledgements of the complex realities of software development and distribution.

+ 

+ Reasons for downstream patches include:

+ 

+ * *Upstream Rejection*:

+     Sometimes, upstream maintainers may reject a patch for various reasons, even if it's beneficial to Fedora.

+     Fedora may still need to carry that patch to address a specific issue or requirement.

+ * *Upstream Progress*:

+     Upstream projects may move forward with new features or changes that require significant adaptation in Fedora.

+     Fedora may need to backport fixes or implement temporary workarounds while the downstream adaptation is completed.

+ * *Distribution-Specific Needs*:

+     Fedora, and its downstream distributions like EPEL, may have unique requirements or constraints that necessitate downstream modifications.

+     These needs might relate to specific hardware support, security considerations, or integration with other Fedora components.

+ * *Non-Free Blobs*:

+     Fedora is committed to promoting free and open source software and building everything from source.

+     Sometimes, upstream projects include non-free or pre-built binary blobs that Fedora needs to patch out to adhere to its principles.

+     While Fedora may discuss potential fixes with upstream, these patches might not always be accepted if there are no suitable alternatives or if they remove functionality.

+ 

+ In these situations, Fedora strives to minimize the scope and duration of downstream patches, and continues to work towards upstreaming changes whenever feasible.

+ Understanding the reasons for downstream changes is essential for maintaining transparency and trust within the community.

+ 

+ 

+ [[examples]]

+ == Examples in Action

+ 

+ The principle of "upstream first" manifests in various ways.

+ Here are a couple of examples:

+ 

+ * *Packaging Improvements*:

+     A Fedora packager identifies a bug or missing feature in a build toolchain.

+     Instead of creating a Fedora-specific patch, they submit a patch upstream to the toolchain's maintainers.

+     After review and discussion, the patch is merged upstream, benefiting all users of the toolchain and eliminating the need for a downstream Fedora patch.

+ * *Community Script*:

+     A Fedora contributor develops a script for analyzing package data.

+     They share the script publicly.

+     Another contributor enhances the script with new features and submits a pull request.

+     The original contributor merges the changes, making the improved script available to the entire community.

+ * *License Clarifications*:

+     A Fedora packager discovers licensing issues with an open source project, such as unclear or non-compliant licenses for included assets.

+     Instead of simply excluding the project from Fedora, they work with the upstream developers to clarify or correct the licenses.

+     This ensures that the project can be included in Fedora and benefits the broader open source community by promoting license compliance.

+ * *Avoiding Bundled Dependencies*:

+     A Fedora packager notices that an upstream project bundles a specific version of a dependency.

+     Instead of using the bundled dependency, they repackage the project to use the system-wide version of the dependency.

+     This ensures consistency across Fedora packages, enables rapid security patch deployment, and maintains compatibility between interdependent packages.

+ * *Early Testing and Bug Reporting*:

+     Fedora often pioneers the integration of new software versions and libraries.

+     This early adoption allows Fedora contributors to identify and report build or compatibility bugs upstream, particularly in the Python ecosystem.

+     These contributions benefit the entire open source community by ensuring smoother transitions and upgrades for everyone.

+ 

+ These examples illustrate how upstream first fosters collaboration, shared ownership, and continuous improvement within the open source ecosystem.

+ We encourage you to think about how you can apply the "Upstream First" principle in your Fedora contributions.

+ Have a story about an upstream contribution?

+ Share it with the community!

This commit adds a new page that offers an explanation and overview of Fedora's place in the Fedora ecosystem. This is inspired by a request that @ankursinha asked to the Council in August 2020, and recent events with the Flatpak/OBS situation.

The goal of creating this page is an attempt at making an implicit part of our community ethos into something explicit. If we do not write it down and make sure we are clear about what we believe, how can we expect that the next generation will be able to steward the project in accordance with our values and beliefs in the betterment of Free Software? This explanation may not be perfect, but it is a first start, and it is placed prominently in our official project documentation.

Closes Fedora-Council/council-docs#84.

Draft document of a Fedora Project docs page titled "Upstream First: A Core Fedora Principle". The page describes the concept of "upstream first" in open source development, Fedora's commitment to it, its benefits, and examples. It features a sidebar navigation menu and Fedora project branding.

Metadata Update from @jflory7:
- Pull-request tagged with: needs feedback, type - new docs

4 months ago

rebased onto bb382f5

4 months ago

I think this is a well-written and thoughtful explanation of "upstream first." It effectively captures the philosophy and reasoning behind Fedora's approach while balancing the broader implications. It aligns well with my understanding of the principle and why Fedora prioritizes it. Kudos.

Adding to and maintaining what Upstream First means over time (i.e., how it's implemented) is critical to keeping the ethos alive.

I like it,
It is a great way to ensure changes are made available for everybody.

This is a really good explanation of the "first" aspect. I'd be interested in seeing more explanation on what's "second". Sometimes Fedora does carry downstream patches. Maybe upstream rejected something that's important to Fedora. Maybe upstream has moved on and Fedora has backported a fix. When people see these patches and question the upstream first principle, it would be good to be able to point them to an explanation of of why sometimes downstream happens.

This is a really good explanation of the "first" aspect. I'd be interested in seeing more explanation on what's "second". Sometimes Fedora does carry downstream patches. Maybe upstream rejected something that's important to Fedora. Maybe upstream has moved on and Fedora has backported a fix. When people see these patches and question the upstream first principle, it would be good to be able to point them to an explanation of of why sometimes downstream happens.

That is a really excellent point Shaun--particularly for Fedora's downstream distributions (and projects like EPEL) which may have longer lifetimes than a given Fedora release but are "inheriting" the relationship with that upstream. In a way, it is what helps define the upstream/downstream relationship from the distribution perspective, which is distinct from that of the relationship between Fedora and it's upstreams.

I really like this wording and it's a big +1 from me with one nit which may be me over analyzing:

This commitment to upstream first distinguishes Fedora from some other distributions.

I read this in somewhat of an exclusive way, but not exclusive enough for it to be exclusive. IE: "We have this commitment and some others do while others don't". For this one line I'd recommend clarifying or removing ... but again, this is a nit. Well done :thumbsup:

This is a nice piece, I find it philosophically engaging. There is an added layer that you touch on, but don't lean into, that I would suggest putting more weight on: Upstream first is a pragmatic engineering principle, in that it reduces or eliminates the need to maintain an external patch set. In the early days of Linux distributions we saw many companies try to differentiate themselves by carrying special patches, but in the end the management of these patches as upstream moved in new directions added maintenance weigh, ultimately sinking the distributions. If you're in it for the long-term, you have to be upstream first, it's the only way to sustain engineering.

This is great, Justin! I hate to add a new left-menu nav item — there are too many already! — but won't block on that. There is probably a bigger reorganization which could fix that.

Other examples of upstreaming: license clarifications and fixes -- e.g. https://pagure.io/fedora-workstation/issue/463#comment-953648

Metadata Update from @jflory7:
- Pull-request untagged with: needs feedback
- Pull-request tagged with: needs changes
- Request assigned

4 months ago

rebased onto d0414bd

4 months ago

rebased onto d0414bd

4 months ago

I addressed all the above feedback points in commit e7c8fa9:

  • Added Section on Downstream Patches: Addressed @shaunm's excellent point about explaining the "second" part of "upstream first" by adding a new section "When Downstream Changes Happen." This section explains the legitimate reasons for downstream patches and emphasizes that they are not contradictions of the core principle.

  • Clarified "Distinguishes Fedora": Removed potentially exclusive phrasing as per @smilner's suggestion. The focus is now on the practice of Upstream First, not comparisons.

  • Emphasized Pragmatic Benefits: Incorporated @blc's point about the practical engineering advantages of upstream first, specifically the reduction of maintenance burden. This adds another layer of justification for the principle.

  • Added License Clarifications Example: This example summarizes @mattdm's point about the importance of upstreaming license-related fixes. It highlights how Fedora contributors actively engage with upstream projects to address licensing issues, ensuring compliance and broader inclusion in the open source ecosystem.

  • Minor Edits: Minor edits for clarity and flow.

Should we actually use the phrase "upstream first"? That's also probably worth a discussion in itself. We didn't use that phrase in the packaging guidelines because it can be particularly ambiguous and doesn't really convey the right meaning.

@ngompa I would offer that the purpose of this new page and Pull Request is to make this ambiguous phrase into something less ambiguous. :wink: I think the phrase "upstream first" is what is historically recognized by Fedora contributors, and Fedora contributors who read this page should not be surprised by anything they read.

As I said earlier in #commops:fedoraproject.org, my goal is for this to be as unshocking as possible. The reaction I hope to solicit is, "Duh!"

1 new commit added

  • šŸ“ docs(project): Add Upstream Comms section for Upstream First
4 months ago

Added commit 73112bf:

This commit addresses feedback from @kevin in Matrix about how we engage and work together with upstream communities. It emphasizes that while we are not always going to agree with upstream's suggestions, we work hard to maintain strong, healthy relationships with our upstreams, and Fedora values the importance of working together in what we do as a community.

Metadata Update from @jflory7:
- Pull-request untagged with: needs changes

4 months ago

Another reason for downstream patching that IMO worth mentioning is that sometimes we need to patch out non-free or pre-built blobs. We are committed to promote free software and to build everything from sources, so while we can discuss with upstream about fixing that upstream, such patches may not be applied upstream because there are no alternatives or may strip out some features.

Also, not sure it worth mentioning here, the concept of avoid bundling dependencies ensures rapid security patches deployment and to maintain compatibility between interdependent packages, rather than relying on specific bundled versions.

1 new commit added

  • šŸ“ docs(project): Add context for nonfree blobs and bundled dependencies
4 months ago

@mattia Great feedback. I addressed these points in commit 013a145 just now:

  • Added "Non-Free Blobs" to Downstream Reasons: Incorporated the point about patching out non-free or pre-built blobs as a reason for downstream patches in the "When Downstream Changes Happen" section.
  • Added "Avoiding Bundled Dependencies" Example: Included an example in the "Examples in Action" section illustrating how Fedora avoids bundling dependencies to ensure consistency, security, and compatibility.

This page does an excellent job of explaining Fedora’s "Upstream First" principle in a way that is both clear and engaging. It provides a solid foundation for understanding upstream and downstream relationships, making it accessible for both newcomers and experienced contributors. The emphasis on sustainability, reducing maintenance burdens, and fostering a strong open-source ecosystem really reinforces Fedora’s long-standing commitment to collaboration.

The section on "When Downstream Changes Happen" is particularly well done—it acknowledges that while Fedora prioritizes upstream contributions, there are valid scenarios where downstream patches are necessary. This balanced approach helps set expectations realistically. The "Examples in Action" section is also a great touch, as it translates theory into real-world cases that people can relate to.

A few suggestions to consider:

  1. Make key concepts pop – Bold key terms like "upstream," "downstream," and "Upstream First" when they are first introduced to make them stand out. This will help readers quickly grasp the core ideas.

  2. Clarify Fedora’s role – The document does a great job explaining the philosophy, but adding a small section about how Fedora practically enforces "Upstream First" in daily operations (e.g., how it interacts with upstream projects or encourages contributions) could make it even stronger.

  3. Expand on challenges – While the section on "When Downstream Changes Happen" covers why downstream patches exist, it might be helpful to briefly touch on some of the challenges Fedora has faced with this principle in the past and how they were resolved.

  4. More engagement at the end – The closing section is good, but it might be more effective if it directly encourages contributors to apply this philosophy in their work. A simple tweak like:

"We encourage you to think about how you can apply the ā€˜Upstream First’ principle in your Fedora contributions. Have a story about an upstream contribution? Share it with the community!"

This would make the document feel more inviting and interactive.

Overall, this is a well-written and much-needed document that successfully captures Fedora’s commitment to Free Software and collaboration. A few small tweaks could make it even more engaging, but as it stands, it’s already a fantastic addition to Fedora’s documentation. Great work! šŸš€

This is a really good explanation of the "first" aspect. I'd be interested in seeing more explanation on what's "second". Sometimes Fedora does carry downstream patches. Maybe upstream rejected something that's important to Fedora. Maybe upstream has moved on and Fedora has backported a fix. When people see these patches and question the upstream first principle, it would be good to be able to point them to an explanation of of why sometimes downstream happens.
(@shaunm)

According to the packaging guidelines, all patches "SHOULD have a comment above them about their upstream status", is that what you mean? (But maybe we should break this discussion out to another place to keep a better overview in this merge request.)

There is also the Staying Close to Upstream Projects page in the packaging guidelines which touches similar points as this new (and very good) proposal. I feel like there should be links between those two documents.

Another example for benefits to the whole ecosystem: Fedora is often the first trying to build Python code with the newest software, I've seen quite a few build bugs reported and fixed upstream by Fedora contributors whenever a new Python version or important lib had some breaking changes. I think the same goes for other software stacks. This may not be relevant for the big players like bottles or obs which may have sparked the creation of this MR but for a lot of smaller or medium packages which are not tested and developed on a daily basis it is a relevant contribution. I'd argue it is even more economical if one fedora contributor fixes similar bugs in 10 packages instead of 10 developers of one package understanding and finding a fix for their project.

@jnsamyak wrote…
Overall, this is a well-written and much-needed document that successfully captures Fedora’s commitment to Free Software and collaboration. A few small tweaks could make it even more engaging, but as it stands, it’s already a fantastic addition to Fedora’s documentation. Great work! šŸš€

Thanks for the positive feedback.

@jnsamyak wrote…
Clarify Fedora’s role – The document does a great job explaining the philosophy, but adding a small section about how Fedora practically enforces "Upstream First" in daily operations (e.g., how it interacts with upstream projects or encourages contributions) could make it even stronger.

In my eyes, this was sufficiently covered in the "Examples in Action" section. My goal there was to provide some concrete examples of how we interact with upstreams. If you have ideas for additions there, I am happy to hear them!

@jnsamyak wrote…
Expand on challenges – While the section on "When Downstream Changes Happen" covers why downstream patches exist, it might be helpful to briefly touch on some of the challenges Fedora has faced with this principle in the past and how they were resolved.

I agree this is a good point, although I do not have any immediate example on-hand from Fedora history that could demonstrate this. Of course, this could always be added in a future revision.

@jnsamyak wrote…
More engagement at the end – The closing section is good, but it might be more effective if it directly encourages contributors to apply this philosophy in their work.

Good feedback. I will make this change.

@dreua wrote…
According to the packaging guidelines, all patches "SHOULD have a comment above them about their upstream status", is that what you mean? (But maybe we should break this discussion out to another place to keep a better overview in this merge request.)

IMHO, this kind of guidelines should fit into the Packaging Guidelines and not into this more philosophical document.

@dreua wrote…
There is also the Staying Close to Upstream Projects page in the packaging guidelines which touches similar points as this new (and very good) proposal. I feel like there should be links between those two documents.

This doc has come up a few times. Some information is duplicated, but for the most part, they are different. Once this page is published, I would be happy to open a Pull Request on that document that cross-references to this new one.

@dreua wrote…
Another example for benefits to the whole ecosystem:

The example with the Python ecosystem is an especially good one, and we have done close work with the Python upstream too. I will work on incorporating this feedback into the draft.

This may be a place where my ignorance on the OBS and Bottle situation is showing, but if one of the goals of this doc is to address the issues those projects have with Fedora, I don't see how that's being addressed. Isn't the issue that neither of those projects want Fedora to be packaging their applications because it causes confusion from a support perspective?

To that end I don't know how the upstream first value applies in that situation, or how you would write about it in a vague way so that it does not forever seem like this doc exists only to address this issue.

Maybe you can say something like:

The philosophy of working upstream first doesn't end with just development. It also encompasses a productive, positive, and respectful relationship with our upstream projects. Our communities overlap and we want to extend our Fedora values to their project as much as we would within Fedora. To that end we always want to deal with challenges between projects together and have clear lines of communication.

I don't want to make it sound too much like its a PR thing, but maybe that helps as a starting point?

I believe this document is meant as a very general guideline / philosophy, not as means to solve any specific situation. Sorry if my message brought confusion here, I probably shouldn't have mentioned obs/bottles specifically.

I think is also important to note that we mention the "Staying close to upstream project" in the onboarding guide to new packagers of new packages

Maybe, this new document should also be commented there.

1 new commit added

  • šŸ“ docs(project): Voluntary nature, early testing, positive relations
4 months ago

@jnsamyak wrote…
More engagement at the end – The closing section is good, but it might be more effective if it directly encourages contributors to apply this philosophy in their work.

@dreua wrote…
Another example for benefits to the whole ecosystem:

@joseph wrote…
To that end I don't know how the upstream first value applies in that situation, or how you would write about it in a vague way so that it does not forever seem like this doc exists only to address this issue. Maybe you can say something like:

Commit e992119 addresses the above feedback.

@dreua wrote…
There is also the Staying Close to Upstream Projects page in the packaging guidelines which touches similar points as this new (and very good) proposal. I feel like there should be links between those two documents.

@x3mboy wrote…
I think is also important to note that we mention the "Staying close to upstream project" in the onboarding guide to new packagers of new packages. Maybe, this new document should also be commented there.

I agree. Once this new page is published, I will volunteer to make a follow-up Pull Request on the other page to cross-link this new page about Upstream First.

I'm missing an actual explanation what "upstream first" means. There are examples and reasons, but no actual definition.

I'd also like to see a clear statement that fedora's final objective is the benefit of the user, and we'll do what we think is best for the user, even if upstream disagrees.

LGTM, in general it provides a good view of "what to expect". This is also what I understood of the principle.

Justin, the content is good, but I think some important stuff is a bit too deep currently. I'll suggest a bit of trimming and reorg:

  1. Move upstream-why to the top
  2. Cut upstream-benefits (it's covered in upstream-why)
  3. Cut first paragraph, it's covered by upstream-why

1 new commit added

  • šŸ“ docs(project): Reorder sections, clarify user focus in Upstream First
2 months ago

See commit d1719d2


šŸ“ docs(project): Reorder sections, clarify user focus in Upstream First

This commit refactors the "Upstream First: A Core Fedora Principle" document to improve clarity and flow, and incorporates feedback from the community.

Changes include:

  • Move the "Fedora's Commitment to Upstream First" section to the top to emphasize the core principle and its rationale.
  • Integrate the key points from the "The Benefits of Upstream First" section into the "Fedora's Commitment to Upstream First" section, removing redundancy and streamlining the document.
  • Add a statement emphasizing Fedora's commitment to prioritizing user benefit, even if it means diverging from upstream decisions.
  • Minor edits for improved readability and consistency.

We have been collecting feedback on this document since February 15th and it has gone through several revisions of feedback. Thank you to all the community members who have helped contribute to and influence this document! I think we are in a good place to merge this and publish it officially. I want y'all to know that I really appreciate the thoughtful feedback and what we were able to define together!

Let's see how long it goes before we need to update it again. :wink:

Merging! :ocean:

Pull-Request has been merged by jflory7

2 months ago