#1321 Can packages not approved for Fedora be placed in non-official branches of the Fedora pkgs repo?
Closed None Opened 5 years ago by toshio.

= phenomenon =

Recently, spec files and content destined to be built only in COPRs has been checked into the Fedora Packages git repo. In the past the Fedora packages git repo was only for content destined for inclusion in Fedora (built via koji, etc). Other uses were told to be placed elsewhere. The content includes spec files which use SCLs which would need a separate package review as well.

We have talked about providing some of that "elsewhere" before (Providing another repo tree for packages under review and whether to provide git repos for people packaging for COPR) but nothing from those talks has ever made it to implementation.

I think we need to decide whether we want to mix these together or not. If we do want to mix them we'll have to solve a few technical issues (a plugin to prevent building from non-official branches in koji. Some means of marking which branches are official or not. Expand our notification (email aliases, pkgdb) to work with non-official branches. Be able to mark other package maintainers as being able to commit to a non-official branch even though they are not able to commit to any of the official Fedora branches.)


The title of this ticket is somewhat misleading. There is a common tendency to read "legal" into "not suitable for Fedora". The content in question (as all content intended for COPRs) is in compliance with the legal requirements for Fedora.

To avoid confusion, could you please re-title this to something more specific, such as "Can content not consistent with Fedora Packaging Guidelines be placed in non-official branches of the Fedora pkgs repo?" I believe that would remove the ambiguity and help focus the discussion.

Something to consider if drafting a policy response:

I use a non-official branch of gstreamer1-vaapi (the [http://pkgs.fedoraproject.org/cgit/gstreamer1-vaapi.git/commit/?h=prerelease prerelease] branch) to make it easy for me to produce SRPMs for my [http://copr.fedoraproject.org/coprs/farnz/gstreamer1-vaapi-prerelease/ gstreamer1-vaapi COPR]. These SRPMs will not go into Fedora proper, as they're random git checkouts at upstream's request.

There's nothing in there that won't end up in Fedora when upstream releases the next version of gstreamer1-vaapi (plus many more upstream patches), but I don't want to give upstream headaches with lots of different master-of-the-day packages in Fedora; it also gives anyone who's got watchcommits on gstreamer1-vaapi a chance to see how I intend to package the next release.

Any policy should consider this workflow, and either explicitly permit it, or explain what I should be doing instead.

I don’t think we ever had any prohibition against personal branches, work-in-progress branches (including branches that just don’t build), or the like. If we are mishandling this (e.g. allowing to build official builds from personal branches that can be deleted), we should fix that in any case.

Secondarily we can talk about standardizing the use of such personal/unofficial branches for other uses (e.g. consistent naming of COPR/SCL branches, or allowing use of pkgdb for ACL management), but strictly speaking ''Fedora proper'' can AFAICS do perfectly fine without this.

@famz <nod> The specific branch that brought this question up are:

  • a nightly snapshot
  • Being used to host code built only via COPRs.
  • diverge from the spec file in Fedora by adding SCL macros
  • the comaintainer's team says that the divergence (other than using upstream code that will eventually make it into Fedora packages) won't be merged back into Fedora.
  • SCL packages need to have (1) a SCL Request approved (similar to the Fedora Change Requests). (2) be reviewed and approved like any other package, neither of which has been done here.

So I could see drawing a line that allowed nightly snapshots such as you are doing but not the divergence. That the code is built only in COPR is probably a symptom of the divergence rather than the actual root of the problem.

Dear FESCo,

I was thinking about filling similar ticket if it's fine to use Fedora dist-git for Fedora related projects. Python guys has started working on prototype of nightly builds of set of packages. They'd like to run nightly rebuilds on Copr (koji doesn't have capacity nor features to support it). The Continuous Integration is a part of Env and Stacks Working Group PRD [1]. There are more sub-projects of Continuous Integration currently in prototype stage.

Accidentally the Continuous integration nightly builds are packaged in SCL, because how else could be set of Python packages installed into rawhide buildroot and don't conflict with old versions already included. The SCL macros are here only for nightly rebuilds. The nightly branch is here only for continuous integration work.

What I ask for is acknowledgement of personal branches for development, which many developers are already using. If it's not possible (why?), then at least approval of nightly branches as branches meant for continuous integration. Python guys can prepare Change for Fedora 22 if necessary.

I'd like to ask for your support on this case, because Env and Stack WG is struggling with finishing projects. Developers want to do useful stuff, so give them chance to do it.

FYI whole discussion with toshio [2].

[1] https://fedoraproject.org/wiki/Env_and_Stacks/Product_Requirements_Document#Continuous_integration_.28CI.29

[2] https://fedorahosted.org/rel-eng/ticket/5894#comment:38

Replying to [comment:5 toshio]:

@famz <nod> The specific branch that brought this question up are:

  • a nightly snapshot
  • Being used to host code built only via COPRs.

Not entirely true, the code is meant to be merged to rawhide branch and therefore is tightly related to what we do in Fedora. It is a "development branch for rawhide", so to say.

  • diverge from the spec file in Fedora by adding SCL macros

Eventually, we want to able to merge these branches into rawhide, thus they will stop diverging (except possibly changelog/release/version).

  • the comaintainer's team says that the divergence (other than using upstream code that will eventually make it into Fedora packages) won't be merged back into Fedora.

I said precisely the opposite. It will be merged back into Fedora.

  • SCL packages need to have (1) a SCL Request approved (similar to the Fedora Change Requests). (2) be reviewed and approved like any other package, neither of which has been done here.

I believe that you're talking about building SCLs in official Fedora Koji instance, which is not what we want to do. We want to build these packages as SCLs in Copr, merge them back to rawhide and build them only as standard system packages (meaning with deactivated SCL macros) in official Fedora Koji.

So I could see drawing a line that allowed nightly snapshots such as you are doing but not the divergence. That the code is built only in COPR is probably a symptom of the divergence rather than the actual root of the problem.

To put this into a perspective, please see [1], where I explain why we're doing this and what benefits it's supposed to have.

[1] https://fedorahosted.org/rel-eng/ticket/5894#comment:40

Replying to [comment:6 mmaslano]:

I was thinking about filling similar ticket if it's fine to use Fedora dist-git for Fedora related projects. Python guys has started working on prototype of nightly builds of set of packages. They'd like to run nightly rebuilds on Copr (koji doesn't have capacity nor features to support it). The Continuous Integration is a part of Env and Stacks Working Group PRD [1].

To be extra precise, this is really doing continuous integration ''of Python'', or at best ''of the python3 package''; not continuous integration ''of Fedora'', which would imply taking the ''exact'' Fedora or to-be-Fedora packaging of Python and thus having all of Fedora use it, not having a special SCL version of Python designed explicitly so that it is ''not used'' by the rest of Fedora (i.e. not “integrating” it with Fedora).

OTOH the motivation is IMHO not all that relevant to the principles of this decision.

@bkabrda: This is what you said: " It's true that this branch uses SCL macros, but if/when we merge this back to Fedora, we won't actually build as SCLs. "

As I said in comment:5 of this ticket, I can see drawing a line that places nightly snapshots on the allowed line. That should be the same package as is in Fedora. However, the SCLs are a different package. The changes are similar to what's in mingw packages. The unreviewed changes definitely don't belong in the git repo for the Fedora package. And the way that people are looking at this makes me more convinced than ever that the reviewed changes don't belong in the same git repo either.

I think pkgs dist-git is particularly ill suited currently for these uses:

  • Branches can not be removed by maintainers (it takes a scmadmin who has to check and make sure no official builds were ever made from the branch). So, it means even short lived stuff will likely sit around forever.

  • Notifications and permissions options are pretty narrow.

  • People look at that as whats being built (or possibly soon to be built as official fedora packages), not myraid other uses.

  • If/when we enable things like scl building in koji, people could start building non approved scls.

So, I can see several ways forward:

  • Enhance pkgs git. I'd love to do this at some point, but I don't think it's a priority and infrastructure has already tons to do. I could see something like a gitlab setup being very nice down the road, but will take a while to get there.

  • Could fedorapeople git repos be used for these use cases? They do require a bit of setup on the part of maintainers, but they can add anyone in cla_done+1 to commit, there's backups, copr could build from them, there's cgit, etc. https://fedoraproject.org/wiki/Infrastructure/fedorapeople.org?rd=Fedorapeople.org#fedora_people_git_hosting_support

  • If others could commit time and energy to driving setup of a seperate git setup for copr we could look at doing that. (possibly gitlab, or something like it).

  • Finally, perhaps github or other external services would be better for these uses?

Thoughts?

do not use pkgs dist-git for changes not directly going into official packages for now. Work with infrastructure to plan a git for these uses, use fedorapeople or external services for now. (+6,0,-2)

I'd be happy to talk with folks on how we can fix up dist-git or add a new service for these needs.

Replying to [comment:10 toshio]:

@bkabrda: This is what you said: " It's true that this branch uses SCL macros, but if/when we merge this back to Fedora, we won't actually build as SCLs. "

Which means that the specfiles in the python-nightly branch and rawhide will be completely same (except that the nightly branch will have updated patches and longer changelog _+ different VR), we will just not build in SCL-enabled buildroot, therefore the resulting binary RPMs will be standard system packages, not SCL.

As I said in comment:5 of this ticket, I can see drawing a line that places nightly snapshots on the allowed line. That should be the same package as is in Fedora. However, the SCLs are a different package. The changes are similar to what's in mingw packages. The unreviewed changes definitely don't belong in the git repo for the Fedora package. And the way that people are looking at this makes me more convinced than ever that the reviewed changes don't belong in the same git repo either.

First of all, what are unreviewed changes? If I make changes to specfile when updating the package, noone has to review these either. Could you please provide an argument why someone should review such changes or why such unreviewed changes don't belong in the git repo?

Adding SCL macros doesn't change the resulting binary RPM in official Fedora build, so I don't see where we would cause some damage with this approach.

Replying to [comment:13 kevin]:

do not use pkgs dist-git for changes not directly going into official packages for now. Work with infrastructure to plan a git for these uses, use fedorapeople or external services for now. (+6,0,-2)

I'd be happy to talk with folks on how we can fix up dist-git or add a new service for these needs.

From discussion around I understand it's needed to fix firstly: https://fedorahosted.org/rel-eng/ticket/5843 It means deleting private branches.

After that it might be possible to change setting of koji and branches - branch can be build only in defined target and srpm can be tagged only in defined version of Fedora.

Replying to [comment:14 bkabrda]:

First of all, what are unreviewed changes? If I make changes to specfile when updating the package, noone has to review these either. Could you please provide an argument why someone should review such changes or why such unreviewed changes don't belong in the git repo?

I think he was speaking of when scls are buildable/shippable in fedora. If you simply start building a python3 scl from the python3 git it would never have passed through package review.

I've moved my extra branches to GitHub, feel free to delete them from dist-git.

Never put anything from this to lookaside cache, no need to inspect that one.

I'm confused by what the decision means for my use case, and would appreciate someone taking my current workflow and writing up how I should do it in future.

Right now, I have a prerelease branch in dist-git. I never put sources in the lookaside cache; anywhere that has my package has the SRPM to get the sources from.

My commits to the prerelease branch are just changes to the spec file; I generate matching sources, which I keep locally, then, to build the RPM in COPR, I use fedpkg srpm to get an SRPM, and upload that to a web server for COPR to claim.

Do I just need to move the prerelease branch to fedorapeople git?

Replying to [comment:16 kevin]:

Replying to [comment:14 bkabrda]:

First of all, what are unreviewed changes? If I make changes to specfile when updating the package, noone has to review these either. Could you please provide an argument why someone should review such changes or why such unreviewed changes don't belong in the git repo?

I think he was speaking of when scls are buildable/shippable in fedora. If you simply start building a python3 scl from the python3 git it would never have passed through package review.

That's only true since the time FPC decided to prohibit SCL macros [1], which was after this FESCo decision had been made. Previously, SCL macros were allowed in Fedora, assuming that you didn't build actual SCL packages from them. So I still don't understand what kind of review Toshio was talking about. I'd really love to hear an explanation here.

[1] https://fedorahosted.org/rel-eng/ticket/5894#comment:55

FYI, toshio is out this week, so likely he will not have a chance to answer until next week.

I'll note that we did have:
https://fedoraproject.org/wiki/SoftwareCollections
before. Which, depending on how you read it could mean that scl macros were not acceptable for fedora packages.

Replying to [comment:18 farnz]:

I'm confused by what the decision means for my use case, and would appreciate someone taking my current workflow and writing up how I should do it in future.

Right now, I have a prerelease branch in dist-git. I never put sources in the lookaside cache; anywhere that has my package has the SRPM to get the sources from.

My commits to the prerelease branch are just changes to the spec file; I generate matching sources, which I keep locally, then, to build the RPM in COPR, I use fedpkg srpm to get an SRPM, and upload that to a web server for COPR to claim.

Do I just need to move the prerelease branch to fedorapeople git?
I would also like to hear something about this. Right now I keep my work on Github witch makes it very uneasy to keep it in sync with Fedora. This question is here for 4 weeks, still without any answer.

In other words: You've banned our workflow, what's next?

Thanks

Well, perhaps you could describe your former workflow vs the new one? I'm not sure what part is more difficult? Git has tons of ways to pull changes in... you could use submodules, gitremote, or just feed any dist git changes into your repo, no?

Would fedorapeople or fedorahosted put your mind more at ease over github?

I'm not sure. Would any of those solutions use ACLs from Fedora packages (+ provenpackagers)? Would any of those repos clone together with packages repos?

Of course, once I clone the package repo and add github/fedorapeople/fedorahosted as a secondary git remote, it works. However the initial procedure is not very straightforward.

I was just curious what is the recommended workflow right now, so it stays somehow predictable. I guess the general idea now is "keep it out of dist git and do it as you please", but that will lead to a mess when a lot of "future Fedora" content would be spread around the Internet. I found the previous workflow more friendly for possible contributors.

This won't change the vote, but I disagree with the decision that was made. The technical short-comings of pkgs.fp.o Kevin noted in comment #12 are valid but they don't outweigh the utility of keeping packaging related changes by the primary maintainers in one place. It's not that other services like fedorahosted/fedorapeople/github can't be used. It's that it makes everything harder on the packager.

Getting people to package things for Fedora is difficult enough with our guidelines and review process. Making it harder for them to efficiently store experimental or non-official changes is just discouraging. I don't think the reasons listed are large enough to prevent community members from storing such changes in separate branches.

As we discussed in the meeting (see log http://meetbot.fedoraproject.org/fedora-meeting/2014-07-02/fesco.2014-07-02-17.00.log.html#l-336) I think the problems currently outweigh the benefits, but that doesn't mean I don't see the benefits.

Is this an area where we should actively recruit help in improving the tools? Is it something that we might want to look at asking the Fedora infrastructure hackers to hack on?

Replying to [comment:23 churchyard]:

I'm not sure. Would any of those solutions use ACLs from Fedora packages (+ provenpackagers)? Would any of those repos clone together with packages repos?

Nope. You could add whatever acls you like to a fedorapeople repo though.

Of course, once I clone the package repo and add github/fedorapeople/fedorahosted as a secondary git remote, it works. However the initial procedure is not very straightforward.

I was just curious what is the recommended workflow right now, so it stays somehow predictable. I guess the general idea now is "keep it out of dist git and do it as you please", but that will lead to a mess when a lot of "future Fedora" content would be spread around the Internet. I found the previous workflow more friendly for possible contributors.

Sure, although when it was in pkgs git it needed contributors to have acls for the package involved no?

Perhaps we could come up with some 'best practice' here and document it so the spread you are worried about doesn't happen.

Replying to [comment:26 kevin]:

Sure, although when it was in pkgs git it needed contributors to have acls for the package involved no?
I think so.

Perhaps we could come up with some 'best practice' here and document it so the spread you are worried about doesn't happen.
That would be great.

Replying to [comment:21 churchyard]:

In other words: You've banned our workflow, what's next?

Quoting the voted FESCo decision:

Work with infrastructure to plan a git for these uses, use fedorapeople or external services for now.

i.e. the generally preferred future is to have pkgs.f.o or something very similar to pkgs.f.o available for your workflow, and the underlying assumption is that this can in fact happen soon enough. So, “next” would be $somebody writing software to make that possible.

I’m afraid that wouldn’t help you just go on maintaining your packages as you were if nobody ever turned up to improve the Fedora pkgs infrastructure to handle the extra branches, though.

A solution I think would make sense is to have everyone put their work-in-progress/unfinished/not-for-Fedora stuff in wip/ branches and have koji disallow doing official builds from wip/ branches. This would also open up the possibility of doing non fast-forward pushes for anything under wip/ and basically consider it throw-away from the Fedora official packages point of view.

It still needs someone to do the koji legwork first of course. :-)

Login to comment on this ticket.

Metadata