Hello,
for those of you who attended DevConf.cz, you probably know what this is about. We have big plans how to improve the package management workflow. I'm Tomas from packit team [1] and we are interested in bringing Fedora project closer into upstream communities. I've tried to write down more what's and why's in readme of the packit tool [2]. I expect this will also be covered by the new Fedora objective(s) which are being formed these days.
This is really just a gentle introduction, please let me know if you need to know more details.
As I said, we'd love to see Fedora project's presence in upstream communities - as a Github application which upstream projects can add to their repositories and start integrating with Fedora OS. For such a thing, we need a bunch of stuff:
Please let me know how do we move this further.
[1] https://github.com/packit-service https://tomastomecek.github.io/speaks/2019-devconf-auto-maintain-your-package/#1
[2] https://github.com/packit-service/packit#what-and-why - the repo used to be called source-git, we renamed it since source-git is just an implementation detail
+1 to a FAS identity that has permissions needed to upload stuff to the lookaside cache, fork repos and send PRs.
-1 to a FAS identity that can push to actual packages until the above works for a while
(*) let's start with pull requests. when this is proven to work reliably, we can proceed with "the automation would merge the change".
(***) I assume you'll need to talk to legal and/or Council about that. As long as there is still an explicit maintainer action, I don't think this is an issue.
Miro, I completely agree with you! Our intention for packit is to create pull requests on src.fp.o only, right now we are not even scoping a feature to push directly to dist-git: every change should go through a pull request review process and the "all packages" CI pipeline.
I have similar feelings to Miro. Note that we do have the-new-hotness, and there is an RFE with it to have it send pull requests:
the-new-hotness
https://github.com/fedora-infra/the-new-hotness/issues/189
It might be good to work with that project and get that feature in, especially since it's already wired into many packager's workflow as is.
As for legal, I agree that FESCo shouldn't be making legal calls. I will say that it may be helpful to use that licensecheck utility that fedora-review has in some capacity. You know how we have CI on src.fpo pull requests? Maybe a licensecheck bot could report there and note if anything changed since the previous version? That wouldn't be perfect, but it would be convenient for packagers to know "hey, I see something changed since last time, please take a look."
As a maintainer of the-new-hotness I'm ready to collaborate on this project.
The thing is that the-new-hotness and packit will likely have different workflows for pulling those releases: 1. hotness needs an entry in upstream release monitoring and then it updates spec in dist-git and sends the PR, right? 2. packit actually requires spec file in upstream git and then uses the spec to perform the update - upstream maintainer controls everything, the update is almost guaranteed to succeed, especially once we start validating the upstream software during development
To me, it makes sense to implement both workflows -- let's use hotness for updates of simpler packages and use packit for people who already use spec files in the upstream.
But yeah we're in touch with @zlopez so we'll figure it out.
You know how we have CI on src.fpo pull requests?
My expectation is that we should run whole fedora-review on every dist-git PR. This is something we should scope for the CI objective. Randy, I really like the suggestion with the differential runs. This is something we already do internally and have the tool open source which does the diff part of results of the static analyzers: https://github.com/kdudka/csdiff
Ah interesting, that does sound pretty different.
I wonder if the FPCA would need to be agreed to by upstreams that do this? Seems like we should probably consult with the legal list about this.
Note that moving specfiles to upstreams is something I definitively and absolutely disagree with.
https://docs.fedoraproject.org/en-US/packaging-guidelines/#_spec_maintenance_and_canonicity
Fedora’s git repository is the canonical location for Fedora spec files. Maintainers MUST expect that other maintainers and automated tooling will make changes to their packages, potentially without communicating prior to doing so (though communication is always encouraged). If some maintainers are also attempting to keep copies of a spec in an outside repository, they MUST be prepared to merge changes made to the spec in Fedora’s repository, and MUST NOT overwrite those changes with a copy from an external repository or using fedpkg import.
Exactly. There's a bunch of projects which keep a spec file upstream, and interacting with them is a PITA. Apart from the issues @churchyard quoted: - the spec files might need to be different between Fedora releases, and upstream can't be expected to keep multiple versions of the spec file - a spec file each distro is slightly different
Both issues can in principle be handled by enough %ifdeffery, but the resulting spec file is a monstrosity. The simple truth is that spec files are a downstream thing, and trying to push them upstream goes against the technical and social split between upstreams and distributions.
The other ideas that were mentioned (PRs, running fedora-review and all PRs, automatic license difference checks) sound great, but pushing the spec file upstream seems wrong. Only a minority of projects will want to have such a spec file, so apart from being inconvenient for Fedora, would also limit the number of projects that can be included in this initiative. Instead, I'd love to see a travis-like choice, where upstream can easily select Fedora versions against which they want to test, and the package is built for all of them.
When I started working on Bodhi, it had a copy of its spec upstream for some reason, and I removed it because I find the separation of upstream and downstream concerns to be helpful.
I also try not to use if statements in my spec files and take advantage of git branching heavily. This helps my spec files to be much more readable, and allows me to have differences between Fedora/EPEL releases. When I do want to make a change across releases, I simply cherry-pick the commits.
If I as the primary Bodhi upstream maintainer don't want to have a spec file in my repository, it seems hard to imagine that many projects that are not affiliated with Fedora would want it.
As a historical perspective, in the 1990's we used to upstream a spec file for pretty much any project that Red Hat had an upstream presence for, and they uniformly either became a) stale - nobody touched them or b) a mess, because they were being used to do builds across multiple distributions without regard to the distribution's packaging standards.
If we literally used upstream spec files, as in this proposal here, then problem a) would go away, but I don't see how we could prevent b) - the upstream would be expected to maintain the spec file to the needs of all distros, and would not want to use Fedora-specific macros, or conform to Fedora guidelines.
I think we should generally attack the problem of needing to maintain things in two places by working to make Fedora spec files more minimal - so that an upstream release generally requires no manual changes to the spec file, not even adding a changelog entry.
we should generally attack the problem of needing to maintain things in two places by working to make Fedora spec files more minimal
We'd also need to convince all other distros to accept such minimal spec files. Otherwise, we're back to b), just with the %ifdefs for Fedora having less stuff than for some others.
Are we okay with @ttomecek using this on his own packages for now? Until it's proven and adopted more broadly? He would obviously not touch other packages until it's ready, and then they opt in. So I guess his choices would be:
a) Use his own credentials for the automation. Place his own credentials in the bots secrets files, and store them in keytabs etc... b) Get some 'packit' credentials for the automation to use.
What do you think?
I'm not OK with @ttomecek or anybody else to put a canonical source of Fedora spec upstream.
I'm OK with setting a special FAS account with pull requests + lookaside cache only privileges, as I've said already.
Are we okay with @ttomecek using this on his own packages for now? Until it's proven and adopted more broadly? He would obviously not touch other packages until it's ready, and then they opt in. [...]
Any Fedora packager is free to use automation for managing their packages - and many do in one way or another. I think the main thing is that - until there is general approval for a better way to do things - the spec file in Fedora needs to be canonical - if there's a mass scriplet fix, or a Requires: needs to be updated to get the distro working, those changes shouldn't 't be overwritten next time an upstream release happens.
I think the main thing is that - until there is general approval for a better way to do things - the spec file in Fedora needs to be canonical
This makes sense on the surface. Is this an existing policy, or are you introducing a new one here? I know of many Fedora packages that already have their canonical spec file stored elsewhere.
Are spec files themselved (and rpm macros) versioned at all? Is there any change management besides a mass change?
Most other files (package.json, Dockerfile) have a stated deprecation, change management, and stability guarantees. Does this exist for spec files? Or is this what you're referring to as the missing @otayor?
@ttomecek It sounds change management of spec file and macro features is a key enabler.
I think the main thing is that - until there is general approval for a better way to do things - the spec file in Fedora needs to be canonical This makes sense on the surface. Is this an existing policy, or are you introducing a new one here? I know of many Fedora packages that already have their canonical spec file stored elsewhere.
Maybe "canonical" is an overloaded word here. It doesn't matter if the spec file in Fedora is the actual place where most edits start out, as long as if a change is made to the spec file in Fedora as part of the Fedora processes, that change is not blindly overwritten in further updates - it gets propagated back as necessary without the person doing mass Fedora spec file fixes having to do research and file a manual PR somewhere.
Can I point to a line in packaging guidelines saying that you shouldn't simply overwrite changes made to your spec files in Fedora distgit? No, But I think that's common sense.
I don't understand the question. Spec files, in Fedora, are in version control - and they have version numbers embedded in them. But you know that :-)
As you know, the spec file format itself is highly stable. But the set of macros used within the spec file and the set of dependencies that a spec file references makes spec files closely tied to the contents of the rest of the distro. Trying to make spec files that aren't tied to the distro is a LSB-like exercise in futility.
In my personal perspective: except in very limited circumstances, specfiles upstream is a proven bad idea and I don't think it's necessary for high quality automation - I would be disappointed if we're putting a lot of work into automation tools that depend on that.
Can I point to a line in packaging guidelines saying that you shouldn't simply overwrite changes made to your spec files in Fedora distgit?
I can and I did: https://pagure.io/fesco/issue/2084#comment-552904
@churchyard Wonderful thank you. That sets the baseline requirement for the automation.
Is there something specific FESCo should vote on?
My opinion:
Hence:
Instead, I'd love to see a travis-like choice, where upstream can easily select Fedora versions against which they want to test, and the package is built for all of them.
I'm + :100: for this. But without specfiles. Let specfiles be what they are: recipe's for getting upstream code into downstream. Upstream doesn't need to worry about specfiles. Give them a reliable, easy to setup super simple service as pictured in your slides:
<img alt="GitHub integration" src="https://tomastomecek.github.io/speaks/2019-devconf-auto-maintain-your-package/failed-fedora-packaging-service.png" />
Except drop the "packaging" part. Say "builds on Fedora". Making Fedora the platform for upstream integration testing? I'd love that. Bother upstreams with specfile maintenance and bother provenpackagers with upstream specfiles? I don't like that.
What Miro said.
I think it should be possible to even further, and do automation of packaging, but that's a later step. In the long term, I'd love to see automation for upgrading of a package to a specific version, so that one could say (in a dist-git checkout):
fedpkg-or-something-else upgrade-to 1.4.8 fedpkg-or-something-else upgrade-to git12322dfda
and that'd
mangle the spec file as necessary, understanding pre- and post-release versioning download the sources optionally upload sources to the cache insert a changelog entry optionally find any bugzilla bugs about a new version being available and mention them in the changelog
Essentially, something like the-new-hotness did (does?), but not through bugzilla, but on demand. Then, it'd be fairly easy to also test packaging under Fedora, i.e. request a scratch build from any arbitrary upstream commit on any Fedora release.
@ttomecek What is the actual ask from FESCO here? Or is this ticket just a "Request for Opinions"?
FESCO doesn't provide hosting (for example) ... or Github idenities. Is the only actual ask about a FAS identity? If so, it should say that in the subject line.
On Wed, 2019-02-13 at 08:31 +0000, Zbigniew J=C4=99drzejewski-Szmek wrote:
and that'd mangle the spec file as necessary, understanding pre- and post- release versioning download the sources optionally upload sources to the cache insert a changelog entry optionally find any bugzilla bugs about a new version being available and mention them in the changelog
mangle the spec file as necessary, understanding pre- and post- release versioning download the sources optionally upload sources to the cache insert a changelog entry optionally find any bugzilla bugs about a new version being available and mention them in the changelog
It'd be suhweeeet if it also:
Essentially, something like the-new-hotness did (does?), but not through bugzilla, but on demand.
There are plans to make the-new-hotness send pull requests, which in my opinion would be better to invest in than an on-demand solution, but I'd be happy to have either solution.
I consider on demand solution better over PR as it gives me more control. The on demand solution is easier to extend to run as a service and to send PRs than bending the service to do what i want, locally, when I want it. Start with the on demand tool as a building block...
Metadata Update from @psabata: - Issue tagged with: meeting
There are plans to make the-new-hotness send pull requests
PRs are a nice step. But they don't serve some very important use cases: - a local developer who wants to do a local update. This is necessary for the cases where we're the-new-hotness wasn't configured yet and doesn't know about the latest update, or we're updating a non-latest version, etc. - a local developer wants to script updates to packages (e.g. a python developer wants to mass-update all 2k python packages) - the PR needs to be edited, because some additional fixes are needed.
If we have the "local" workflow first, we can always build the "remote" workflow on top, incrementally. Essentially, what Miro said: it
is easier to extend to run as a service and to send PRs than bending the service to do what i want, locally, when I want it
Sorry for the lack of response, but I was on a holiday for the past week. (I definitely wanted to send this out to kickstart the discussion, thank you everyone for all the answers)
One thing I'd like to point out - our expectation with spec files upstream is that downstream maintainer = upstream maintainer, or at least ⊆.
⊆
The other thing to highlight is that we want packit to keep upstream & downstream spec files in sync - whenever there is a change downstream, the tooling would open a PR upstream with the changes.
I disagree. As an upstream project, you want to make sure that your software works on platforms you support, such as Fedora. How do you make sure of such a thing when you don't have the spec file upstream?
Also what we are proposing is not meant to be a silver bullet for everyone. I actually feel like that the-new-hotness solution may cover more packages than packit, especially those language-ecosystem libraries where changes to spec file usually require just a new changelog entry, setting release to 1 and updating version. For the more complicated things (kernel, systemd, anaconda, pykickstart, cockpit) we are aiming for packit.
@zbyszek Such tool already exists, it's called rebase-helper.
Since the discussion is so heated, do we want to talk about this? @contyk, thanks for tagging this for your meeting, would such a thing be enough?
I will update main original post with the ask.
I think we need something more generic. We have 2(3?) main use-cases:
+1 @ignatenkobrain
The problem with 2) is that you would lose changelog. Luckily we can use the work of @clime on rpkg-util to generate changelog and automatically populate release number. This is also on our radar but I fear we won't have the bandwidth to make it happen, by Flock, so we would need help.
2) is that you would lose changelog
Leave this to implementation. In future we probably won't need to generate changelog at all ;)
IMHO FESCo is not a place to discuss this like we are doing now. We can do that on devel ML. This is a ticket for FESCo: What exactly do you ask FESCo to do?
As I said in https://pagure.io/fesco/issue/2084#comment-554449 (original post is now updated with the ask, I can make it even more clear), I just need approval from fesco so that I can start asking other teams for credentials and permissions, and initiate the new github service using Fedora brand.
If such approval is not needed from fesco, please close this and let's move the discussion to devel list (I'll start a thread today).
Edit: I need to prepare more materials for the devel discussion so that everyone is on the same page. This thread helped me with that, thank you. I start the devel thread once I have all the materials prepared.
From yesterday's meeting:
* ttomecek will share this on devel@, FESCo will continue discussing in the ticket and will revisit in a week (contyk, 17:06:09)
If we decide to close it before then, we certainly can :)
OK, here's my (repeated) vote on the matter:
+1 to a FAS identity that has permissions needed to upload stuff to the lookaside cache, fork repos and send PRs. (this includes kerberos and all technical things to make it work)
-1 to a FAS identity that can push to actual packages
+1 @ignatenkobrain The problem with 2) is that you would lose changelog. Luckily we can use the work of @clime on rpkg-util to generate changelog and automatically populate release number. This is also on our radar but I fear we won't have the bandwidth to make it happen, by Flock, so we would need help.
I think Igor was talking about a case where a whole spec file is generated from scratch from upstream sources. In the case of rpkg-util, only certain parts are generated dynamically, those for which there is a respective preproc template language tag in an input spec file template. The rest of the spec file is is kept unchanged so you can combine dynamic parts with static ones. That being said, a spec file template with a single line:
{{{ pyp2rpm motionpaint }}}
is a possibility although probably nobody would use it like that.
What I can quite imagine though is to use the part of pyp2rpm that generates BuildRequires and make a preproc template tag from it. That would be super cool but to this moment I've been focusing on just git macros that render changelog, release, name, version, etc. from git metadata. They don't read package content like e.g. pyp2rpm which reads setup.py to extract some package data, they are only concerned with what is stored in git metadata, like e.g. name and content of the latest annotated tag.
Another thing is that rpkg-util mainly focuses on package maintainers that care about their spec file, polish them, cuddle them locally, etc. It just offers tools to make this workflow more smooth by integration with git and using what is already provided by git (like git commit messages).
But yet another focus is very much what zbyszek described:
rpkg-util will be able to provide this kind of workflow (in addition to rebase-helper). The last point about Bugzilla could be e.g. solved by extending the default git macros to do the bug lookup.
You could e.g. also think about closing or modifying bugs which are mentioned in the git commit messages. This won't be probably be part of the rpkg-util itself but instead rpkg-util will offer a way for a packager to easily define his/her own commands from within a configuration file so it will be possible to define rpkg close-bugs command which will do whatever packager wants it to do.
rpkg close-bugs
Part of its operation will probably also be verifying that the local git content is synced with remote properly. Those commands can be also defined distribution-wide by providing definitions for them in /etc/rpkg.conf. Package-wide configuration can be done by placing rpkg.conf into git toplevel directory in a package repository.
/etc/rpkg.conf
I welcome any help or feedback here.
As for spec files maintained in upstream, rpkg-util's take on this is that you should be able to have spec file next to unpackaged/raw sources but it does not necessarily mean that those sources are "upstream". They can be an upstream in which case they should (might) probably be placed directly at src.fp.o or they can be just a downstream mirror of the respective upstream projects. Unpacked sources are suitable for dynamic BuildRequires generation and you don't need to handle patches.
So I just wanted to introduce rpkg-util here a bit. It is a local command-line packaging utility that should offer possibility to "plugin" add-on commands and that offers good rpm-git integration. But sure, it can be also used on a server to e.g. automatically render a spec file or to generate an srpm.
I will try to also give this information when rpkg-util-3.rc1 is introduced on Fedora mailing list. It might take a little bit of time cause right now i don't have any colleagues :(.
@clime Plese move this to the devel ML.
@churchyard at the moment i would like to focus on producing a release candidate....i might chip in a related discussion started by someone else though
The thread is up: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/2JLBYZXTFSR42RWGTQUVUGYK66UJXPFA/
Thank you very much for the discussion in this thread and let's move it to devel@ now. I really value all your very good points and concerns.
Metadata Update from @zbyszek: - Issue close_status updated to: Accepted - Issue status updated to: Closed (was: Open)
This was discussed in today's FESCO meeting: AGREED: Creating a FAS identity that has permissions to upload stuff to the lookaside cache, fork repos and send PRs (including kerberos and all technical things to make it work), but not to push to repos or build packages, is approved (+7, 0, 0)
FESCo is concerned that the presented idea of how this automation should work is only applicable to a very limited set of packages and would rather see a focus on automating stuff for greater audience.
FESCo is also concerned that moving spec to upstream breaks experience for multiple interested parties.
Metadata Update from @zbyszek: - Issue untagged with: meeting
Thank you.
Yes and no. With source-git this is applicable to any project.
We shall see, that's why we're doing the prototype.
Please reply to my devel thread if you'd like to discuss.
For the record: I've met with @ttomecek and he encouraged me (and generally anybody) to communicate more ideas for "automating stuff for greater audience."
Please use the thread on devel or packit issue tracker to do so if you are interested.
Login to comment on this ticket.