In order for rpmautospec to work, there needs to be a git history of the package

1) which lets us find out where versions were changed and
2) whose commit logs can be meaningfully used as commit messages.

From looking at an automatically generated dist-git repo of a COPR of mine, I think 1) should be fulfilled, but I have serious doubts about 2)—a package changelog consisting mostly of entries saying "automatic import of <package name>" isn't very useful.

As far as I know, copr build doesn't build from a local git repository (like fedpkg), but you need to roll an SRPM on your own before. At least I'm not aware of a different mode, and have used fedpkg srpm (plus some flags) in the past to build SRPMS for COPR packages I maintained in a distgit-like git repository. If you did the same, current versions of rpkg (1.62-6) would preprocess the spec file so the resulting SRPM would have incrementing release numbers and a meaningful changelog—without the COPR backend having to care about %autorelease/%autochangelog because of the local preprocessing.

/cc @msuchy @praiskup I'm interested in your thoughts and corrections.

Yes. Copr does not provide an option for users to interact with dist-git. We use dist-git as a storage so we are able to do a rebuild if needed. Therefore there are no meaningful commits.

IMO autospec support in Copr does not have sense. It needs to go one level up. I.e. to Mock-scm, Packit, Tito...

As far as I know, copr build doesn't build from a local git repository (like fedpkg), but you need to roll an SRPM on your own before. At least I'm not aware of a different mode, and have used fedpkg srpm (plus some flags) in the past to build SRPMS for COPR packages I maintained in a distgit-like git repository.

As Mirek said above, slightly different wording:

Copr DistGit is more like proxy currently - we don't want to re-download and re-compile
all the sources for each chroot.. so we first build the source once, import that to
the proxy dist-git, and from there we build all the chroots.

If you want to build from dist-git, you can do copr build-distgit. That is using
/usr/bin/copr-distgit-client tool for putting the distgit sources (various instances)
together. Patches are welcome there!

IMO autospec support in Copr does not have sense. It needs to go one level up.
I.e. to Mock-scm, Packit, Tito...

I'm afraid this is the expected design of autospec that it has to be implemented
in Koji build system (not in Copr on the lower level, like Mock). At least for the
release number.

I was thinking about autospec + copr several days back and I think koji should
probably provide - via API? - the latest built release number. And Copr should
consume that. Otherwise, we will not be able to replicate the NVR?

Honestly, I think that we could live (for some time) with %changelog unfixed or
shortened to a single-entry pointer to the original git repo.

I checked once more, we do the "eco" shallow clone from the given DistGit clone
url, then we run copr-dist-git sources && copr-dist-git srpm` (download sources
&& generate source RPM).

The Koji builds are irrelevant for release number, It is calculated from commits instead.

Using the Fedora distgit source option, I would expect that Copr correctly calculates the release and changelog via fedpkg (rpkg-1.62-6+ required).

[fedora-scm]$ fedpkg clone rust-ammonia
Cloning into 'rust-ammonia'...
...
[fedora-scm]$ cd rust-ammonia/
[rust-ammonia (rawhide)]$ fedpkg verrel
rust-ammonia-3.1.2-1.fc35
[rust-ammonia (rawhide)]$ git commit --allow-empty -m test
[rawhide 86ecbe9] test
[rust-ammonia (rawhide)]$ fedpkg verrel
rust-ammonia-3.1.2-2.fc35

Well ... if you use COPR to build directly from a Fedora dist-git repository, you could use the manual "make srpm" method, I guess? And include a simple .copr/Makefile with an "srpm" target that only runs "fedpkg srpm" ... And adding "fedpkg srpm" as another built-in "How to build SRPM from the source" method in COPR should be possible (?), so no Makefile would be needed.

The Koji builds are irrelevant for release number, It is calculated from commits instead.

Yeah, I see. So we need to have git history to build the package now.
I'm really trying to be as much open as this concept, but you know I think.

Using the Fedora distgit source option, I would expect that Copr
correctly calculates the release and changelog via fedpkg (rpkg-1.62-6+
required).

We didn't expect the DistGit concept is going to be complicated..

Till now, we had checked-out spec + lookaside. And yes, we used to have
'fedpkg' in use for some time in Copr, but there were never-ending
compatibility fixes... copr-only patches .. The problem is that rpkg
is designed to work with only one dist-git instance at one time.
While we have 'dist-git' component, we don't have 'dist-git-client'
component.

So later there was a request
to support other dist-git instances than the Fedora one. So -- since work
distgit was just 'git clone => checkout => download sources', we created a
script that is able to do this task for all the DistGit instances we
needed to support (CentOS, Fedora, RHEL, Copr). That tool is basically
something like 'fedpkg-minimal', 'rhpkg-simple', etc. together.

Btw., I failed to find how Koji builds the source RPM when rpmautospec is
in use. https://koji.fedoraproject.org/koji/taskinfo?taskID=69330277
Where are the logs? Perhaps we could re-use some fedpkg-minimal logic?
Or what happened with fedpkg-minimal?

So I think I agree with Mirek, that some low-level tool should do this.
Could this be done directly in rpmbuild? That would bring us back
a distro agnostic way of generating source RPM from spec file.
@pmatilai

what happened with fedpkg-minimal?

Ping on this question. Fedora Koji seems to nicely obfuscate the way it calculates
the source RPM Release number nowadays....

If you want to build from dist-git, you can do copr build-distgit. That is using
/usr/bin/copr-distgit-client tool for putting the distgit sources (various instances)
together. Patches are welcome there!

Now I checked we can not just fix copr-distgit-client. Copr obtains sources
only once for one build (to avoid DDOS-ing the external world), even when package
is built from DistGit. From those sources, we build the packages into all chroots ...
(we don't have full git history on the builders). For that we need to have the
Release and %changelog expanded - I filed #226 for this.

I checked once more, we do the "eco" shallow clone from the given DistGit clone

Relying on git-history is IMO a very bad design step (or compromise), though
we absolutely have to support at least shallow clones. Filed as #227.

Relying on git-history is IMO a very bad design step (or compromise)

The alternative would've been to make a network query to koji for this, which would arguably be even worse?

The Release number and changelog need to be calculated ~somehow~ without storing them in git - because that was the whole point here: make parallel changes not conflict on changelog entries and / or Release bumps, etc. so the only way to calculate those values reliably is by looking at the actual commit history.

The alternative would've been ...

Sorry for turning it a bit off-topic (the cons were clear before the
implementation, I tried to discuss alternatives before on fedora-devel,
and cons were accepted). Now when ~800 packages are using this
approach it is serious, and it's time to prove it is maintainable.

We are about to start Python 3.11 rebuilds in Copr and if this isn't solved somehow properly, we will need to invent our own hacks.

One idea I have is to override the %autorelease macro in Copr to always return the UNIX timestamp. That should work for our purposes quite nicely. However, I am not sure if the distgit SRPm builder installs packages from its Copr or if it uses some pre-defined shared buildroot.

@praiskup Do you know how could we "inject" a package or a macro into the SRPM builder?

FYI, @nphilipp is out on PTO this week...

This was mostly a question for the Copr folks.

The configuration options for SRPM build are limited; I am afraid that you only can
fallback to the custom method, and generate the sources manually:
$ fedpkg clone -a "$pkg" ; cd "$pkg" ; sed "s/%autorelease/$(date +%s)/" *spec -i ; fedpkg srpm

^ if you do it with the custom method, you could actually just let "fedpkg srpm" process autorelease and autochangelog for you (since it has the repository to work with this should give the same srpm as building it in Koji)

I don't think so, because #226 (fwiw, this repo looks abandoned for several months).

This should be fixed since rpkg 1.63:
https://docs.pagure.org/rpkg/releases/1.63.html

(which you can confirm by running "fedpkg srpm" locally)

It is not:

$ rpm -q fedpkg python3-rpkg
fedpkg-1.41-1.fc35.noarch
python3-rpkg-1.63-2.fc35.noarch
$ fedpkg srpm
Not downloading already downloaded zim-0.74.1.tar.gz

setting SOURCE_DATE_EPOCH=1632528000
Wrote: /tmp/Zim/Zim-0.74.1-1.fc36.src.rpm
$ bsdtar xf /tmp/Zim/Zim-0.74.1-1.fc36.src.rpm -O Zim.spec  | grep ^Release
Release:        %autorelease

Well, given the Release: %autorelease I might just need to inject the macro to the phony macro to the actual buildroot, no?

@praiskup look at the actual .spec file inside the SRPM, not just grepping for "Release" line. The %autorelease macro should be defined as an RPM %global there with a lua script in the processed .spec file.

I'll run some tests:

• dist-git source
• custom source with fepdkg srpm
• uploading the srpm generated locally with fedpkg srpm
• adding the autorelese macro stub to the buildroot

Indeed, this is what would be committed into copr dist-git, and built:

## START: Set by rpmautospec
## (rpmautospec version 0.2.5)
%define autorelease(e:s:pb:) %{?-p:0.}%{lua:
    release_number = 2;
    base_release_number = tonumber(rpm.expand("%{?-b*}%{!?-b:1}"));
    print(release_number + base_release_number - 1);
}%{?-e:.%{-e*}}%{?-s:.%{-s*}}%{?dist}
## END: Set by rpmautospec

Do we have links to some discussions about this approach?

It was part of the final version of this Change proposal:
https://fedoraproject.org/wiki/Changes/rpmautospec#The_.25autorelease_macro

This approach was then approved by FESCo:
https://pagure.io/fesco/issue/2582

Relevant discussion on the devil list in this thread (among others):
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/JLY5NWWLB3JOOCQPFSJFIKMPSK3A4UV6/

Or did you mean something else?

Building rust-ammonia in https://copr.fedorainfracloud.org/coprs/churchyard/autospec/builds/

The release should be 2.

• dist-git source:

  • release is 1 trough the entire build
  • results in rust-ammonia-devel-3.1.2-1.fc36.noarch.rpm

• custom method, fedpkg required:

  • SRPM build wrote rust-ammonia-3.1.2-2.fc36.src.rpm
  • copr unpacked it and recreated it as rust-ammonia-3.1.2-1.fc34.src.rpm instead
  • results in rust-ammonia-devel-3.1.2-1.fc36.noarch.rpm

• custom method, fedpkg and rpmautospec required:

  • same as above

• uploading the srpm generated locally with fedpkg srpm

  • SRPM uploaded as rust-ammonia-3.1.2-2.fc36.src.rpm
  • copr treats it as rust-ammonia-3.1.2-2.fc36.src.rpm
  • results in rust-ammonia-devel-3.1.2-2.fc36.noarch.rpm

So, the custom method is apparently not a solution :/

dist-git source:

Let's consider this experiment in copr-distgit-client:
https://pagure.io/copr/copr/pull-request/1942
Till something like this is implemented, it can not work.

custom method, fedpkg required:

If rust-ammonia-3.1.2-2.fc36.src.rpm, such RPM should be actually imported into copr dist-git, with macros expanded. Looks like some old library was used to generate the src.rpm?

uploading the srpm generated locally with fedpkg srpm

This isn't a different thing from the custom+fedpkg methods before. The fact it
works for you must be that the environment the method is run in is slightly different
than your local environment.

Looks like some old library was used to generate the src.rpm?

https://download.copr.fedorainfracloud.org/results/churchyard/autospec/srpm-builds/02874741/builder-live.log.gz

Yeah, indeed the thing is that Copr actually isn't using the fedpkg srpm source RPM product. Instead, it uses the source files only (spec + sources), and generates a new source RPM from that. So to make the custom method work, another hack is needed ... something like:
bsdtar xf *.src.rpm '*.spec' once the src.rpm is generated by fedpkg srpm.

Installing my override to the custom SRPM method is not possible.
Installing my override to the buidlroot produces rust-ammonia-devel-3.1.2-1633431412.fc36.noarch.rpm (but the build is marked as 3.1.2-1.fc34 in copr interface).

Yeah, indeed the thing is that Copr actually isn't using the fedpkg srpm source RPM product. Instead, it uses the source files only (spec + sources), and generates a new source RPM from that. So to make the custom method work, another hack is needed ... something like:
bsdtar xf *.src.rpm '*.spec' once the src.rpm is generated by fedpkg srpm.

fedpkg import --offline maybe?

fedpkg import --offline *.src.rpm --skip-diffs should do the trick as well, yes

Copr-rpmbuild 0.54 is deployed in Copr production, so the dist-git method is able to handle
%auto* macros.

https://copr.fedorainfracloud.org/coprs/churchyard/autospec/build/2884791/

Awesome, thanks!

This could possibly be closed.

Hey !

Coming back to this discussion, but can't COPR build IDs be simply be used as a release number for other, non dist-git, methods ? COPR builds are unique and incremental.

This could cause problems, so maybe just adding it as an option ?

Or at least, having the ability to put the release ourselves.