The easier way for COPR to do this would be to setup a lighttpd alias on backend for fedora-29 to point to fedora-rawhide for each given arch.

The problem is that it requires manual adjustments after each branching to increment the number and that's not particularly pleasant (the branching process is quite difficult even without it - we need to take into account when mock gets released, when public repos for the new branch become available, etc.). Also I don't think, people should rely on this in code in any way because there would be human factor included.

I actually went through something like this before when I tried to use numerical denominator for 'rawhide' in COPR...it wasn't a good idea. 'rawhide' is simply rawhide and there are not many ways around it. I think, it is simply a special case by definition so software that needs some special handling is doing it well.

Can you show me concrete code sample(s) where naming 'rawhide' by a number would help?

Thanks for looking into this. One of the examples in how to run tests in Taskotron (then showing up in Bodhi for package updates). For each new Koji build, we spin up a VM image matching the build (F27 for fc27 build, etc). For each image, we install additional dnf repos into the system. That is used for example in dev environment, where we want to use the latest (not necessarily stable) libtaskotron builds. So in our config file, we have this:

## Additional repos for minion to install packages from
minion_repos:
  - https://copr.fedorainfracloud.org/coprs/kparal/taskotron-dev/repo/fedora-27/kparal-taskotron-dev-fedora-27.repo

This works for all releases except rawhide. I can add a lot of logic, i.e. replacing minion_repos field with minion_repos_no_rawhide and minion_repos_rawhide (to allow users to specify which repos to install on non-rawhide images and which ones on rawhide images), then I can detect which images we're booting from (via a heuristic from the image file name, which includes using a rawhide=NN config option), and then using the right set of repos for that image. However, this is a) quite complicated for us (maintenance) and for our users (running tasks locally and editing the config file) and b) very Fedora-specific. I'm unhappy about it and I'm trying to figure out a way how to use the same copr repo without this heavy logic.

For our custom repos maintained manually, I already use the symlink approach and of course I'll need to manually maintain it. But for copr repos, I can't add the symlinks, so I'm hoping some option can be added from your side. If it includes a lighttpd alias that might be forgotten during branching, I'd be honestly very happy about it. It's still much easier for me to remind you twice a year (I have to update symlinks in our custom repos anyway) and I don't need to maintain the extra logic in code. Additionally, the copr fix will save me this work in all our projects (I just mentioned one here). And other people's projects.

However, this http alias solution makes me uneasy in one part. If you implement the second or the third idea in my original post, and then you forget out it during branching, nothing bad happens. Just the new rawhide won't have a redirect, but everything will work fine. But if you forget to change the http alias, people building into Branched (F29) will get redirected to Rawhide repos, until you remember to fix the alias. And that might cause problems for some projects.

Thanks for looking into this. One of the examples in how to run tests in Taskotron (then showing up in Bodhi for package updates). For each new Koji build, we spin up a VM image matching the build (F27 for fc27 build, etc). For each image, we install additional dnf repos into the system. That is used for example in dev environment, where we want to use the latest (not necessarily stable) libtaskotron builds. So in our config file, we have this:

 ## Additional repos for minion to install packages from
 minion_repos:
 - https://copr.fedorainfracloud.org/coprs/kparal/taskotron-dev/repo/fedora-27/kparal-taskotron-dev-fedora-27.repo

Alright, this is a good example. So the original problem comes from the fact that the rawhide builds have .fc29 disttag event before the fedora is actually branched from master. One simplification could be that rawhide builds are always tagged with 'rawhide' disttag. Actually I would suggest to put 'master' there so that it matches branch name in DistGit. The dist tag should be set to a particular .fc value after branching, not before.

In rawhide, it can just be set to master all the time. This way, you wouldn't need to know that a particular .fcXY tag needs to be translated into 'rawhide' now but not in a half year period. It will be always 'master' translating to 'rawhide'.

How about using dnf copr enable ..., or at least the logic inside? It should know best what to use on Rawhide or f29 respectively.

How about using dnf copr enable ..., or at least the logic inside? It should know best what to use on Rawhide or f29 respectively.

That is of course possible, and I might be forced to do it if there's no automagical solution from copr side. The downside is that, using my example above, instead of having a generic set of mirrors to enable, I need to have separate config options extra_repos and extra_copr_repos with a different implementation. Also, this approach can't be used everywhere (e.g. during OS installation). I was hoping for a generic solution to this.

OK. Let's get back to the original example:

 ## Additional repos for minion to install packages from
 minion_repos:
 - https://copr.fedorainfracloud.org/coprs/kparal/taskotron-dev/repo/fedora-27/kparal-taskotron-dev-fedora-27.repo

Do you always request a dnf config from copr-frontend when enabling it? (because https://copr.fedorainfracloud.org is a frontend address)

In that case, we could just implement a logic in copr-frontend that would return fedora-rawhide dnf config for requests to fedora-$(($latest+1)) url paths.

(We could also make it slightly more "liberal" and return `fedora-rawhide`` dnf config for each non-existent integer given all fedora chroots in our DB).

Would this be an acceptable solution for you?

I guess a better option here is that dnf introduced some variable like e.g. $rawver, which would return 'rawhide' on rawhide and number (as string) on branched Fedoras. This would then enable using:

 baseurl=https://copr-be.cloud.fedoraproject.org/results/kparal/taskotron-dev/fedora-$rawver-$basearch/

everywhere.

So if you agree with this, we could bring it to dnf.

So if you agree with this, we could bring it to dnf.

+1

Do you always request a dnf config from copr-frontend when enabling it? (because https://copr.fedorainfracloud.org is a frontend address)

No, we use the same address for all clients, so in fact they access https://copr-be.cloud.fedoraproject.org/results/kparal/taskotron-dev/fedora-$releasever-$basearch/
which is a backend address I assume.

Btw, a similar issue has just been "fixed" by Fedora Infra by providing redirects in mirrormanager, so you can now request repo=fedora-29 and you'll get rawhide repo. Of course COPR doesn't have a mirrormanager, that's why I proposed some alternative solutions.

I guess a better option here is that dnf introduced some variable like e.g. $rawver, which would return 'rawhide' on rawhide and number (as string) on branched Fedoras.

I can try to report a RFE. I'll link it here.

Do you always request a dnf config from copr-frontend when enabling it? (because https://copr.fedorainfracloud.org is a frontend address)

No, we use the same address for all clients, so in fact they access https://copr-be.cloud.fedoraproject.org/results/kparal/taskotron-dev/fedora-$releasever-$basearch/
which is a backend address I assume.
Btw, a similar issue has just been "fixed" by Fedora Infra by providing redirects in mirrormanager, so you can now request repo=fedora-29 and you'll get rawhide repo. Of course COPR doesn't have a mirrormanager, that's why I proposed some alternative solutions.

I guess a better option here is that dnf introduced some variable like e.g. $rawver, which would return 'rawhide' on rawhide and number (as string) on branched Fedoras.

I can try to report a RFE. I'll link it here.

Thank you. It would be cool if we could backport this into existing Fedoras once done. Btw. original idea belongs mostly to @praiskup so you can discuss further details with him.

Here's the RFE: https://pagure.io/releng/issue/7445

Here's the RFE: https://pagure.io/releng/issue/7445

Ok, thank you, I would be in favor of introducing a new variable instead of changing the meaning of the existing one but we will see, I guess.

I spent a lot of time thinking about it, and I decided using $releasever is the proper solution. It's used just for repos and nowhere else, and there's exactly where we need to have the string changed. Introducing a new variable would mean implementing it in all the tools (dnf, packagekit, anything creating repo files, etc) and I just couldn't justify the complexity. However, I'll be very happy to discuss this, let's use the releng ticket I linked.

I'm discussing this in the releng ticket and I have a question for you. It didn't occur to me before, but what if the proposed solution sounded like this?

Get rid of all fedora-rawhide-$arch directories, only keep fedora-$releasever-$arch (where $releasever is always a number, even for Rawhide, which is the existing implementation). This means no need for maintaining symlinks or having duplicated files. After branching point, add new repos automatically (so if Rawhide is F29 and is enabled in project config, after branching you add F30 to project config and create fedora-30-$arch dir for new builds. The previous fedora-29-$arch dir will automatically be used for Branched builds since then).

@kparal I can do that in Mock and then it will be basically no-op for Copr (minus the one shot work to fix branch-from-rawhide script)

I'm discussing this in the releng ticket and I have a question for you. It didn't occur to me before, but what if the proposed solution sounded like this?
Get rid of all fedora-rawhide-$arch directories, only keep fedora-$releasever-$arch (where $releasever is always a number, even for Rawhide, which is the existing implementation). This means no need for maintaining symlinks or having duplicated files. After branching point, add new repos automatically (so if Rawhide is F29 and is enabled in project config, after branching you add F30 to project config and create fedora-30-$arch dir for new builds. The previous fedora-29-$arch dir will automatically be used for Branched builds since then).

My understanding of rawhide is that it is a "rolling" distro which mirrors what's in DistGit's master branch. In other words, I understand it as an always moving target. I think many people understand it the same way. So in my opinion, this change might be confusing for some. Note that we already went through this a year ago and it was confusing. I then reverted it back for that reason.

We now have "Follow Fedora Branching" feature which takes what is in rawhide chroots and copies it into a newly created ones if a user chooses to do so. There is also created documentation for it. There would be also a problem what to do with the current fedora-rawhide chroots.

Also, if I am running rawhide on my machines (which I am not doing right now, just f28), I might like to have certain Copr repos with packages built years ago that will work no matter what $releasever is. Not sure if anyone is there doing this but without 'rawhide' chroots, this would no longer be possible. Better example is that I am running rawhide on my machine and I build my packages only for 'rawhide' chroot and I will be always able to install them.

I personally like the dnf variable solution with$rawver or just $ver, which would return rawhide on rawhide. I think that would be a fair solution and we could start using that in COPR instead of $releasever almost immediately (if the change is backported). I think it is very smooth solution to the problem and I would prefer a smooth solution after my experience.

Sigh, I was hoping we have a nice solution already.

We now have "Follow Fedora Branching" feature which takes what is in rawhide chroots and copies it into a newly created ones if a user chooses to do so.

The same could be done with the proposed approach. When Rawhide changes to F30, take F29 dir and copy it to F30 dir.

There would be also a problem what to do with the current fedora-rawhide chroots.

You're right, I haven't thought of that. The dirs are not truly a problem, but the fact that some people might have them set up on their computers is. Since your current rawhide repos hardcode rawhide in the url, their repos will break if the directory is removed, or get outdated if it is kept in place. That is unfortunate, and I can only see a) asking everyone to reconfigure their rawhide repo b) let rawhide dir be a symlink, which means we're back at the problem of maintaining symlinks (symlinks in this direction might be easier to handle, though).

Also, if I am running rawhide on my machines (which I am not doing right now, just f28), I might like to have certain Copr repos with packages built years ago that will work no matter what $releasever is. Not sure if anyone is there doing this but without 'rawhide' chroots, this would no longer be possible. Better example is that I am running rawhide on my machine and I build my packages only for 'rawhide' chroot and I will be always able to install them.

This would not be a problem if the FN contents got copied to FN+1 when bumping the Rawhide number during branching.

I personally like the dnf variable solution with $rawver or just $ver, which would return rawhide on rawhide. I think that would be a fair solution and we could start using that in COPR instead of $releasever almost immediately (if the change is backported). I think it is very smooth solution to the problem and I would prefer a smooth solution after my experience.

The problem with this is that it's not a generic solution. It would only be used by copr. And it would be confusing to other repo maintainers why we have several variables and which they should use, leading to many broken repos. I want to simplify it and make it more reliable, not more complex. If we really see rawhide as a moving target, then $releasever should return it and it should get used consistently throughout all our tools and infra (fedora repos, copr, other repos). However, Fedora Releng folks inquired why we shouldn't simply make sure the repos are accessible using a number, and I figured, if COPR implemented this, it would really be the simplest solution, and consistent. See the releng ticket for more details. I haven't considered the existing users, though, yeah.

The problem with this is that it's not a generic solution. It would only be used by copr. And it would be confusing to other repo maintainers why we have several variables and which they should use, leading to many broken repos. I want to simplify it and make it more reliable, not more complex. If we really see rawhide as a moving target, then $releasever should return it and it should get used consistently throughout all our tools and infra (fedora repos, copr, other repos). However, Fedora Releng folks inquired why we shouldn't simply make sure the repos are accessible using a number, and I figured, if COPR implemented this, it would really be the simplest solution, and consistent. See the releng ticket for more details. I haven't considered the existing users, though, yeah.

I agree with most of what you have written. However, I don't see adding the variable as a problem.

You simply use what works for you. $releasever has it in the name that it should return the release number of Fedora. $ver (or $rawver) would simply differ in that it would return the 'rawhide' string on rawhide. I think having two variables when one returns only release numbers (even for future releases) and the second which returns release numbers and 'rawhide' on not yet branched Fedora version is alright. People that work with repo files are used to work with many variables. I don't think this should really be a problem for anyone.

It would be used only by Copr in the beginning, yes. Later, it could be used by others if anyone finds use for it.

I've read the releng ticket. The problem is that Fedora has mirror manager so they can make the numbers work but Copr only has base urls so we would really need to change the way we store the build results. Note that i am all for finding some common ground in trying to solve this issue but I actually believe the current state in Copr is alrght and I really don't want to even remotely break anything.

Of course, if mock takes away fedora-rawhide-* configs or /etc/yum.repos.d/fedora-rawhide.repo will stop existing on rawhide machines, then we would probably react in some way but until then I would really prefer the variable solution. I don't mind actually opening a PR if we could agree on it being at least "a sufficient solution" (I see it as good solution but not sure if we can agree on that).

The thing that discourages me from extra variables like $ver is that this is not used just by repo authors, it's also used by users when interacting with dnf. If you want to query which packages are available in a different release (or try to install some of them), you use dnf --releasever=. This argument is even mandatory for dnf system-upgrade, so the user has to understand its concept. If there are multiple variables, the user needs to understand them all and the differences between them. Dnf tools need to expose additional command line options to set them. What's even worse, some repos will use one variable and other repos another, so you will never be able to tell which variable to set without inspecting the .repo files contents. That means you need to set both every time, and correctly (i.e. dnf --releasever=29 --ver=rawhide dosomething). All of this doesn't look user friendly to me.

Do you have a specific reason why you prefer two variables over having $releasever returning rawhide on Rawhide?

The problem is that Fedora has mirror manager so they can make the numbers work

Yeah. OTOH they say they can't use symlinks, while you can (I assume).

I really don't want to even remotely break anything.

I understand.

The thing that discourages me from extra variables like $ver is that this is not used just by repo authors, it's also used by users when interacting with dnf. If you want to query which packages are available in a different release (or try to install some of them), you use dnf --releasever=. This argument is even mandatory for dnf system-upgrade, so the user has to understand its concept. If there are multiple variables, the user needs to understand them all and the differences between them. Dnf tools need to expose additional command line options to set them. What's even worse, some repos will use one variable and other repos another, so you will never be able to tell which variable to set without inspecting the .repo files contents. That means you need to set both every time, and correctly (i.e. dnf --releasever=29 --ver=rawhide dosomething). All of this doesn't look user friendly to me.

Hmm...okay, that's true. But I don't think it is necessary to add --ver to dnf command-line...at least not immediately.

Do you have a specific reason why you prefer two variables over having $releasever returning rawhide on Rawhide?

I don't really have a reason except that when a new variable is introduced, we are sure it won't break anything. Otherwise I would be ok with just one variable, of course. If your 'Proposed solution 1` in the releng ticket can work just like that, then I am all for it.

As a side note, I would be also interested if you can change what's in /etc/os-release on rawhide:

[root@582c98529eb2 /]# cat /etc/os-release
NAME=Fedora
VERSION="29 (Rawhide)"
ID=fedora
VERSION_ID=29
PRETTY_NAME="Fedora 29 (Rawhide)"
...

I mean when you are on rawhide, you probably expect the version reported by distro.version() (http://distro.readthedocs.io/en/latest/#distro.version) to be 'rawhide' but instead it is

>>> distro.version()
'29'

People tend to say: "I run rawhide" (usually accompanied by "It doesn't break that often anymore") so from this point of view, "rawhide" should be the actual reported version, not '29'. Rawhide is an actual thing thanks to the existence of 'rawhide' repositories, rawhide images, rawhide composes...

Here we are treating it like it is always just a pre-release of something, which is kind of True but I might just want to run rawhide without actually caring if there is any release of it ever at all. I mean...if it works and it gives me the latest stuff, why should I care about an actual release.

The problem is that Fedora has mirror manager so they can make the numbers work

Yeah. OTOH they say they can't use symlinks, while you can (I assume).

Yes, but I see the dnf variable as a better solution because it shouldn't require any maintenance. Also, I think it would be much more reliable than whatever we tried to do with symlinks.

I really don't want to even remotely break anything.

I understand.

I don't really have a reason except that when a new variable is introduced, we are sure it won't break anything.

It would break some things, like the upgrade and other dnf commands. And packagekit, now that I think of it. But I understand the general notion.

Otherwise I would be ok with just one variable, of course. If your 'Proposed solution 1` in the releng ticket can work just like that, then I am all for it.

That's surely my preferred solution over an extra variable. It still requires fixing packagekit, but it looks cleaner to me. The cleanest solution would be to use numbers everywhere, but the compatibility concerns unfortunately make it difficult.

As a side note, I would be also interested if you can change what's in /etc/os-release on rawhide:
[root@582c98529eb2 /]# cat /etc/os-release
NAME=Fedora
VERSION="29 (Rawhide)"
ID=fedora
VERSION_ID=29
PRETTY_NAME="Fedora 29 (Rawhide)"
...

There are places where a number has to stay. I'm not certain about /etc/os-release, but for example in RPM packaging, all if 0%{?fedora} > 28-like comparisons would break if a number got replaced by a string. We could perhaps add VERSION_CODENAME=Rawhide to /etc/os-release, but changing the VERSION_ID might do mark harm than good (the spec also defines it as "mostly numeric", including examples). I'm just speculating here, I don't know the full implications. (That's why I mention that using numbers everywhere would be easier than our current mix-and-match approach.)

This whole ticket can now go away if #639 gets implemented (per instructions from #1763). I think it's fine to close this as a duplicate.

This issue has been migrated to GitHub: https://github.com/fedora-copr/copr/issues/267