Learn more about these different git repos.
Other Git URLs
Background I've been working on Fedora QA automation for many years. All these years, I've had to fight with the way how Fedora structures its repositories especially regarding Rawhide. I can't even count in how many tools we needed to define the current Rawhide <-> $releasever mapping and maintain that mapping. That includes special paths for Rawhide (or Branched) in terms of repository location. Today I finally forced myself to file this ticket and ask Releng for improving this from their side (while maintaining full compatibility with existing approaches).
$releasever
Problem Currently, $releasever and repository handling uses a completely different approach for stable releases, Branched and Rawhide.
Stable releases use $releasever NN number when accessing repos, either over mirrormanager: https://mirrors.fedoraproject.org/metalink?repo=fedora-$releasever&arch=$basearch or directly using a repo path: http://dl.fedoraproject.org/pub/fedora/linux/releases/$releasever/ They have fedora, updates and updates-testing system repos defined.
fedora
updates
updates-testing
Branched release uses $releasever for mirrormanager and for repo path, but it's placed in development/ directory rather then releases/: http://dl.fedoraproject.org/pub/fedora/linux/development/$releasever/ It has fedora, updates (empty) and updates-testing (empty or used, depending on release schedule) system repos defined.
development/
releases/
Rawhide doesn't use $releasever neither for mirrormanager nor for repo path, and it's again placed in development/: http://dl.fedoraproject.org/pub/fedora/linux/development/rawhide/ It has rawhide system repo defined (no fedora, updates or updates-testing).
rawhide
Due to Rawhide not using $releasever, it needs to have its custom set of repository files (and a differently named base repo), which point to the "right" mirrormanager and baseurl URLs.
So in conclusion, this is the list of things that don't work:
dnf --releasever=
dnf --enablerepo=updates-testing
The consequence of all of this is that our tools are littered with if clauses, doing special things for Rawhide (and sometimes Branched). We need to have separate configs for different releases, often quite complicated. We can't use the same additional repositories in all releases. We need to do hacks to figure out if a provided disk image is Rawhide or not, or deciding if we're running on Rawhide release during runtime. And the worst of all, we need to maintain all of this and reconfigure it several times per year (branching, going stable). This is error-prone, tedious and causes a delay during which things crash or provide incorrect results (e.g. between the branching happens and the configs are updated and deployed).
I'd like all of these issue to go away, and be maintenance free.
Proposed solution 1) Teach mirrormanager to recognize both rawhide and its matching $releasever at the same time. When Rawhide is 29, these two URLs should return the same content: https://mirrors.fedoraproject.org/mirrorlist?repo=fedora-rawhide&arch=x86_64 https://mirrors.fedoraproject.org/mirrorlist?repo=fedora-29&arch=x86_64
2) Create empty updates and updates-testing repositories for rawhide. These repos will be forever empty. Make them exposed in mirrormanager again under both names (29 and rawhide). This will allow to use the same DNF commands in stable releases and in Rawhide without side-effects.
3) Create a directory in repo layout that contains symlinks to all releases regardless of their release status. For example let's create http://dl.fedoraproject.org/pub/fedora/linux/releases/all showing this content:
linux/ ├── development │ ├── 28 │ ├── 29 -> rawhide │ └── rawhide └── releases ├── 26 ├── 27 └── all ├── 26 -> ../26 ├── 27 -> ../27 ├── 28 -> ../../development/28 ├── 29 -> rawhide └── rawhide -> ../../development/rawhide
As you can see, all releases are addressable from there. Not only that, but when Rawhide is 29, both these names are included and translate to the same target. This is important, because it allows people to use $releasever everywhere.
4) Have a $releasever number symlink also for updates and updates/testing paths:
updates/testing
linux/ └── updates ├── 26 ├── 27 ├── 28 ├── 29 -> rawhide ├── rawhide └── testing ├── 26 ├── 27 ├── 28 ├── 29 -> rawhide └── rawhide
Future improvements The changes proposed above would improve the situation immensely for me, and for other developers/tool authors interacting with releases in a similar fashion. But when this is in place, other, releng-related improvements could be achieved. For example, fedora-repos-rawhide could be got rid of. Currently you need to maintain two different set of repos, one for rawhide and one for others. With the above changes, you could use the same repos for all releases, i.e. fedora, updates and updates-testing. The last two would be empty for rawhide. The base repo would automatically work using $releasever. However, this way the release would follow the $releasever even after branching happens, so people running Rawhide would suddenly run Branched. That might not be desired, and could be solved by fedora-repos-rawhide containing just a single file - a config drop-in for DNF, that would override $releasever to rawhide (this part would need to be implemented, because it seems it's not possible today). With this approach, just by installing fedora-repos-rawhide, the users would make sure they always stay on Rawhide. The package contents would never need to be updated, same as fedora-repos wouldn't. Everything would work without regular maintenance.
fedora-repos-rawhide
fedora-repos
CC @ausil @kevin @puiterwijk @dmach @jskladan @tflink @adamwill @ralph
Metadata Update from @mohanboddu: - Issue tagged with: meeting
Another solution, tackling some of these problems from a different angle, would be to ensure that $releasever equals to rawhide on Rawhide. It is populated from the system-release(releasever) provide from fedora-release rpm. However, I'm not sure whether that would influence the value of %{fedora} and %{dist} rpm macros (which could cause many issues), and generally whether it's a good idea or not.
system-release(releasever)
fedora-release
%{fedora}
%{dist}
If I may add something...I think having %{dist} being equal to 'master' (or 'rawhide') for rawhide builds would be very beneficial. Mainly, because then there always exists 1:1 matching between DistGit branch names and final build dist tags. It could make things simpler here and there.
symlinks are not an available tool for us to use. many mirrors have follow symlinks turned off so parts of what you propose will be broken on the mirrors and not work as expected
what happens at branching then?
If I may add something...I think having %{dist} being equal to 'master' (or 'rawhide') for rawhide builds would be very beneficial. Mainly, because then there always exists 1:1 matching between DistGit branch names and final build dist tags. It could make things simpler here and there. what happens at branching then?
Well, if there is always the same value in master branch for %dist in the fedora-release package, at branching, a maintainer will create new branch for new Fedora in all packages (before there was mkbranch_branching script for it), then clone the fedora-release, switch to the new branch, change, value of %dist (and also %master and %fedora macros will need to be updated) to the appropriate dist tag, and pushes. Afterwards, the procedure should be the same as before.
%dist
mkbranch_branching
%master
%fedora
The big advantage of this is that in software all around, you can have just a simple mapping to get the appropriate DistGit branch and appropriate metalink URL just from a build disttag itself. @kparal knows much more where this is used. Currently, such mapping needs to be updated in the software every half-year at braching to accompany for the rawhide changing dist tags all the time and I believe that might be quite annoying for maintainers.
fedora-release package in the master branch should be mostly unchanging (unless e.g. a new distribution macro is added into Fedora). The question is what the 'Version' of the package should be in the master branch. The current value is 29. It should be kept set to that version until there is an actual change needed in the package, which won't be happening regularly as now. Also in created branches, the value of the version should not actually mean the "what Fedora we are releasing here", it will be just be a version that corresponds to the master version (representing "upstream" here). What Fedora we are releasing can be fully described by DistGit branch name only.
what would be the dist value of master branch? .fc? how would upgrade paths be ensured, how would people differentiate what was built in rawhide/master for fedora 29 and fedora 30 and fedora 50? it would seem that most packages would always have the rawhide/master disttag and users would end up with a different level of confusion as to what they have today
what would be the dist value of master branch? .fc?
dist value of master branch would be 'master'.
how would upgrade paths be ensured
I would need more precise example of a possible problem here. I guess simply by Version and Release.
how would people differentiate what was built in rawhide/master for fedora 29 and fedora 30 and fedora 50
We should just ensure always the tip of master branch is built in rawhide for each package.
it would seem that most packages would always have the rawhide/master disttag
That's the point.
...and users would end up with a different level of confusion as to what they have today
I see the current behavior around rawhide as confusing. Dist values should match DistGit branch names because then you can write simple software around it. Also, this is what one would really expect by intuition. If there are any technical reasons for the current behavior, then they should be relatively easy to resolve because the current behavior is not correct. Dist macro values in master branch should have a constant value over time because it makes things simpler.
Maybe it'd be better if we didn't have the disttag to begin with? (ducks)
In all seriousness, it'd be a big problem if we had %dist set to match the branch name. It would essentially necessitate a mass-rebuild every time we branched, and we've been trying to avoid that, though admittedly not with the greatest of success...
Maybe it'd be better if we didn't have the disttag to begin with? (ducks) In all seriousness, it'd be a big problem if we had %dist set to match the branch name. It would essentially necessitate a mass-rebuild every time we branched, and we've been trying to avoid that, though admittedly not with the greatest of success...
Well, there are mass rebuilds at each branching. Also, as I got to know recently, there are mass rebuilds even for rawhide? That's something that would be unnecessary (if Koschei is in full charge there) with what I am suggesting. Making sure that everything builds in the new branch sounds like a pretty reasonable thing to do anyway.
If you want to do a distribution without full rebuilds at particular points, then you don't need to branch at all. You just need to tag packages in the versions you want to release (with name in the tag corresponding to the distribution release), create repo out of them, test it and mirror it. The information what distribution this is would be fully included in repodata, not in individual package names. You are good with just one branch in DistGit and you can drop the disttag. That will take some time to get there so until then, I think we should focus on improving what we have (which is good in any case).
I need to admit that having constant values for all dist macros in master is probably not possible. The main problem is what %fedora macro should be set to. It cannot be 'master' because %if master < 29 in spec just gives a type error. It cannot be zero because that changes the current semantics. It could be infinity but rpm does not support something like that as far as I know. I think I was probably mistaken but it deserves a thought.
%if master < 29
I have intentionally not proposed changing %{dist} or %{fedora} exactly because of the complications it would bring.
Can we figure out which mirrors those are, and ask them to change? What is their reason for not using rsync --links?
rsync --links
IIUIC, even if they use rsync --copy-links instead of rsync --links, it should not break anything. Just the content will be multiplied (which will not matter if they use hardlinks, or will matter if they don't). It should not break any mirror.
rsync --copy-links
If we decide that this is something we don't want to do due to mirror limitations, implementing at least ideas 1) and 2) would still be very helpful. It doesn't solve all the annoyances we fight with, but at least some. But I would very much like to find a solution to all the described issues.
What is their reason for not using rsync --links? IIUIC, even if they use rsync --copy-links instead of rsync --links, it should not break anything. Just the content will be multiplied (which will not matter if they use hardlinks, or will matter if they don't). It should not break any mirror
What is their reason for not using rsync --links?
IIUIC, even if they use rsync --copy-links instead of rsync --links, it should not break anything. Just the content will be multiplied (which will not matter if they use hardlinks, or will matter if they don't). It should not break any mirror
The NAS device with which we mirror does not have a 'hardlink' binary (I have an RFE upstream to get it, but this is not at all guaranteed to be provided)
Moving to rsync for this task will require writing additional tooling for this task alone. Doable, but not immediately. What time frame for this change?
So, getting back to the orig proposal: Overall I kind of like it. Some comments/questions for @kparal
With this we could get rid of fedora-repos-rawhide. The reason we had that in the first place was to make it so users who didn't know what they were doing wouldn't mix rawhide and stable content. I'm not sure how big an issue this would be these days, but it is a concern.
How often/much/why do you use https://dl.fedoraproject.org/... links? Using those is why you want the 'all/' directory right? Do you always use dl or do you use download (a redirect)? If the use is valid and you always use dl for this we could just add redirects at the apache level and not require all mirrors to keep the all dir and links? Or does it need to be generically all mirrors?
I think we actually did have mirrormanager answering for those repos in the past (the rawhide ones by number) to make it easier to do things. I'm not sure why they arent currently. @adrian any ideas or input here?
I think we should do all of these as much as we can in mirrormanager and only change mirrors the very smallest we can.
MirrorManager has support to create repository redirects. As far as I remember it was always a manual process but the scripts to do that have not been ported to MirrorManager2 (the current version). We have not used repository redirects for a long time. I am not aware of any repository redirects in the last three years. (fedora-rawhide is actually a repository redirect to rawhide)
The main problem I see with automatic creation is that until now we only created new versions in MirrorManager when the actually appeared on the master mirror. Now we would need to create them in advance.
There is also a ticket which requests just this here: https://github.com/fedora-infra/mirrormanager2/issues/217
We can definitely add repository redirects right now by directly adding them to the database or through MirrorManagers admin panel.
We can also redirect 'updates' repositories to the 'rawhide' repository so that right now everything for *29 could go to rawhide.
I'm not sure I understand the concern. If you go through all the way with what I described in the "Future improvements" section, there would be no rawhide.repo, and so users won't be able to mix content (apart from using dnf --releasever=rawhide). There probably still would be something like fedora-repos-rawhide, but that would only contain the dnf config snippet to force $releasever=rawhide (so that you stay on rawhide forever).
rawhide.repo
dnf --releasever=rawhide
$releasever=rawhide
We tried to avoid using mirrormanager too much in the past, because it sometimes responded with HTTP 5xx code. Even if it happened once in 1000 requests, it still meant 10 failed tests per day for us. So, for tasks running in Fedora infra, we used baseurl instead (for which you need to know the paths). So we ended up with config files like this, that defines which releases are in releases/, which are in development/, etc. Those configs are hardcoded to dl.fp.o (infrastructure.fp.o internally) for our Fedora deployment, and use download.fp.o when a user runs it locally. Having the symlinks I proposed would mean we don't need to update the config the very moment branching/stable release happens, there are no intermediate errors, and our users don't have their tests broken until we push the package update through Bodhi into stable.
dl.fp.o
infrastructure.fp.o
download.fp.o
I heard that mirrormanager might not fail as much nowadays, so we could try to use it more once again, instead of using baseurls. It's still an extra middleman that increases the risk of things going wrong, but if it's now reliable enough, it might be worth it instead of always catching up with the configs. However, MM can't replace all use cases. Some tasks might be interested in content other than DNF repos (like ISOs, compose metadata, etc), so we'll need to maintain the direct links anyway.
Adding the symlinks (or http redirects) just for dl.fp.o would mean that we can use them for Fedora infra, therefore things don't break that often and there's less maintenance in there, but things would still break and needed to be updated for anyone running the task locally. I was hoping for a generic solution, yes, that would work with download.fp.o (i.e. even if not all mirrors would contain the symlinks, download.fp.o would take care of redirecting the request to a mirror that has the right file path). I'm not sure if it's worth the effort to enable this just for dl.fp.o.
If the symlinks are a no-go for some mirrors, I was also thinking about placing the symlinks elsewhere in the file hierarchy, e.g. in /pub/fedora/linux/links (a completely new top-level directory, which is unlikely to get mirrored unintentionally), which would then link to ../releases/NN, ../development/NN and ../updates(/testing)/NN. However, I'm not sure how many mirrors would pick it up, if any.
/pub/fedora/linux/links
../releases/NN
../development/NN
../updates(/testing)/NN
I think we actually did have mirrormanager answering for those repos in the past (the rawhide ones by number) to make it easier to do things. I'm not sure why they arent currently. @adrian any ideas or input here? I think we should do all of these as much as we can in mirrormanager and only change mirrors the very smallest we can.
Understood. If the changes are in MM, I'm sure I can substantially improve our workflows.
MirrorManager has support to create repository redirects. As far as I remember it was always a manual process but the scripts to do that have not been ported to MirrorManager2 (the current version). We have not used repository redirects for a long time. I am not aware of any repository redirects in the last three years. (fedora-rawhide is actually a repository redirect to rawhide) The main problem I see with automatic creation is that until now we only created new versions in MirrorManager when the actually appeared on the master mirror. Now we would need to create them in advance.
Synchronization is a pain point for all the tooling in our infra. We currently do all of it manually (periodically check until repo files are in place, etc). I was hoping this could get more atomic with this change (i.e. adding branched and setting a number redirect for rawhide only once the files are in place).
That would be great.
That sounds like a great idea.
We tried to avoid using mirrormanager too much in the past, because it sometimes responded with HTTP 5xx code. Even if it happened once in 1000 requests, it still meant 10 failed tests per day for us. So, for tasks running in Fedora infra, we used baseurl instead (for which you need to know the paths).
Hopefully there are no 500's now...
If it's outside the mirrored content, none.
We can definitely add repository redirects right now by directly adding them to the database or through MirrorManagers admin panel. That would be great. We can also redirect 'updates' repositories to the 'rawhide' repository so that right now everything for *29 could go to rawhide. That sounds like a great idea.
ok, lets do that and see what's left.
I added the following repository redirects:
> select * from repository_redirect where from_repo like '%29'; id | from_repo | to_repo -----+-----------------------------+---------------- 558 | fedora-29 | rawhide 559 | fedora-debug-29 | rawhide-debug 560 | fedora-source-29 | rawhide-source 561 | updates-released-f29 | rawhide 562 | updates-released-debug-f29 | rawhide-debug 563 | updates-released-source-f29 | rawhide-source 564 | updates-testing-f29 | rawhide 565 | updates-testing-debug-f29 | rawhide-debug 566 | updates-testing-source-f29 | rawhide-source (9 rows)
Metalink for those are working now.
Not sure what will happen with the redirects once F29 is actually branched and MirrorManager detects those actual repositories.
Thank you.
So, with those in place we are down to the items that affect the master mirrors directly, not via metalink.
Do you want to try and use the metalink for a while and see if you are seeing failures or still have needs for the direct links?
@smooge Do these changes affect stats collection at all?
Yes, since symlinks seem to be problematic for mirrors, I'll try to use metalinks as much as I can, and see it works well for us. Thanks for your help.
Closing this ticket for now and please open it again if you need anything else.
Thanks.
Metadata Update from @mohanboddu: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.