Learn more about these different git repos.
Other Git URLs
I have an use case that I like to be addressed. I don't need a solution, I'd rather need a hint and then I can help script a proof of concept for this. I'm partially opening this to track the use case, rather than to ask somebody to work on this.
From time to time, as a "metapackager" (a packager that also packages/creates tools (such as RPM macros) or polices for other packagers) I need to run a quick grep over all Rawhide specfiles. This allows for example to analyze how often is a particular macro used in Fedora, or how it is used in most cases.
For non-modular content, a solution exist. There is a tarball with all rawhide specfiles from Fedora, generated daily-ish.
https://pkgs.fedoraproject.org/repo/rpm-specs-latest.tar.xz
With the raise of the amount of modular content, it is probably no longer desirable to ignore it completely. I'd like to get all rawhide spec files including the spec files of modular packages.
What would be the way to get all rawhide modular specfiles?
I am looking for a way a script can get the list of all Rawhide modules and all their source packages (components) for all their streams and for each such package, get the modular branch. From there, I can construct a Pagure spec URL and fetch it.
So we need to file an infra ticket with steps how to gather the files.
Metadata Update from @psabata: - Issue assigned to psabata
I just need the steps how to gather the files. I can attempt to get it done then and once ready, we can open the infra ticket.
Today, rawhide does not contain any module. I will show it on Fedora 34:
Open repodata/-modules.yaml. file from a repository. The file contains a sequence of YAML documents. You are interested in the documents with a "document: modulemd" field. For each document locate /data/artifacts/rpms list. The lists enumerates binary and source packages by NEVRA. Take the source package NEVRAs and ask Koji for buildinfo for those builds. E.g. perl-IO-Socket-SSL:2.068:3420210211145652:7f057cb7:x86_64 module has perl-IO-Socket-SSL-0:2.069-2.module_f34+11369+5aafbf50.src artifact and "koji buildinfo perl-IO-Socket-SSL-2.069-2.module_f34+11369+5aafbf50" command reports "Source: git+https://src.fedoraproject.org/rpms/perl-IO-Socket-SSL#3b3e46f57680a9c3e392058c603ff7b69d04f946". There you get the spec file.
This procedure is fast, but incomplete. It hides packages which were built in a module but filtered out from the binary module. For instance, when the package is a pure build-only RPM dependency. That's the case of perl-ExtUtils-PkgConfig package which was also built within perl-IO-Socket-SSL:2.068:3420210211145652:7f057cb7:x86_64 module but not distributed. A complete list of the module's packages is in /data/components/rpms. But it misses the dist-git references.
If you are interested in all source packages, you need to retrieve a modulemd YAML document before it was stripped. That YAML document can also be retrieved from Koji with "koji buildinfo perl-IO-Socket-SSL-2.068-3420210211145652.7f057cb7" (watch the delimiters: name-stream-version.context). There you can read a JSON document in "Extra" field which encodes the modulemd YAML document. And for the perl-ExtUtils-PkgConfig package, there is "rpms:\n perl-ExtUtils-PkgConfig:\n ref: 28befb3c5dd44b0909a8db14651bfb31ccb41b00\n" That means commit 28befb3c5dd44b0909a8db14651bfb31ccb41b00 of repository perl-ExtUtils-PkgConfig of namespace rpms in Fedora's dist-git.
If you do not want to parse JSON, you can download the module build with "koji download-build perl-IO-Socket-SSL-2.068-3420210211145652.7f057cb7 --type module" and parse "modulemd.txt" file.
Please be ware that a hostname of the git repository defaults to a value of "scmurl" field of the document. it can be overridden per component with "repository" field. But In case of Fedora, the non-Fedora dist-git is forbidden by policy and it is enforced by Fedora MBS. So you don't have deal with the overridden repositories.
These procedures used Koji. The nonstripped modulemd YAML document can also be obtained from MBS at https://mbs.fedoraproject.org/module-build-service/2/module-builds/?name=perl-IO-Socket-SSL&stream=2.068&version=3420210211145652&context=7f057cb7&state=5&verbose=1.
Thanks. Given the fact that modules are almost nonexistent, I do no longer plan to work on this. Ignoring modules altogether when dealing with Fedora spec files is easier and almost correct. The pain/gain ratio to not ignore them is not worth it.
Login to comment on this ticket.