#1266 RFE: --with-separate-src for build-repos
Closed: Fixed 3 years ago by mikem. Opened 4 years ago by ignatenkobrain.

In Fedora we want to have src in build repos, but in separate repo so the size of metadata for builds won't grow.


I have a working code and I will submit a pull request tomorrow.

Metadata Update from @mikem:
- Issue tagged with: discussion

4 years ago

For context, here is the current state of things:

  • normal Koji repos (not dist repos) default to not including srpms
  • there is a src option to NewRepo() that causes srpms to be included in each repo (i.e. for each arch)
  • kojira has a source_tags option that causes it to pass the src option to NewRepo for any tags that match the specified patterns.
  • kojira also has a with_src option that claims to globally cause source inclusion, but does not actually do so. This option appears to be an old artifact. It doesn't look like kojira ever honored it.
  • This is an old and rarely used feature

Use of Koji's repos for anything other than generating buildroots is basically "off-label" use. Koji makes repos for itself to use, and srpms are not needed to perform builds. That said, modest concessions to practicality are fine.

When this was first suggested to me, my first thought was why not use dist repos? Those are more full featured. I believe the reason was a combination of overhead and lack of automatic regeneration.

Could someone confirm my recollection and perhaps give some more details about the reasons for this request?

Metadata Update from @mikem:
- Issue tagged with: feature

4 years ago

Could someone confirm my recollection and perhaps give some more details about the reasons for this request?

Basically yes. dist-repos are expensive and we don't do them for f30-build/f31-build tags because nobody should consume them. But we do build-repos and having repos with source RPMs are very useful (e.g. to query BuildRequires and such).

How widely do you expect to use the feature? All tags? Just a few? If it is a limited set of tags, how often do you expect the set of tags to change?

I agree with everything @mikem said, except one part which says "srpms are not needed to perform builds". With modularity, we want to build individual packages in Koji and then build modules out of these packages. Modules are expected to contain set of API packages with their transitive runtime and build dependencies, and nothing more. Information about build requires is contained only in SRPMs. With this use case information contained in SRPMs is essential to generate minimal module constituent set of packages. Therefore, SRPMs are needed to build modules.

My intended use of this feature is to allow getting src repodata for set of builds corresponding to given build repo. Source repos need to be in sync with build repos. It would be possible to generate a dist-repo (with exact eventID specified, matching eventID of build repo) for each build repo, but that feels like a hack and not proper solution.

Different software needs this feature, including Koschei, scripts used for generating Java modules, similar scripts used by @ignatenkobrain for different types of packages. Lack of this feature can be worked around by either of the following:

  • using Koji XML-RPC API to query all latest SRPMs in a tag and get their build requires (getRPMDeps multicall) - this approach is used by Koschei, it is slow and puts needless load onto Koji hub
  • downloading all SRPMs from kojifiles and building repodata locally - it is slow, consumes lots of network bandwidth and local storage
  • using source repos from different sources (eg. from pungi composes) - such source repos don't match contents of build repos, so using them would often give wrong results

Each of the above workarounds has different drawbacks, that would be solved by generating proper src build repos by Koji.

If this feature is accepted then I would use it in:

  • Fedora Koji, for a very limited set of tags (likely just a single tag, at least for now); the set of tags is expected to be changed approximately once every six months
  • public Koji deployments used to develop Fedora modules, for all tags
  • private Koji deployments, for all tags

Metadata Update from @tkopecek:
- Custom field Size adjusted to None
- Issue set to the milestone: 1.18

3 years ago

Login to comment on this ticket.

Metadata
Related Pull Requests
  • #1273 Merged 3 years ago