#5967 Enable source repo generation
Closed: Fixed 3 months ago by kevin. Opened 5 years ago by mizdebsk.

Please enable source repository generation in kojira, at least for rawhide (currently f22-build).

Justification: There are many use cases which require access to YUM source repository. One of them is Koschei (a continous rebuild service), which requires SRPM repo for resolving package build-dependencies.

Implementation of this feature should be simple - it just involves editing kojira.conf file


msimacek can you please attend the releng meeting at 14:30 UTC so we can discuss this please.

Replying to [comment:3 ausil]:

msimacek can you please attend the releng meeting at 14:30 UTC so we can discuss this please.

Today there's a public holiday in Czech Republic, so I don't think msimacek will be there, but I can attend to the meeting instead.

Koschei prioritizes package rebuilds according to changes in build
dependency tree. After new repo for f22-build is generated, Koschei
has to calculate differences in build deps between previous and newly
generated repo, for all packages it tracks.

Koschei generates package build dependency tree using hawkey library,
which is also used by DNF. Hawkey works with YUM metadata. In this
case metadata for both source and binary packages is needed. Source
repo contains the SRPM and info about its build requires, binary repo
contains dependency info for all RPMs which may be transitive build
deps. For binary repository Koschei simply downloads metadada for
f22-build from Koji, but there is no repo for source RPMs, so Koschei
has to construct it by itself. To do this it downloads SRPMs from Koji
and runs createrepo_c on them.

Current solution works, but it has disadvantages:

  • extra storage required for SRPMs (about 46 GiB),
  • SRPMs have to be transfered from Koji over network,
  • extra code has to be maintained.

We can keep current solution, but enabling source repo generation
for f22-build would make things considerably simpler. Additionally
I think such repository could be used in other ways too and as such it
would be worth enabling.

One problem mentioned during the meeting was arch-dependant SRPMs
(SRPMs which have different contents depending on what architecture
they have been built). SRPM contents depend not only on architecture,
but a number of other things too. Theoretically spec file can contain
line like:
{{{
%{lua: if(os.time()%2==0) then print("BuildRequires: foo"); end}
}}}
which will add BuildRequires semi-randomly. I think that there is no
way to properly fix all such cases. If some package has
non-deterministic build then it's a problem in that package and we
shouldn't try to workaround it.

due to the high cost of having the srpm repos and the issues that exist. releng does not see enough benefit at this time to fulfil this request

Metadata Update from @mizdebsk:
- Issue tagged with: meeting

2 years ago

Reopening after discussion with @kevin and @ignatenkobrain on IRC. Igor, can you add your use cases?

Metadata Update from @mizdebsk:
- Issue status updated to: Open (was: Closed)

10 months ago

My use-case is:

  • Rust crates will be rawhide-only (they are not going to be branched)
  • Rust applications will be shipped as a modules which will include building all crates and then filtering them out
  • I need repodata out of source rpms to generate modulemds
  • Waiting for compose to happen is not going to work. It requires waiting for the end of the day instead of doing continious development and sometimes it fails.

ok, some questions:

  • Is rawhide only sufficent here? Or would other branches/buildroots be needed? (For both rust sig and koschei)

  • can we see in stg how much resource this uses? How much longer are newrepo tasks?

For rust we need just rawhide. For koschei I think we ultimately need for all releases, but I think having it for rawhide would be huge improvement

From our grooming discussion on #fedora-releng channel on Apr 12 2019

remove meeting label, add 'waiting for external' and link to upstream pr if it's not already there.

https://pagure.io/koji/pull-request/1273

Metadata Update from @mohanboddu:
- Issue untagged with: meeting
- Issue tagged with: waiting on external

8 months ago

Koji pull request has been merged.

@kevin @mohanboddu so can we now backport this into our infra?

Yep. I will get it backported into the version we will use...

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

3 months ago

Login to comment on this ticket.

Metadata