#1905 RFE: Add support for spawning multiple builds from one buildSRPMFromSCM task
Closed: Dropped 3 years ago by tkopecek. Opened 4 years ago by ignatenkobrain.

The use-case is following. There might be multiple spec files in one git repo and we would like to build all of them at the same time. make srpm can generate multiple src.rpm files and I think it should not be very complicated to spawn multiple build tasks if multiple SRPM found. I would love to help with the code, but I don't have enough knowledge of koji for that =(

Would it be possible to mark this for 1.21?


@tkopecek @mikem is there anything in the Koji data model that assumes that buildSRPMFromSCM generates exactly one build?

@ignatenkobrain a couple of questions about your use case... would there be any dependency between these builds? Is there any expectation that if one build fails then all of them fail?

Metadata Update from @dgregor:
- Custom field Size adjusted to None

4 years ago

@dgregor - yes, If multiple srpms are found in the output (or specs in input), build will fail. There is no clear rule, which srpm is the right one in such case.

If we would like to build all of them, it could be rewriten. How many such rpms does exist? What is typical usecase for this? Does it make sense to allow it for some tags and not for other ones?

https://pagure.io/koji/blob/master/f/builder/kojid#_1995 (and for specs few lines above)

Metadata Update from @tkopecek:
- Issue tagged with: discussion, feature

4 years ago

As I said, internally in our company we have one repo with tens of spec files (because that is simply cheaper to buy few private repos from github :moneybag: ).

Another case which comes to my mind in Fedora is: compat packages, you could have multiple spec files in one repo, python-sqlalchemy.spec, python-sqlalchemy_0.5.spec and it could trigger multiple builds.

Ofc, I can create buildCompany task and have some random parameter passed into it with package name, and spawn hundreds of such tasks, but it would be much easier if koji would do that, right?

@kevin can you comment on the Fedora use case mentioned in the previous comment?

I'm hesitant to support a change that is significant (both code and behavior) unless there is value to a significant part of the Koji user base. Perhaps this would be a good topic for discussion at the next Koji community meeting (second week in February).

This is actually a feature OBS has, and it'd be interesting to have for Koji.

There are two variants of this feature in OBS:

  • Multiple specs + sources in one "source repo". The source repo can be used to build multiple source packages, and it enumerates source package entries for it. rust-packaging in openSUSE:Factory is an example of this.

  • Single spec with multiple "flavor" definitions. The source repo contains a spec file plus a file that describes its variants. These variants are enumerated to generate source package entries for it and those are built accordingly. meson in openSUSE:Factory is an example of this.

Yeah, I am not sure I see the use cases myself... whats the advantage of multiple spec files in the same git repo over one per repo? Wouldn't you want to normally change those things seperately anyhow?

The main case would be if they both used the same source tarball. It would make bumping them and keeping them in sync a lot simpler.

Alternatively, the multi-build variation of this would allow you to still have a single spec file that would support building multiple source packages with different configurations as needed (e.g. test suite on or not). This is a more package-centric version of the capability that module builds provide...

Thanks ignatenkobrain for bringing this feature request to my attention. One use case for this is building non-mingw and mingw packages from the same dist-git repo without having over-complex spec files. The packages would share the same source tarballs.

An issue I'm not clear on is what happens if one of the builds fails? In the mingw case it might be that the non-mingw build succeeds and the mingw build fails (perhaps upstream didn't test the mingw case). It would be nice if the successful build could continue to bodhi in this case.

@rjones I think this is just matter of configuration, same as you have today possibility if one arch fails, still submit update (if koji is configured that way).

@tkopecek any chance this could go in some near future into koji?

I think, it should trigger multiple builds, not one. Build is 1:1 with nvr, so it would be one parent task with multiple "almost classic build" subtasks with option of specifying which spec file from checkout should be used. It looks okish for me.

Other way is to hack fedpkg (or any other source generation script) to select proper spec and delete others) Maybe we can pass some option/env_variable there (to SCM checkout script) which would select proper spec if called in specific way. Then we don't need any new tasks?

@mikem ?

Metadata Update from @tkopecek:
- Issue set to the milestone: 1.22

4 years ago

Other way is to hack fedpkg (or any other source generation script) to select proper spec and delete others) Maybe we can pass some option/env_variable there (to SCM checkout script) which would select proper spec if called in specific way. Then we don't need any new tasks?

In this case, you would need manually to spawn all tasks for each spec... Which is PITA.

Other way is to hack fedpkg (or any other source generation script) to select proper spec and delete others) Maybe we can pass some option/env_variable there (to SCM checkout script) which would select proper spec if called in specific way. Then we don't need any new tasks?

Although even this would be acceptable for me.

Adding a multi-build feature to Koji is a fairly significant process change and not something that should be shoe-horned in by hacking buildSRPMFromSCM.
The 1:1 connection between builds and tasks is longstanding behavior, and while there is nothing obvious in the schema that enforces it, I would be extremely cautious about changing that. It would be less invasive to have a new task type to govern this.
Longer term, we expect to have more integration with MBS, which would mean some sort of multi-build task.

In the short term, something like #2014 seems like a reasonable loosening of the existing sanity check.

Metadata Update from @julian8628:
- Issue tagged with: testing-ready

3 years ago

Metadata Update from @mfilip:
- Issue tagged with: testing-done

3 years ago

Commit b88d87e relates to this ticket

Commit 8cd79f4 relates to this ticket

PR #2257 is merged for 1.22, but does not implement this feature as stated. As my comment above indicates, I don't believe that this is the right way to implement a multi-build feature.

Dropping this issue from 1.22 for now. I expect it will be closed in favor of a future proposal.

Metadata Update from @mikem:
- Issue set to the milestone: None (was: 1.22)

3 years ago

Metadata Update from @tkopecek:
- Issue close_status updated to: Dropped
- Issue status updated to: Closed (was: Open)

3 years ago

Login to comment on this ticket.

Metadata
Related Pull Requests
  • #2257 Merged 3 years ago