#1441 Implement project-linking & build trigger strategy
Closed: MIGRATED a year ago by nikromen. Opened 3 years ago by pgfed.

I'm migrating from *Suse -> Fedora.

Among many years of habits & workflow -- both good & bad -- I used the openSUSE Build Service (https://build.opensuse.org/) to make patches/tweaks to official packages, building & installing my own modified instances of the packages.

I'm attempting to replicate similar workflow/functionality in Fedora-land, using COPR.

TL;DR: Requesting a COPR implementation similar to OBS's online project-linking & build-strategies.

The use-case I'm after is simple; e.g.,

I want to use/stay up-to-date with any of Fedora's 'official' package builds, add a patch, and build an installable pkg instance for my own distribution/use. Ideally, all ONLINE 'within' the COPR ui.

IIUC, the 'easiest' current option is to grab the spec directly from src/koji/bodhi, make my mods locally, and push to my COPR instance.
Regular/timed pulls @ my desktop check for updates to original source, at which point I reapply mods/patches, and push again to COPR, retriggering the build.

In brief chat on #irc, I was told that no such UI capability currently exists; hence this RFE.

I was pointed to

https://docs.pagure.org/copr.copr/custom_source_method.html

but haven't given that a whirl yet.

OBS provides the concept of project linking,

 openSUSE:Build Service Concept project linking
  https://en.opensuse.org/openSUSE:Build_Service_Concept_project_linking

which allows -- completely within the online OBS UI -- to 'link' to an existing 'official' project and make mods to its build spec and env. One can link either a single package, or clone a complete project.

Then in 'my' space, I can either

(1) make direct edits to the spec,

or

(2) add patches that the build exec applies

On exec, packages are built, & schedulers push results to the repos.

Here, e.g., is a "_link" to upstream's 'gcc-10' package,

https://build.opensuse.org/package/show/home:pgnd:devel:gcc10/gcc10?expand=0

to which I currently add a trivial patch.

With OBS's "Rebuild mechanisms", I've set the project build strategy/schedule,

 openSUSE:Build Service Concept build scheduling strategies
  https://en.opensuse.org/openSUSE:Build_Service_Concept_build_scheduling_strategies

to, @

https://build.opensuse.org/projects/home:pgnd:devel:gcc10/meta

==> <repository name="openSUSE_Leap_15.2" rebuild="transitive" block="local" linkedbuild="all">
^^^^^^^^^^

which automatically triggers a rebuild, on OBS, when upstream sources change.

With this^ setup, my "modified" pacakges stay up to date with official (or whatever specific versions/repos I've linked) releases, and appear for update when done in the respective subscribed repos.

Once setup, there's no further interaction required; if there's a problem with a build -- e.g., a server issue, a breaking change in sources, etc. I'm notified ...

Finally, all of that^ can also be setup/managed from my local shell via OBS's 'osc' cli tools.


Thank you for this RFE! It was a good reading to understand your use-case.

'link' to an existing 'official' project and make mods to its build spec and env. One can link either a single package, or clone a complete project.

By upstream, in Fedora land, you mean src.fedoraproject.org (not a random
other copr project). As long as you only need to depend on this,
build triggering is already solved via fedora-messaging bus (all package
change events are sent over the bus to copr, and your packages react on
the events, and trigger builds).

The thing is that we have no way to apply changes, which is tough to
design right a declarative way (unless we delegate everything to the
end-user scripting).

(1) make direct edits to the spec,

What does this mean? Some declarative patching, or manual patch file, or
script?

(2) add patches that the build exec applies

Does this require spec file change ^^^, or how this is done?

I was pointed to
https://docs.pagure.org/copr.copr/custom_source_method.html

This allows you to do anything you want, but you can not trigger the
custom builds by src.fedoraproject.org changes. Custom builds can only be
triggered by custom webhook.

(Well, there's an ugly option -- you can mirror out (e.g. to GitHub) the
src.fedoraproject.org respository (but you need to be the project admin to
setup the mirroring). And from the external mirror you can trigger custom
webhooks, and do whatever you want...)

The problem of the current copr code is that webhooks are tightly coupled
with concrete SCM methods (you can not trigger SCM build by custom method,
or custom method by fedora-messaging events from Pagure). The other
problem is that none source method, except for the Custom one, allows you
to do additional scripting (we need to fix #534 first, so we add the
patching in one place).

Another thing is that the source methods work with src.rpms, not just with
just sources, or git (if we added trivial script hook support, you only
would be able to extract the source RPM, modify sources, and re-create
it). So we probably need to

  • Support custom scripts that are executed after cloning the sources, and
    before executing the source method logic. So you can tweak things
    before the tarball is generated from git.
  • Make the source methods only generate sources, not source RPMs.
  • Support custom scripts that are executed after the source method
    logic. This will allow us to do changes to the generated spec files, e.g.
  • And at the end, once the script ^ has run, we should create src.rpm (or
    any other archive format, tarball would be just enough) used for transporting
    the generated sources into copr dist-git.

Metadata Update from @praiskup:
- Issue tagged with: RFE

3 years ago

By upstream, in Fedora land, you mean src.fedoraproject.org

yes

(not a random other copr project).

actually, yes to that, too.

again, with the ack that I'm 'comparing' to OBS, and what my workflow 'is used to', it's a trivial exercise to link to any other package/repo in any *suse repo, whether 'official' or not.

As long as you only need to depend on this,
build triggering is already solved via fedora-messaging bus (all package
change events are sent over the bus to copr, and your packages react on
the events, and trigger builds).

I do have pagure.io -> copr auto-rebuild triggering set up and functioning.

pushes to my pagure.io package repo do trigger a rebuild @ COPR.

I have not (yet) seen that, e.g., changes to the build chroot's mock env (e.g., with .so version bumps to system libs, etc) automatically trigger a rebuld. Assuming that the mock env corretly tracks the release -- i.e., the 'same' env that a dnf update has installed on my local box -- then, that is something that I typically depend on as well.

(1) make direct edits to the spec,

What does this mean? Some declarative patching, or manual patch file, or
script?

yes, yes & yes.

repeating a tangible example,

here's a patch I apply to 'my' GCC10 build project, linked to 'upstream',

https://build.opensuse.org/package/view_file/home:pgnd:devel:gcc10/gcc10/gcc10.spec.patch?expand=0

it can be created/edit/uploaded directly into the OBS, via UI, or from cmd-line/local edits + git push.

(2) add patches that the build exec applies

Does this require spec file change ^^^, or how this is done?

that patch is then defined for use in the '_link' file,

https://build.opensuse.org/package/view_file/home:pgnd:devel:gcc10/gcc10/_link?expand=0

and applied at build time

this doesn't require a 'spec file change' ... although you can also directly edit the spec file in the UI's text-editor; that creates a version of the link'd diff in 'my' project.

those edits can be as simple as just adding a patch, or can rip-out/replace entire sections of the spec, or, of course, replace it completely.

I was pointed to
https://docs.pagure.org/copr.copr/custom_source_method.html

This allows you to do anything you want, but you can not trigger the
custom builds by src.fedoraproject.org changes. Custom builds can only be
triggered by custom webhook.

(snip)

I'm still figuring out all of what's doable; this helps a bit, thx.

Even that which is, though, is not (yet) 'simply' doable via the UI ...

e.g.., setting up a pagure.io project, pushing a .spec to it, linking it to a COPR project, enabling auto-rebuild, turning on fedmsg hooks, etc ... that's "a lot" of steps needed. and that are frequently repeated.

for comparison with 'my' old workflow?

create a 'ONE liner', e.g.

cat _link
    <link project="devel:gcc" package="gcc10" >

done.

  • Support custom scripts that are executed after cloning the sources, and
    before executing the source method logic. So you can tweak things
    before the tarball is generated from git.
  • Make the source methods only generate sources, not source RPMs.
  • Support custom scripts that are executed after the source method
    logic. This will allow us to do changes to the generated spec files, e.g.
  • And at the end, once the script ^ has run, we should create src.rpm (or
    any other archive format, tarball would be just enough) used for transporting
    the generated sources into copr dist-git.

i'm clear that this is not a trivial undertaking!

i would strongly suggest that, if there's the slightest interest in these sorts of 'improvements', that creating an OBS account and trying it is a far better experience than depending on my terse/incomplete descriptions.

btw, you can build Fedora packages, for targeted release, there as well; not well-integrated with the 'rest' of fedora-land, of course. there are plenty of reasons for me that I'm moving to Fedora/RH, and choosing to use the tools 'here'; that said -- and the reason for this RFE -- is that it's been, and continues to be, a very inefficent experience 'here', for me so far.

and I think it could be improved.

hth.

Metadata Update from @nikromen:
- Issue close_status updated to: MIGRATED
- Issue status updated to: Closed (was: Open)

a year ago

Login to comment on this ticket.

Metadata