PR#765: Copr - install modules into the buildroot - fm-orchestrator

		`@@ -82,10 +82,16 @@`
		`"""`
		`self.copr = self._get_copr_safe()`
		`self._create_module_safe()`
		`+ mmd = self.module.mmd()`

		`- # @FIXME Not able to use gcc-c++ in chroot (RhBug: 1440889)`
		`- packages = groups["build"] - {"gcc-c++"}`
		`- self._update_chroot(packages=list(packages))`
		`+ buildrequires = ["@{}:{}/{}".format(n, s, "buildroot")`
		`+ for n, s in mmd.buildrequires.items()]`
		`+`
		`+ buildroot_profile = mmd.profiles.get("buildroot")`
		`+ if buildroot_profile:`
		`+ buildrequires.extend(buildroot_profile.rpms)`
		`+`
		`+ self._update_chroot(packages=buildrequires)`

		`if self.copr and self.copr.projectname and self.copr.username:`
		`self.__prep = True`
		`@@ -105,6 +111,8 @@`
		`copr = self._get_copr(**kwargs)`

		`self._create_chroot_safe(copr, self.chroot)`
		`+ self.client.modify_project(copr.projectname, copr.username,`
		`+ use_bootstrap_container=True)`
		`return copr`

		`def _get_copr(self, ownername, projectname):`
		`@@ -121,11 +129,7 @@`
		`chroots=current_chroots + [chroot])`

		`def _create_module_safe(self):`
		`- from copr.exceptions import CoprRequestException`
		`-`
		`- modulemd = tempfile.mktemp()`
		`- self.module.mmd().dump(modulemd)`
		`-`
		`+ modulemd = self._dump_mmd()`
		`kwargs = {`
		`"username": self.module.copr_owner or self.owner,`
		`"projectname": self.module.copr_project or`
		`@@ -142,6 +146,18 @@`
		`finally:`
		`os.remove(modulemd)`

		`+ def _dump_mmd(self):`
		`+ # Write module's name, stream and version into the modulemd file`
		`+ # so Copr can parse it from there`
		`+ mmd = self.module.mmd()`
		`+ mmd.name = str(self.module.name)`
		`+ mmd.stream = str(self.module.stream)`
		`+ mmd.version = int(self.module.version)`
		`+`
		`+ modulemd = tempfile.mktemp()`
		`+ mmd.dump(modulemd)`
		`+ return modulemd`
		`+`
		`def buildroot_ready(self, artifacts=None):`
		`"""`
		`:param artifacts=None : a list of artifacts supposed to be in the buildroot`
		`@@ -193,8 +209,20 @@`
		`# Kojipkgs repos have been prematurely disabled without providing any`
		`# suitable alternative for Copr. This is a temporary workaround until`
		`# we figure out how to solve this permanently.`
		`- repos.append("https://kojipkgs.fedoraproject.org/compose/"`
		`- "latest-Fedora-Modular-26/compose/Server/x86_64/os/")`
		`+ compose = ("https://kojipkgs.fedoraproject.org/compose/"`
		`+ "latest-Fedora-Modular-{}/compose/Server/x86_64/os/")`
		`+`
		`+ # We need to enable copr repositories with modularity DNF`
		`+ # so we can install modules into the buildroot`
		`+ copr = ("https://copr-be.cloud.fedoraproject.org/results/"`
		`+ "@copr/{}/fedora-26-x86_64/")`
		`+`
		`+ repos.extend([`
		`+ compose.format("27"),`
		`+ compose.format("Rawhide"),`
		`+ copr.format("dnf-modularity-nightly"),`
		`+ copr.format("dnf-modularity-buildroot-deps"),`
		`+ ])`

		`self._update_chroot(repos=repos)`

		`@@ -282,8 +310,7 @@`
		`chroots=[self.chroot])`

		`def finalize(self):`
		`- modulemd = tempfile.mktemp()`
		`- self.module.mmd().dump(modulemd)`
		`+ modulemd = self._dump_mmd()`

		`# Create a module from previous project`
		`result = self.client.make_module(username=self.copr.username,`

frostyx commented 6 years ago

@jkaluza can you please review this?

In Copr, we would like to install modules into the buildroot rather than packages from the modules.
This is achievable by enabling mock's bootstrap chroot for the project and installing DNF with modularity features [1] into it.

I also added a change of how module in Copr is created. If you prefer this in separate PR, I can split it.

[1] https://copr.fedorainfracloud.org/coprs/g/copr/dnf-modularity-nightly/

ignatenkobrain commented 6 years ago

This doesnt make sense for me.. You should be using "buildroot" and "srpm-buildroot" profiles..

jkaluza commented 6 years ago

Since this is in copr builder, I don't mind merging that, but what's the motivation here?

In Koji/Mock backends, we just install what is in buildroot and srpm-buildroot profile and it works just fine. I presume the "dnf" call installing the module is somewhere in the copr code, since I don't see it in this PR. Does that dnf code installs the "buildroot" and "srpm-buildroot" profile by default in a copr buildroot? That way it would probably make sense, otherwise what profile do you install?

ignatenkobrain commented 6 years ago

@jkaluza that's basically + self._update_chroot(packages=buildrequires)

frostyx commented 6 years ago

I will try to explain you my motivation and how it works on Copr side

Motivation:
1. Since module depends on modules, it makes more sense to me install the required modules into the buildroot instead of querying their packages and installing them.
2. it will make building modules dependent on other Copr modules easier. For instance I will not have to query packages contained in the module and also I will not have to solve recursive dependencies like you do in the PDC, cause DNF will do it for me
3. I would like to use DNF modularity features before they are in fedora, so I can provide the feedback to devs

Now the implementation on Copr side.
MBS sets the dependent modules to the Copr project. That is what this PR is about, right? Those module dependencies (such as @platform:master/buildroot) are passed to the mock. This PR also enables bootstrap container for the mock and installs the modularity DNF into it, so it is able to install modules into the final buildroot.

I have a tests suite and it works fine https://pagure.io/fork/frostyx/copr/copr/blob/modularity-tests-2/f/beaker-tests/Sanity/copr-cli-basic-operations/runtest-modules.sh

ralph commented 6 years ago

See also #787. What do you think @frostyx, would that make it easier to experiment with things like this?

frostyx commented 6 years ago

Hello @ralph, thank you for joining this discussion. I've commented the #787 so we can talk about it there. What do you think about this PR? Can we merge it?

frostyx commented 6 years ago

Wait! The Bikeshed compose is gone. I need to update the PR and use different repo.

rebased onto e0d1c6e

6 years ago

frostyx commented 6 years ago

Well, the Bikeshed compose is gone, so I yesterday I tried to use F27 compose instead because MBS evaluated the dependency as @platform:f27/buildroot, but today the dependency was actually @platform:master/buildroot, so I needed to enable Rawhide compose.

Not sure what exactly is going on, but I suppose that allowing both compose repositories shouldn't hurt something. At this point building modules on Copr dev machines work again.

msuchy commented 6 years ago

ping. Any progress on merging of this one?

jkaluza commented 6 years ago

I think I'm +1 for this, since it touches only Copr plugin.

It would be great to remove the hardcoded URLs in the future and move them to configuration, so when modularity people decide they will rename the composes, you will be able to just change the configuration instead of releasing and deploying new mbs version or doing some hotfixes.

jkaluza commented 6 years ago

Now some comments on the way you are going to do that in Copr:

I will try to explain you my motivation and how it works on Copr side
Motivation:
1. Since module depends on modules, it makes more sense to me install the required modules into the buildroot instead of querying their packages and installing them.

This might be tricky, because "installing" a module does not mean installing all its packages if I'm right. It will just install default profile. But I presume you just want to enable that module, so "dnf install" later takes packages from those list of enabled modules and not from any other module from the compose.

it will make building modules dependent on other Copr modules easier. For instance I will not have to query packages contained in the module and also I will not have to solve recursive dependencies like you do in the PDC, cause DNF will do it for me

I see what you mean by that, makes sense to me.

I would like to use DNF modularity features before they are in fedora, so I can provide the feedback to devs
Now the implementation on Copr side.
MBS sets the dependent modules to the Copr project. That is what this PR is about, right? Those module dependencies (such as @platform:master/buildroot) are passed to the mock. This PR also enables bootstrap container for the mock and installs the modularity DNF into it, so it is able to install modules into the final buildroot.
I have a tests suite and it works fine https://pagure.io/fork/frostyx/copr/copr/blob/modularity-tests-2/f/beaker-tests/Sanity/copr-cli-basic-operations/runtest-modules.sh

My only question to that is: Is copr/mock configured to install packages from "buildroot" and "srpm-buildroot" when building the RPM or SRPM?

MockModuleBuilder or KojiModuleBuilder are going through the packages in "buildroot" module profile (+ package in "buildroot" profiles of all buildrequires) and just sets following in mock.cfg (or Koji tag):

config_opts['chroot_setup_cmd'] = 'install unzip tar cpio gawk xz sed findutils util-linux bash info gcc grep redhat-rpm-config fedora-modular-release diffutils make module-build-macros patch shadow-utils coreu
tils which rpm-build gzip gcc-c++ bzip2'

Does Copr do something like this too?

Edited 6 years ago by jkaluza

frostyx commented 6 years ago

Thank you @jkaluza for the feedback.

My only question to that is: Is copr/mock configured to install packages from "buildroot" and "srpm-buildroot" when building the RPM or SRPM?

I finally understand what you meant. Good point, because the answer is no. The MBS just configures Copr to install buildroot profile for all buildrequired modules, but not packages from the buildroot of the module itself.

I wasn't actually aware that it should, sorry. I will update my test suite to caption this use-case and update the PR. Thanks.

jkaluza commented 6 years ago

@Frostyx: OK :) I was thinking you don't install those packages at all anywhere. The way you do it now is probably not a big issue. It would be problem only for some low-level modules like base-runtime.

msuchy commented 6 years ago

With mock, you can combine config_opts['module_install'] and config['chroot_setup_cmd']. So you leave out chroot_setup_cmd if installing profile is engough for you. If you need additional package you can list them in chroot_setup_cmd or chroot_additional_packages.
Copr allows you to change those values, so Copr should be fine.

1 new commit added