I'm going to work on this, but first I'd like to ask if there are any known issues with building RPMs in containers I should be aware of - in case its not even worth pursuing

No, I think it's a good idea.

https://bugzilla.redhat.com/show_bug.cgi?id=1336750
https://bugzilla.redhat.com/show_bug.cgi?id=1334701
(I'll post more links related to this topic, if I'll remember ...)

Basically the blocker is: in OpenShift you'll never get privileged container (as far as I understand the security issues), and it prevents you from using mock there. Using docker in isolation doesn't make sense, there's systemd-nspawn support in mock .. but you still need to run this on VM.

Its fairly easy to run a privileged container in Openshift: https://docs.openshift.com/container-platform/3.5/admin_guide/manage_scc.html#grant-access-to-the-privileged-scc.

Last comments in https://bugzilla.redhat.com/show_bug.cgi?id=1336750 mention that mock needs just SYS_ADMIN privilege - I'll play with that again.

Using docker in isolation doesn't make sense, there's systemd-nspawn support in mock

Correct, although the plan is to make use of Openshift's quotas and limits to make sure the system is always stable. It would also allow us to scale system easily. systemd-nspawn support is not related, but also it won't stand in our way.

Note, that initially I'm planninf to make copr run for debug purposes using docker-compose - there is a long way until actual Openshift deployment and security discussions

I don't claim it is "techincal" problem, that's "security" problem, and unless you start your own OpenShift for copr purposes, you'll never get generally available OpenShift instance allowing you to run copr builders there... (and it makes this pretty useless feature at least from my POV).
Without virtualization, granting copr users to access mock in --privileged container means that users can arbitrarily and trivially control the whole openshift cloud.

The good step forward is to actually fix mock to work properly in non-privileged container first.

the good step forward is to actually fix mock to work properly in non-privileged container first.

Of course such mock wouldn't be able to build any package in the wild, but only particular subset of all packages.

you'll never get generally available OpenShift instance allowing you to run copr builders there...

I can always setup my own instance with whatever permissions are required, that shouldn't be a problem.

granting copr users to access mock in --privileged container means that users can arbitrarily and trivially control the whole openshift cloud

While "arbitrarily" is debatable it surely isn't "trivial"

The good step forward is to actually fix mock to work properly in non-privileged container first.

I'll check if SYS_ADMIN is sufficient for mock to do the job - that should be safe enough, as its a usernamespaced SYS_ADMIN

I can always setup my own instance with whatever permissions are required,
that shouldn't be a problem.

Well, in copr-world we need to be slightly more careful than we are in Koji
world (for example) because anybody can build there; the builders need to be
perfectly protected so that Bob user can't affect Alice's builds... never.

So while I admit it is possible to have your own kube cloud, it would be rather
development stuff -- not production solution. But yeah, we need to move
somewhere so +1 for any progress this way. I just answered your question:

I'm going to work on this, but first I'd like to ask if there are any known issues with building RPMs in containers I should be aware of

I agree that this is premature. Mock still does not run flawlessly in docker. See https://bugzilla.redhat.com/show_bug.cgi?id=1416813
and relevant https://bugzilla.redhat.com/show_bug.cgi?id=1336750
I would welcome some testing and patches and documentation to Mock itself. And only when everything is settled down with Mock we can think about Copr.

This is a chicken-and-egg problem - its very inconvenient to test Mock in docker without a proper frontend and options (read: Copr). And vice versa - its pointless to go on with 'builders in Openshift' when mock in docker is poorly tested.

Anyway, in https://pagure.io/fork/vrutkovs/copr/copr/branch/docker-compose I've got minimal copr setup. Mock is being run locally in 'backend' container, this should be sufficient to try it out. Any feedback is welcome.

This is a chicken-and-egg problem - its very inconvenient to test Mock in docker without a proper frontend and options (read: Copr).

This is not chicken-egg problem. Mock is and always was naturally used by package maintainers without Copr frontend (not inconvenient).

I agree that this is premature. Mock still does not run flawlessly in docker. See https://bugzilla.redhat.com/show_bug.cgi?id=1416813
and relevant https://bugzilla.redhat.com/show_bug.cgi?id=1336750
I would welcome some testing and patches and documentation to Mock itself. And only when everything is settled down with Mock we can think about Copr.

This is really a nice work. The first commit could be made as PR immediately. It will actually break testing framework right now, but fixing it should be relatively easy.

Note, however, that I would actually much prefer to use Ansible for the container setup instead of Dockerfiles and docker-compose.

If you know ansible or you are willing to learn it, you can do it yourself. Otherwise you can make a PR immediately and we will continue on that.

The second commit (the dist-git one) might be used only in case we will stick to actually "building srpms" on the dist-git machine. Currently, we want only fetch spec, fetch sources, and import it all to dist-git without actually building anything (MockSCM provider is currently an example of that).

What you can focus on, however, is making copr-rpmbuild run in a openshift container.

Note, however, that I would actually much prefer to use Ansible for the container setup instead of Dockerfiles and docker-compose.

I agree, however ansible-container is not yet stable now. I think it would be a good idea to do that when it hits 1.0 milestone

The second commit (the dist-git one) might be used only in case we will stick to actually "building srpms" on the dist-git machine. Currently, we want only fetch spec, fetch sources, and import it all to dist-git without actually building anything (MockSCM provider is currently an example of that).

Right, although this behaviour is configurable - its just this container is setting it by default. I'll think about a better way to enable/disable this.

What you can focus on, however, is making copr-rpmbuild run in a openshift container.

Right, that's my current focus atm.

Note, however, that I would actually much prefer to use Ansible for the container setup instead of Dockerfiles and docker-compose.

I agree, however ansible-container is not yet stable now. I think it would be a good idea to do that when it hits 1.0 milestone

I was actually thinking about using this http://docs.ansible.com/ansible/docker_container_module.html#docker-container but feel free to use anything else. Also you may just send the PR now and we can continue on it after merge.

The second commit (the dist-git one) might be used only in case we will stick to actually "building srpms" on the dist-git machine. Currently, we want only fetch spec, fetch sources, and import it all to dist-git without actually building anything (MockSCM provider is currently an example of that).

Right, although this behaviour is configurable - its just this container is setting it by default. I'll think about a better way to enable/disable this.

We won't be using containers on dist-git (likely). They were needed for security when we were building srpms as root inside mock chroot. This is no longer happening and the code can be rewritten to something more simple.

I was actually thinking about using this http://docs.ansible.com/ansible/docker_container_module.html#docker-container but feel free to use anything else

Oh, I see - the plan is replace docker-compose with ansible-based orchestration? That's a good idea, although its not really handy for local development. I'll look into this, but this is not a priority yet.

We won't be using containers on dist-git (likely)

Oh, I see. I started this when this code was still in place, sure, I'll rewrite that part

Copr's parts can be run in containers for some time. The builders are still problematic. And even after 3 years, Mock still has issues running in the container. Although the setup is fully Ansible-ized.

I do not expect any more work on this one.

Agreed, with one fix ....

Copr's parts can be run in containers for some time. The builders are still problematic. And even after 3 years, Mock still has issues running in the container.

Mock doesn't have the issue anymore, speaking of rootless podman containers.
https://github.com/rpm-software-management/mock/wiki#mock-inside-podman-fedora-toolbox-or-docker-container

But still it isn't solution for all packages.