There's a possibility that copr_chroot is None in /usr/share/copr/coprs_frontend/coprs/views/backend_ns/backend_general.py

Can you debug how this can happen?

I have added some logging and looks like builds 1236107 and 1224541 are somehow broken because the errors repeat them over and over again:

2020-02-16 17:45:37,464 [ERROR][/usr/share/copr/coprs_frontend/coprs/views/backend_ns/backend_general.py:121|backend_general:get_build_record] {'_sa_instance_state': <sqlalchemy.orm.state.InstanceState object at 0x7efc42e0c510>, 'git_hash': 'aea82efa29e07eb631fc060bc281de8af59d72f9', 'mock_chroot_id': 75, 'started_on': None, 'ended_on': None, 'build_requires': None, 'build_id': 1236107, 'status': 4, 'result_dir': '', 'build': <Build 1236107>, 'mock_chroot': <MockChroot 75>}
2020-02-16 17:45:57,681 [ERROR][/usr/share/copr/coprs_frontend/coprs/views/backend_ns/backend_general.py:121|backend_general:get_build_record] {'_sa_instance_state': <sqlalchemy.orm.state.InstanceState object at 0x7efc42e28410>, 'git_hash': '5bf3220e4e4b0e829dd0112f3faa8e8d8f28b824', 'mock_chroot_id': 82, 'started_on': None, 'ended_on': None, 'build_requires': None, 'build_id': 1224541, 'status': 4, 'result_dir': '', 'build': <Build 1224541>, 'mock_chroot': <MockChroot 82>}

The easiest way to manually reproduce this on production is

[root@copr-fe ~][PROD]# sudo -u copr-fe /usr/bin/copr-frontend shell
>>> from coprs import models
>>> models.CoprChroot.query.join(models.MockChroot).filter(models.CoprChroot.copr_id == 29464).filter(models.MockChroot.id == 75).one()

Okay, I cracked it. The problem is, that the project has initially enabled

• centos-stream-x86_64
• epel-8-x86_64
• fedora-30-x86_64
• fedora-31-x86_64
• fedora-rawhide-x86_64

then submitted a build 1236107 for all of them. Then in the meantime disabled everything except fedora-rawhide-x86_64. So the build is stuck in the pending state for the rest of them.

I guess it will be same for 1224541

I canceled the pending chroots

coprdb=# update build_chroot set status=2 where build_id=1236107 and status=4;
UPDATE 4
coprdb=# update build_chroot set status=2 where build_id=1224541 and status=4;
UPDATE 1

I guess this particular issue is fixed and I don't think that backend_general.py is not the correct place to make sure this won't happen again. I think we should update the code to automatically cancel the nonfinished chroots if the project is edited and they are not enabled anymore. What do you think?

I think we should update the code to automatically cancel the nonfinished chroots if the project is edited and they are not enabled anymore.

Could we filter-out related pending jobs instead? I'm not sure off-hand how difficult it would be.

Could we filter-out related pending jobs instead? I'm not sure off-hand how difficult it would be.

I am not sure whether I understand it correctly. But if you want just an easyfix, I would simply go with

- if copr_chroot.module_toggle_array:
+ if copr_chroot and copr_chroot.module_toggle_array:

It would be much easier and IMHO faster, than querying already filtered chroots from the database.

Perhaps, but what did you mean by:

I think we should update the code to automatically cancel the nonfinished chroots if the project is edited and they are not enabled anymore.

Ok, I looked more deeply into this ... I agree that we need the easy-fix:

- if copr_chroot.module_toggle_array:
+ if copr_chroot and copr_chroot.module_toggle_array:

And it will likely lead to build failure, which is just fine. On top of that,
I think you are right -> once the CoprChroot is disabled, we should
fail (not cancel..) the relevant BuildChroot.

But I'd appeal to resolve #1239 which is the core issue here I think. If we had
CoprChroot <- BuildChroot it would be very trivial to handle the related
stuff in BuildChroot (perhaps delete the BuildChroots with CoprChroots?).

(perhaps delete the BuildChroots with CoprChroots?).

This would, btw., disallow people to delete CoprChroots when there are some
related pending BuildChroots; (we can not delete builds which are not finished)
... which sounds very neat to me.

This would, btw., disallow people to delete CoprChroots when there are some
related pending BuildChroots; (we can not delete builds which are not finished)
... which sounds very neat to me.

Yea, this sounds to me like a good approach

Commit aa37a89 fixes this issue

Commit 6eebe92 relates to this ticket