#2082 Never-ending task due to Frontend error?
Closed: Fixed 2 years ago by praiskup. Opened 2 years ago by praiskup.

Spent 7 days already on:

[2022-02-09 14:21:33,921][WARNING][PID:707231] Retry request #423 on https://copr.fedorainfracloud.org/backend/update/: Request server error on https://copr.fedorainfracloud.org/backend/update/: 500 INTERNAL SERVER ERROR
[2022-02-09 14:56:49,420][WARNING][PID:707231] Retry request #424 on https://copr.fedorainfracloud.org/backend/update/: Request server error on https://copr.fedorainfracloud.org/backend/update/: 500 INTERNAL SERVER ERROR
[2022-02-09 15:32:10,029][WARNING][PID:707231] Retry request #425 on https://copr.fedorainfracloud.org/backend/update/: Request server error on https://copr.fedorainfracloud.org/backend/update/: 500 INTERNAL SERVER ERROR
[2022-02-09 16:07:35,205][WARNING][PID:707231] Retry request #426 on https://copr.fedorainfracloud.org/backend/update/: Request server error on https://copr.fedorainfracloud.org/backend/update/: 500 INTERNAL SERVER ERROR
[2022-02-09 16:43:05,384][WARNING][PID:707231] Retry request #427 on https://copr.fedorainfracloud.org/backend/update/: Request server error on https://copr.fedorainfracloud.org/backend/update/: 500 INTERNAL SERVER ERROR
[2022-02-09 17:18:40,808][WARNING][PID:707231] Retry request #428 on https://copr.fedorainfracloud.org/backend/update/: Request server error on https://copr.fedorainfracloud.org/backend/update/: 500 INTERNAL SERVER ERROR
...

https://download.copr.fedorainfracloud.org/results/@authselect/test:pr:1/srpm-builds/03294765/backend.log


[Fri Feb 11 14:17:01.247130 2022] [wsgi:error] [pid 920524:tid 920528] [remote 2600:1f18:8ee:ae00:d553:8ed5:d8b6:9f83:36180] ERROR:coprs.error_handlers:Admin-only exception 
[Fri Feb 11 14:17:01.247156 2022] [wsgi:error] [pid 920524:tid 920528] [remote 2600:1f18:8ee:ae00:d553:8ed5:d8b6:9f83:36180] Request: POST https://copr.fedorainfracloud.org/backend/update/ 
[Fri Feb 11 14:17:01.247164 2022] [wsgi:error] [pid 920524:tid 920528] [remote 2600:1f18:8ee:ae00:d553:8ed5:d8b6:9f83:36180] User: None 
[Fri Feb 11 14:17:01.247170 2022] [wsgi:error] [pid 920524:tid 920528] [remote 2600:1f18:8ee:ae00:d553:8ed5:d8b6:9f83:36180] Traceback (most recent call last): 
[Fri Feb 11 14:17:01.247175 2022] [wsgi:error] [pid 920524:tid 920528] [remote 2600:1f18:8ee:ae00:d553:8ed5:d8b6:9f83:36180]   File "/usr/lib/python3.10/site-packages/flask/app.py", line 1513, in full_dispatch_request 
[Fri Feb 11 14:17:01.247181 2022] [wsgi:error] [pid 920524:tid 920528] [remote 2600:1f18:8ee:ae00:d553:8ed5:d8b6:9f83:36180]     rv = self.dispatch_request() 
[Fri Feb 11 14:17:01.247187 2022] [wsgi:error] [pid 920524:tid 920528] [remote 2600:1f18:8ee:ae00:d553:8ed5:d8b6:9f83:36180]   File "/usr/lib/python3.10/site-packages/flask/app.py", line 1499, in dispatch_request 
[Fri Feb 11 14:17:01.247193 2022] [wsgi:error] [pid 920524:tid 920528] [remote 2600:1f18:8ee:ae00:d553:8ed5:d8b6:9f83:36180]     return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) 
[Fri Feb 11 14:17:01.247198 2022] [wsgi:error] [pid 920524:tid 920528] [remote 2600:1f18:8ee:ae00:d553:8ed5:d8b6:9f83:36180]   File "/usr/share/copr/coprs_frontend/coprs/views/misc.py", line 363, in decorated_function 
[Fri Feb 11 14:17:01.247204 2022] [wsgi:error] [pid 920524:tid 920528] [remote 2600:1f18:8ee:ae00:d553:8ed5:d8b6:9f83:36180]     return f(*args, **kwargs) 
[Fri Feb 11 14:17:01.247210 2022] [wsgi:error] [pid 920524:tid 920528] [remote 2600:1f18:8ee:ae00:d553:8ed5:d8b6:9f83:36180]   File "/usr/share/copr/coprs_frontend/coprs/views/backend_ns/backend_general.py", line 375, in update 
[Fri Feb 11 14:17:01.247216 2022] [wsgi:error] [pid 920524:tid 920528] [remote 2600:1f18:8ee:ae00:d553:8ed5:d8b6:9f83:36180]     logic_cls.update_state_from_dict(obj, to_update[i]) 
[Fri Feb 11 14:17:01.247243 2022] [wsgi:error] [pid 920524:tid 920528] [remote 2600:1f18:8ee:ae00:d553:8ed5:d8b6:9f83:36180]   File "/usr/share/copr/coprs_frontend/coprs/logic/builds_logic.py", line 979, in update_state_from_dict 
[Fri Feb 11 14:17:01.247250 2022] [wsgi:error] [pid 920524:tid 920528] [remote 2600:1f18:8ee:ae00:d553:8ed5:d8b6:9f83:36180]     if not cls.assign_buildchroots_from_package( 
[Fri Feb 11 14:17:01.247256 2022] [wsgi:error] [pid 920524:tid 920528] [remote 2600:1f18:8ee:ae00:d553:8ed5:d8b6:9f83:36180]   File "/usr/share/copr/coprs_frontend/coprs/logic/builds_logic.py", line 736, in assign_buildchroots_from_pack
[Fri Feb 11 14:17:01.247262 2022] [wsgi:error] [pid 920524:tid 920528] [remote 2600:1f18:8ee:ae00:d553:8ed5:d8b6:9f83:36180]     build.package.chroots, 
[Fri Feb 11 14:17:01.247267 2022] [wsgi:error] [pid 920524:tid 920528] [remote 2600:1f18:8ee:ae00:d553:8ed5:d8b6:9f83:36180]   File "/usr/share/copr/coprs_frontend/coprs/models.py", line 936, in chroots 
[Fri Feb 11 14:17:01.247272 2022] [wsgi:error] [pid 920524:tid 920528] [remote 2600:1f18:8ee:ae00:d553:8ed5:d8b6:9f83:36180]     return self.main_pkg.chroots 
[Fri Feb 11 14:17:01.247277 2022] [wsgi:error] [pid 920524:tid 920528] [remote 2600:1f18:8ee:ae00:d553:8ed5:d8b6:9f83:36180] AttributeError: 'NoneType' object has no attribute 'chroots' 

And seems like the project was deleted.

Can anyone take a look?

WARNING: please don't cleanup the build or the worker till we know how this happened

And seems like the project was deleted.

This can also be a clue because there is supposed to be a protection not allowing to delete projects with unfinished builds

Metadata Update from @praiskup:
- Issue assigned to frostyx

2 years ago

File "/usr/share/copr/coprs_frontend/coprs/models.py", line 936, in chroots
return self.main_pkg.chroots
AttributeError: 'NoneType' object has no attribute 'chroots'

I am not able to reproduce this - every time I somehow (e.g. by EOL-ing the chroots) manage to disable all chroots while a build is running and building SRPM, the build is marked as failed, not running infinitely.

But maybe this is some mishap that will be fixed by PR#2205?

Ah, yes. I finally reproduced the error.

First, create a project with just one chroot, then run the following code

from copr.v3 import Client
client = Client.create_from_config_file()
client.build_proxy.create_from_scm(
    "frostyx",
    "never-ending-task",
    clone_url="https://pagure.io/copr/copr.git",
    subdirectory="cli",
    source_build_method="tito",
    project_dirname="never-ending-task:custom",
)

A build is created in the :custom CoprDir. I suppose instead of this step, one can create a pagure PR or something else that creates a custom directory as well.

Once the SRPM build is running, go to the project settings, disable the current chroot, enable another one, and save.

The build now runs infinitely, and I have the same error messages on backend and frontend.

But maybe this is some mishap that will be fixed by PR#2205?

This PR seems to be currently deployed on copr-fe-dev, and I am unable to reproduce the issue there. Once the SRPM build is finished, it picks up the chroot change and builds for the new chroot (which is IMHO correct behavior).

This PR seems to be currently deployed on copr-fe-dev, and I am unable to reproduce the issue there.

Ok, cool, I mentioned this issue as "Fixes: " in the corresponding PR.

Login to comment on this ticket.

Metadata