#1531 Build 1654724 has source_status succeeded, but no build_chroots
Closed: Fixed 3 years ago by praiskup. Opened 3 years ago by praiskup.

E.g. here:
https://copr.fedorainfracloud.org/coprs/cleber/avocado-copr-test/package/python-avocado/

Those builds are all in waiting state, while those should be processed.


While we are on this bug, we should make sure that we fix the problems with the DB, the frontend.log basically contains only this error:

2020-10-05 07:14:38,934 [ERROR][/usr/share/copr/coprs_frontend/coprs/models.py:1168|models:status] Build 744032 has source_status succeeded, but no build_chroots
2020-10-05 07:19:05,982 [ERROR][/usr/share/copr/coprs_frontend/coprs/models.py:1168|models:status] Build 1085446 has source_status succeeded, but no build_chroots
2020-10-05 07:19:05,982 [ERROR][/usr/share/copr/coprs_frontend/coprs/models.py:1168|models:status] Build 1085446 has source_status succeeded, but no build_chroots
2020-10-05 07:19:05,982 [ERROR][/usr/share/copr/coprs_frontend/coprs/models.py:1168|models:status] Build 1085446 has source_status succeeded, but no build_chroots
2020-10-05 07:19:05,982 [ERROR][/usr/share/copr/coprs_frontend/coprs/models.py:1168|models:status] Build 1085446 has source_status succeeded, but no build_chroots
2020-10-05 07:36:37,642 [ERROR][/usr/share/copr/coprs_frontend/coprs/models.py:1168|models:status] Build 796875 has source_status succeeded, but no build_chroots
2020-10-05 07:36:37,642 [ERROR][/usr/share/copr/coprs_frontend/coprs/models.py:1168|models:status] Build 796875 has source_status succeeded, but no build_chroots
2020-10-05 08:00:37,187 [ERROR][/usr/share/copr/coprs_frontend/coprs/models.py:1168|models:status] Build 1344345 has source_status succeeded, but no build_chroots
2020-10-05 08:00:37,187 [ERROR][/usr/share/copr/coprs_frontend/coprs/models.py:1168|models:status] Build 1344345 has source_status succeeded, but no build_chroots
2020-10-05 08:02:03,236 [ERROR][/usr/share/copr/coprs_frontend/coprs/models.py:1168|models:status] Build 1344345 has source_status succeeded, but no build_chroots
2020-10-05 08:02:03,236 [ERROR][/usr/share/copr/coprs_frontend/coprs/models.py:1168|models:status] Build 1344345 has source_status succeeded, but no build_chroots
2020-10-05 08:02:03,237 [ERROR][/usr/share/copr/coprs_frontend/coprs/models.py:1168|models:status] Build 1344345 has source_status succeeded, but no build_chroots
2020-10-05 08:10:14,588 [ERROR][/usr/share/copr/coprs_frontend/coprs/models.py:1168|models:status] Build 1344345 has source_status succeeded, but no build_chroots
2020-10-05 08:10:14,588 [ERROR][/usr/share/copr/coprs_frontend/coprs/models.py:1168|models:status] Build 1344345 has source_status succeeded, but no build_chroots
2020-10-05 08:10:14,589 [ERROR][/usr/share/copr/coprs_frontend/coprs/models.py:1168|models:status] Build 1344345 has source_status succeeded, but no build_chroots
...

This is now really dominant output in frontend.log.

Assigning Jakub per mtg discussion.

Metadata Update from @praiskup:
- Issue assigned to frostyx

3 years ago

Initially I thought that this is just a duplicate of #1619 but not all
of the linked builds are resubmits

coprdb=# select id, resubmitted_from_id from build where id in (744032, 1085446, 796875, 1344345);
┌─────────┬─────────────────────┐
│   id    │ resubmitted_from_id │
├─────────┼─────────────────────┤
│  744032 │                     │
│  796875 │                     │
│ 1085446 │                     │
│ 1344345 │             1322760 │
└─────────┴─────────────────────┘

There needs to be a further investigation specific for this issue.

Okay, so the listed builds are all year (and more) old and at that time we didn't have the resubmitted_from_id column yet. It was introduced in 6af1e7f on Oct 23 2019. So I would say this should be fixed by PR#1624.

We may want to update the database and set the status of those builds to be canceled.

Metadata Update from @praiskup:
- Issue status updated to: Open (was: Closed)

3 years ago

I still see those ...

2021-03-29 10:20:38,623 [ERROR][/usr/share/copr/coprs_frontend/coprs/models.py:1195|models:status] Build 1680944 has source_status succeeded, but no build_chroots
2021-03-29 10:20:38,624 [ERROR][/usr/share/copr/coprs_frontend/coprs/models.py:1195|models:status] Build 1680944 has source_status succeeded, but no build_chroots
2021-03-29 10:20:38,625 [ERROR][/usr/share/copr/coprs_frontend/coprs/models.py:1195|models:status] Build 1681105 has source_status succeeded, but no build_chroots
2021-03-29 10:20:38,625 [ERROR][/usr/share/copr/coprs_frontend/coprs/models.py:1195|models:status] Build 1681105 has source_status succeeded, but no build_chroots
2021-03-29 10:20:38,626 [ERROR][/usr/share/copr/coprs_frontend/coprs/models.py:1195|models:status] Build 1681861 has source_status succeeded, but no build_chroots
2021-03-29 10:20:38,626 [ERROR][/usr/share/copr/coprs_frontend/coprs/models.py:1195|models:status] Build 1681861 has source_status succeeded, but no build_chroots
2021-03-29 10:20:38,626 [ERROR][/usr/share/copr/coprs_frontend/coprs/models.py:1195|models:status] Build 1681863 has source_status succeeded, but no build_chroots
2021-03-29 10:20:38,626 [ERROR][/usr/share/copr/coprs_frontend/coprs/models.py:1195|models:status] Build 1681863 has source_status succeeded, but no build_chroots

Perhaps we should fix the database?

Metadata Update from @praiskup:
- Assignee reset

3 years ago

Metadata Update from @praiskup:
- Issue assigned to praiskup

3 years ago

I executed this locally to "fix" the database, so we don't pollute the logs too much:

coprdb=# update build set source_status = 0 where id in (select build.id from build left outer join build_chroot on build.id = build_chroot.build_id where build_chroot.id is null and source_status = 1 order by build.id);
UPDATE 115

There's still bug that's going to be fixed in #1773, though we can re-execute this
sql statements in the future, too.

Can anyone please review the command, and give ACK for running this in prod?

coprdb.public> update build set source_status = 0 where id in (select build.id from build left outer join build_chroot on build.id = build_chroot.build_id where build_chroot.id is null and source_status = 1 order by build.id)

[2021-04-15 18:24:21] 115 rows affected in 4 s 635 ms

Works for me, even though I think order by build.id is unnecessary

Commit f9ac535 relates to this ticket

Login to comment on this ticket.

Metadata
Related Pull Requests
  • #1777 Merged 3 years ago
  • #1773 Merged 3 years ago
  • #1624 Merged 3 years ago