For some reason bodhi is not picking them up, even though they are in the right tag.

I re-tagged them assuming bodhi lost the message or something, but still bodhi is not doing anything about it.

Sorry @cverna, any thoughts?

I don't know if this is the same problem or not, but I created a few updates from side tags and they aren't getting processed by bodhi. They are stuck in the pending state:

• https://bodhi.fedoraproject.org/updates/FEDORA-2020-554d104278
• https://bodhi.fedoraproject.org/updates/FEDORA-2020-2b573e2ac6

This is the first time I've used sidetags, so I might be doing something wrong.

You can click Actions -> Push to testing. I don't know why this is (only sometimes?) needed, but it is not the same problem.

I did it for https://bodhi.fedoraproject.org/updates/FEDORA-2020-38df99d013, but still got stuck, seems like an issue with side tags.

You can click Actions -> Push to testing. I don't know why this is (only sometimes?) needed, but it is not the same problem.

Thanks @churchyard - I did that and now it looks like they are pending -> testing so let's see if they progress on the next bodhi run. Sorry for the noise here.

Same here, with my sidetag: https://bodhi.fedoraproject.org/updates/FEDORA-2020-5bc381f3f0

It's in testing but I cannot see any Action to run.

Ok, so I was able to get the first update listed here ( https://bodhi.fedoraproject.org/updates/FEDORA-2020-38df99d013 ) pushed. For this, I've add to go to the database and remove the comment about This update can be pushed to stable now if the maintainer wishes which apparently makes the approve_testing.py script ignore the update.

It's in testing but I cannot see any Action to run.

This is normal for updates that are automatically pushed to stable (which is the case for rawhide).

I'm going to remove the "update can be pushed to stable" comment on https://bodhi.fedoraproject.org/updates/FEDORA-2020-5bc381f3f0 and see if that helps the update.

The second update is stable as well.

This is how I removed the comment that blocked the udpate:

On bodhi-backend:

# bodhi-shell 
[...]
>>> update = m.Update.get("FEDORA-2020-5bc381f3f0")
>>> update.has_stable_comment
True
>>> for idx, c in enumerate(update.comments_since_karma_reset): print(idx, c.text)
... 
0 This update can be pushed to stable now if the maintainer wishes
1 This update's test gating status has been changed to 'ignored'.
2 This update's test gating status has been changed to 'waiting'.
3 This update's test gating status has been changed to 'ignored'.
4 This update's test gating status has been changed to 'waiting'.
>>> update.comments_since_karma_reset[0].id
1568837

Then on db01:

bodhi2=# select * from comments where id = 1568837;
   id    | karma | karma_critpath |                               text                               |         timestamp          | update_id | user_id 
---------+-------+----------------+------------------------------------------------------------------+----------------------------+-----------+---------
 1568837 |     0 |              0 | This update can be pushed to stable now if the maintainer wishes | 2020-08-14 04:02:34.809732 |    232114 |      91
(1 row)

bodhi2=# delete from comments where id = 1568837;
DELETE 1

Now how did this happen?

• https://bodhi.fedoraproject.org/updates/FEDORA-2020-38df99d013 was created 4 days ago
• https://bodhi.fedoraproject.org/updates/FEDORA-2020-5bc381f3f0 was created 6 days ago

So it isn't really a race condition around branching or so.

Any ideas?

@mattia since you've been quite a bit on the bodhi code lately, this may interest you ^ :)

@pingou just a suspect, but may be again https://github.com/fedora-infra/bodhi/issues/4002

But I see F33 is still not composed_by_bodhi, so might be unrelated or another aspect of the same bug.

@mattia the edit button existed for those updates and I was in fact able to edit them a couple of time while trying to figure out what was blocking them (well, I only edited one, the second one benefited from the discovery made on the first one).

This looks like something added the comment that the update could be pushed to stable then either crashed or simply didn't run the subsequent checks to see if the update could be automatically pushed.

I hope to have some free time in the weekend to review the whole update-from-sidetag lifespan.
There were no changes related to sidetag managing in 5.5, so I would bet the freeze had caused some weirdness with the update processes. I also want to check that setting F33 as pre-beta (https://pagure.io/fedora-infra/ansible/c/a7a2c0232fc62ec4554d0fb01ee454907239017f?branch=master) has no strange effects on the sidetag workflow.

Had we got any other problematic update created from sidetag on F33 after the freeze ended? Or any other problematic update from sidetag on F34?

Here's another f33 update form side tag, created now

https://bodhi.fedoraproject.org/updates/FEDORA-2020-c3e2a83528

it went to testing within seconds and to stable within a minute.

I have spent the morning in reviewing the 'update from side-tag' workflow and I didn't find any particular flaw.

Since the update got the stable comment, the workflow must be interrupted somewhere in the approve_testing task which is performed after the signing handler stuff, so it should have nothing to do with the freeze.
The only thing I can came up is to be sure to not set update.stable_days when creating it from side-tag for a release not composed by Bodhi, which can break the if statement at line 93 of approve_testing task [1].
I've filed a PR upstream for that.

https://bodhi.fedoraproject.org/updates/FEDORA-2020-59e144acee was another case affected by this.

So what's the status on this? Is it still happening?

We haven't seen any issue lately, closing this ticket.

Please reopen if it happens again.