For the record here's the history:

Tue Sep 13 03:56:07 2022 libcxx-15.0.0-2.fc38 tagged into f38-build-side-58070 by nikic
Tue Sep 13 06:24:13 2022 libcxx-15.0.0-2.fc38 tagged into f38-updates-candidate by bodhi
Tue Sep 13 06:24:35 2022 libcxx-15.0.0-2.fc38 tagged into f38-build-side-58070-signing-pending by bodhi
Tue Sep 13 06:24:37 2022 libcxx-15.0.0-2.fc38 tagged into f38-signing-pending by bodhi
Tue Sep 13 06:25:34 2022 libcxx-15.0.0-2.fc38 untagged from f38-build-side-58070-signing-pending by autopen
Tue Sep 13 06:25:34 2022 libcxx-15.0.0-2.fc38 tagged into f38-build-side-58070-testing-pending by autopen
Tue Sep 13 06:26:33 2022 libcxx-15.0.0-2.fc38 untagged from f38-signing-pending by autopen
Tue Sep 13 06:26:33 2022 libcxx-15.0.0-2.fc38 tagged into f38-updates-testing-pending by autopen
Tue Sep 13 06:37:12 2022 libcxx-15.0.0-2.fc38 untagged from f38-updates-testing-pending by bodhi
Tue Sep 13 06:37:13 2022 libcxx-15.0.0-2.fc38 tagged into f38 by bodhi [still active]
Tue Sep 13 06:37:14 2022 libcxx-15.0.0-2.fc38 untagged from f38-updates-candidate by bodhi
Thu Sep 15 06:01:16 2022 libcxx-15.0.0-2.fc38 untagged from f38-build-side-58070-testing-pending by bodhi
Thu Sep 15 06:01:17 2022 libcxx-15.0.0-2.fc38 untagged from f38-build-side-58070 by bodhi

Wed Sep 14 05:00:48 2022 mlir-15.0.0-3.fc38 tagged into f38-build-side-58070 by nikic
Thu Sep 15 02:29:38 2022 mlir-15.0.0-3.fc38 tagged into f38-updates-candidate by bodhi
Thu Sep 15 02:29:56 2022 mlir-15.0.0-3.fc38 tagged into f38-build-side-58070-signing-pending by bodhi
Thu Sep 15 02:30:01 2022 mlir-15.0.0-3.fc38 tagged into f38-signing-pending by bodhi
Thu Sep 15 02:32:53 2022 mlir-15.0.0-3.fc38 untagged from f38-build-side-58070-signing-pending by autopen
Thu Sep 15 02:32:53 2022 mlir-15.0.0-3.fc38 tagged into f38-build-side-58070-testing-pending by autopen
Thu Sep 15 02:33:12 2022 mlir-15.0.0-3.fc38 untagged from f38-signing-pending by autopen
Thu Sep 15 02:33:12 2022 mlir-15.0.0-3.fc38 tagged into f38-updates-testing-pending by autopen
Thu Sep 15 02:34:20 2022 mlir-15.0.0-3.fc38 untagged from f38-updates-testing-pending by bodhi
Thu Sep 15 02:34:21 2022 mlir-15.0.0-3.fc38 untagged from f38-updates-candidate by bodhi
Thu Sep 15 02:34:44 2022 mlir-15.0.0-3.fc38 tagged into f38 by bodhi [still active]
Thu Sep 15 06:01:16 2022 mlir-15.0.0-3.fc38 untagged from f38-build-side-58070-testing-pending by bodhi
Thu Sep 15 06:01:17 2022 mlir-15.0.0-3.fc38 untagged from f38-build-side-58070 by bodhi

So, according to koji, bodhi was the thing that tagged it into f38-updates-candidate. ;(

Just to clarify: You did all the builds in the side tag, then created the update from that sidetag, then you needed to update something so you did another build(s) and tried to refresh the the side tag update, but by then bodhi already made a new update with just that updated package?

Just to clarify: You did all the builds in the side tag, then created the update from that sidetag, then you needed to update something so you did another build(s) and tried to refresh the the side tag update, but by then bodhi already made a new update with just that updated package?

This is about right, with one caveat: The single-package updates were created when editing the side tag update. The new builds would get added to the side tag update, but a separate update for just a single build would get submitted as well.

So I assume the incorrect tag was applied as part of editing the side tag update (though this only happened for two out of six edits, so clearly doesn't happen every time -- and also only for one build each time, even though the edits added multiple new builds).

CC: @mattia any ideas here?

This is quite strange...

I think this change doesn't work as expected, the added builds should not be tagged into up.release.pending_signing_tag but only into the side-tag pending tag.
But I will not be able to look at that in the next two weeks.

Well, I managed to find some spare time and came out with https://github.com/fedora-infra/bodhi/pull/4745

I still don't understand why it happens only for some builds while other are not affected, but the problem is that builds tagged into f38-updates-candidate causes the automatic update consumer to create an automatic update for those builds. This change should prevent that.

If you decide this bug is serious enough to apply the patch as hotfix, please proceed (automated tests results look fine).

Yeah, it's very odd that it's only hitting some (at least I have not seen more reports).

I'd love to get @abompard to weigh in...

My guess is that only some builds are hit because of parallelization due to multiple workers: when the message about a new build tagged into f38-updates-candidate is processed after the side-tag update build list is saved to database, the automatic update is not created, otherwise it is. The change in the PR I've proposed upstream should prevent the problem.

bodhi 7.0.1 is now deployed and hopefully this is fixed. :)