According to analysis by @erolkeskin Karma series of badges cause badges-backend01 to run out of memory:
it's related to the fact that those badges are trying to get every row in a topic from datanommer
Since we are currently putting out fires, buying time for further investigation and for structural changes, the Karma badges will be turned of for the time being.
Full error message
Metadata Update from @gui1ty: - Custom field artwork adjusted to None - Custom field concept_review_passed adjusted to None
Metadata Update from @gui1ty: - Custom field external_requirements adjusted to on
Metadata Update from @gui1ty: - Issue set to the milestone: short term
It tuned out disabling the Karma badges was not getting the message queue in check. These badges are now enabled again.
Instead I chose to disable Copr badges. There's way to many Copr builds running each day emitting hundreds if not thousands of messages. A lot of these builds are automated builds triggered by commits, merges or a CI backend. The majority would never lead to a badge, anyway, because the responsible user, a bot, is not in FAS. Yet they put a tremendous strain on the processing of messages.
Metadata Update from @gui1ty: - Issue priority set to: no deadline (was: critical (next week)) - Issue set to the milestone: medium term (was: short term)
Login to comment on this ticket.