Observations:

• most of the bugzillas received a duplicate comment, within 0-1 seconds after the first one
• not all of them thou, but no obvious difference between the bugzillas, so I'm gonna say it's random

This function will attempt to send the comment again if there was a specific exception:

def send_reminder(bug, comment=TEMPLATE, set_needinfo=True):
    created = date(*bug.creation_time.timetuple()[:3])
    orphanon = created + timedelta(days=7*8)
    comment = comment.format(orphanon=orphanon.isoformat())
    flags = [needinfo(bug.assigned_to)] if set_needinfo else []
    update = bzapi.build_update(comment=comment, flags=flags)
    try:
        bzapi.update_bugs([bug.id], update)
    except Exception as e:
        LOGGER.exception(bug.weburl)
        if "You can't ask" in getattr(e, 'faultString', ''):
            print(e.faultString, file=sys.stderr)
            return send_reminder(bug, comment=comment, set_needinfo=False)
        if 'set multiple times' in getattr(e, 'faultString', ''):
            return send_reminder(bug, comment=comment, set_needinfo=False)
        failed.append(bug)
    else:
        updated.append(bug)
        with open(ALREADY_FILED, 'a') as f:
            print(bug.id, file=f)

I suspect it is the cause, yet I am not entirely sure. Having the log would be helpful.

So, I've opened https://pagure.io/fedora-infra/ansible/pull-request/416 and that should preserve the log next time hopefully.

In the meantime, can somebody please get the log from the cron logs on the server? It run on 2021-02-14 around 04:30 UTC.

Ah ha. I have a theory...

I think staging is also running this and NOT using partner-bugzilla as it should.

I can forward you the two emails (one stg, one prod).

So, we need to either not run it at all in stg, or make sure stg points to partner-bugzilla (which doesn't send emails).

I have thought of double-running, but how much likely is it that staging and proper would be so synchronized that each duplicated comment would be 0-1 second from the original? Anyway, let's do that, yes.

This file here https://pagure.io/fedora-infra/ansible/blob/main/f/roles/releng/templates/ftbfs.cfg.j2 needs a bugzilla key with https://bugzilla.redhat.com / https://partner-bugzilla.redhat.com value based on whether it is prod or staging. The same most likely applies to the tracking bug number if we create a fake tracker there.

https://pagure.io/fedora-infra/ansible/c/63c21881860755792a9b27a117c5ebd3a2a19362?branch=main

https://pagure.io/fedora-infra/ansible/c/5633044539f99b590f6e44c1f71c1160c5b8f87a?branch=main
https://partner-bugzilla.redhat.com/show_bug.cgi?id=1732434