Please check the e-mail setup so a notification only gets sent once.
I just got "Embryo", "Tadpole", "Egg" badge notification e-mails and every one was sent twice (different Message ID).
Eg. the "Egg" e-mails have
Message-Id: <20210415123022.EB4DC30AEB04@bastion01.iad2.fedoraproject.org> Message-Id: <20210415123024.1140930B73D3@bastion01.iad2.fedoraproject.org>
@misc can you look at this to see if we are sending things from staging and prod?
Metadata Update from @smooge: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: low-gain, medium-trouble, ops
AFAIK, tahrir do not send email directly, and I do not have access to bastion to check the log of the smtp server.
@umenges we need more information, like the header of the emails, can you share them ?
Metadata Update from @misc: - Issue priority set to: Waiting on Reporter (was: Waiting on Assignee)
I hope it's ok if I omit/edit some parts of the headers for privacy. I guess you're mainly looking for the Received: parts for mail routing? They seem to be the same for both e-mails.
Received:
First e-mail:
Return-Path: <notifications@fedoraproject.org> [some Received: lines from internal infra] Received: from bastion.fedoraproject.org ([38.145.60.11]) by mx-ha.web.de (mxweb112 [212.227.17.8]) with ESMTPS (Nemesis) id 1MYwgM-1l2Ehx0uZM-00Ut7o for <edited>; Thu, 15 Apr 2021 14:30:37 +0200 Received: from notifs-backend01.iad2.fedoraproject.org (notifs-backend01.iad2.fedoraproject.org [10.3.163.63]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by bastion01.iad2.fedoraproject.org (Postfix) with ESMTPS id EB4DC30AEB04 for <edited>; Thu, 15 Apr 2021 12:30:22 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 bastion01.iad2.fedoraproject.org EB4DC30AEB04 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fedoraproject.org; s=bastion-iad; t=1618489823; bh=3mrwthmbFXUgr+H3mKj8c/3wLDscZBzsuk4GdA6pBEg=; h=From:To:Subject:Date:From; b=NqzmFJLBQE1oSxHPCTSjuy7uN4U1J3vTIrCDCMGqFyIgXjuauA62OGOpLt/TaOWGh HXpHuW24dPYxHcj699FYbq0B/FDdZbW9UxpV1PVfoXblBzA52mT/e/zEbqCuU2lA7A fxZvGaX2jzCHWrbRz3I07BQ421g8o22Iw7nSV9cg= Precedence: Bulk Auto-Submitted: auto-generated From: notifications@fedoraproject.org To: [edited] X-Fedmsg-Topic: org.fedoraproject.prod.fedbadges.badge.award X-Fedmsg-Category: fedbadges X-Fedmsg-Username: umenges X-Fedmsg-Num-Packages: 0 Subject: umenges has been awarded the "Egg" badge MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Message-Id: <20210415123022.EB4DC30AEB04@bastion01.iad2.fedoraproject.org> Date: Thu, 15 Apr 2021 12:30:22 +0000 (GMT) Envelope-To: <edited>
Second e-mail:
Return-Path: <notifications@fedoraproject.org> [some Received: lines from internal infra] Received: from bastion.fedoraproject.org ([38.145.60.11]) by mx-ha.web.de (mxweb110 [212.227.17.8]) with ESMTPS (Nemesis) id 1M2e9d-1lWkxG40Tf-004CK8 for <edited>; Thu, 15 Apr 2021 14:30:38 +0200 Received: from notifs-backend01.iad2.fedoraproject.org (notifs-backend01.iad2.fedoraproject.org [10.3.163.63]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by bastion01.iad2.fedoraproject.org (Postfix) with ESMTPS id 1140930B73D3 for <edited>; Thu, 15 Apr 2021 12:30:24 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 bastion01.iad2.fedoraproject.org 1140930B73D3 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fedoraproject.org; s=bastion-iad; t=1618489824; bh=3mrwthmbFXUgr+H3mKj8c/3wLDscZBzsuk4GdA6pBEg=; h=From:To:Subject:Date:From; b=Pri0ugb5/HX9m0/lUrWwh1to7qXNIDczdNZ6H+Exhyd5ZGZc/0CrrjQs2t+y69TAi hKC9hnC1cm2CHi5aPkyWDJ8hcfP0PTk945rmxB7K99bgtdU8pKebOoOvsUaS29IZSV QoztVcmj3yXhPo+ZmipSQnnDIflpfk+pzM3sH9U0= Precedence: Bulk Auto-Submitted: auto-generated From: notifications@fedoraproject.org To: [edited] X-Fedmsg-Topic: org.fedoraproject.prod.fedbadges.badge.award X-Fedmsg-Category: fedbadges X-Fedmsg-Username: umenges X-Fedmsg-Num-Packages: 0 Subject: umenges has been awarded the "Egg" badge MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Message-Id: <20210415123024.1140930B73D3@bastion01.iad2.fedoraproject.org> Date: Thu, 15 Apr 2021 12:30:24 +0000 (GMT) Envelope-To: <edited>
Yeah, privacy edit is fine. I am mostly trying to understand what sent the email. As they were sent on the same moment (2 seconds difference), it could be a race condition in the award script or something.
You got these 3 badges at once, no other badge duplicated ?
Seems there was only 1 award event (according to datagrepper: http get https://datagrepper-monitor-dashboard.app.os.fedoraproject.org/raw delta==172800 user==umenges ), and both mail come from the same server.
I got those not really at once, there were some hours in between. It also seems strange to get a one, two and three year badge on the same day.
Embryo:
Date: Thu, 15 Apr 2021 07:28:20 +0000 (GMT) Date: Thu, 15 Apr 2021 07:28:23 +0000 (GMT)
Tadpole:
Date: Thu, 15 Apr 2021 10:09:52 +0000 (GMT) Date: Thu, 15 Apr 2021 10:09:56 +0000 (GMT)
Egg:
Date: Thu, 15 Apr 2021 12:30:22 +0000 (GMT) Date: Thu, 15 Apr 2021 12:30:24 +0000 (GMT)
This could be related to the fix from https://github.com/fedora-infra/tahrir/issues/379
I guess something on the badge server side started to process a queue of users and it took a while to run.
I think I figured this out and fixed it. :)
Basically when proxy01.stg booted it didn't have stg.iad2.fedoraproject.org in it's dns search path. So, it's trying to connect to prod for a lot of things. Most prod hosts reject connections from stg, but notifs is old and doesn't do this.
So, I fixed things so stg hosts should start with the right resolver, fixed proxy01.stg, and pushed that change to ansible.
So, I think this is fixed now. Can you reopen if you see any more duplicates?
Metadata Update from @kevin: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Thanks for checking and caring! Finding and removing a stg/prod mixup IMHO qualifies for more than "low-gain" even. ;) Yes, I'll reopen if I get more duplicates. However, I'm not really a badge hunter, so it may take a while.. :)
Login to comment on this ticket.