I am experiencing delays in the deliver of email forwarded from my fedoraproject.org account to my rh account.
This is a mail header excerpt of one of the most egregious delays 2 days ago (6+ hours):
Received: from bastion.fedoraproject.org (unknown [10.3.163.31]) by smtp.corp.redhat.com (Postfix) with ESMTP id 748E417582 for <ssorce@redhat.com>; Wed, 16 Nov 2022 16:59:25 +0000 (UTC) Received: by bastion01.iad2.fedoraproject.org (Postfix) id 67CD7305719B; Wed, 16 Nov 2022 10:40:27 +0000 (GMT)
This is from today (25 minutes):
Received: from bastion.fedoraproject.org (unknown [10.3.163.31]) by smtp.corp.redhat.com (Postfix) with ESMTP id 23BD653AA for <ssorce@redhat.com>; Fri, 18 Nov 2022 18:49:29 +0000 (UTC) Received: by bastion01.iad2.fedoraproject.org (Postfix) id A0AEC304939C; Fri, 18 Nov 2022 18:24:07 +0000 (GMT)
Also odd that Fedora uses GMT instead of UTC for timezone ...
Just to be clear, not all emails are delayed the same way, but most are delayed at least 5/10 minutes.
Note: most of these emails are github notifications sent to fedoraproject.org and forwarded to redhat.com
Funnily enough I received the notification for the second comment immediately, while the ticket creation and first comment notifications were delivered 6 minutes later ...
So it is not dependent on the sender or even external senders, seem completely confined within fedorainfra and rh's smtp server.
The problem is due to a hardware problem in border firewalls. The firewalls are up to be replaced in mid December which will hopefully remove the issue.
Metadata Update from @phsmoura: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: lists, medium-gain, medium-trouble
Metadata Update from @smooge: - Issue assigned to smooge
OK the firewall has been changed out and hopefully will fix the root cause. Will keep this open for a week to see if after mass rebuild to see if it is fixed fixed.
Was it changed right now? I still saw today delivery delay on fedora mailing lists messages landing in my mailbox out of order and late
My apologies. I had not seen this problem recently and the nagios alerts which showed queued up email had not triggered in a bit. Now I check and see that there are a couple hundred backed up emails still so I was just lucky.
The firewall change happened at 14:00 ET (1900 UTC) so I will check later today if new alerts happen after that.. if so will let networking know.
OK I have checked the logs since the change over to the new firewall. There have been no 'Connection timed out' to the internal mail servers since that change over. All queued email has been delivered after around 2100 UTC.
I have not seen out of order messages today, I will report here if I see anything.
would you be ok if I closed this? It can be reopened if you or others find it again.
No problem
Metadata Update from @simo: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Log in to comment on this ticket.