#9726 Long email delivery times
Closed: Will Not/Can Not fix 2 years ago by kevin. Opened 3 years ago by vondruch.

Describe what you would like us to do:


Hi, I am not sure if it is just me, but it appears that Fedora (maybe just FMN) related emails are massively delayed. E.g. this build failed on 06:24:04 UTC, but I have received notification on 10:47 UTC. This is part of the header:

Return-Path: notifications@fedoraproject.org
Received: from zmta03.collab.prod.int.phx2.redhat.com (LHLO
 zmta03.collab.prod.int.phx2.redhat.com) (10.5.81.10) by
 zmail26.collab.prod.int.phx2.redhat.com with LMTP; Fri, 12 Mar 2021
 05:51:28 -0500 (EST)
Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16])
    by zmta03.collab.prod.int.phx2.redhat.com (Postfix) with ESMTP id 02E8A50190
    for <vondruch@mail.corp.redhat.com>; Fri, 12 Mar 2021 05:51:28 -0500 (EST)
Received: by smtp.corp.redhat.com (Postfix)
    id F35895C277; Fri, 12 Mar 2021 10:51:27 +0000 (UTC)
Delivered-To: vondruch@redhat.com
Received: from mx1.redhat.com (ext-mx16.extmail.prod.ext.phx2.redhat.com [10.5.110.45])
    by smtp.corp.redhat.com (Postfix) with ESMTPS id ED2E05C276
    for <vondruch@redhat.com>; Fri, 12 Mar 2021 10:51:27 +0000 (UTC)
Received: from bastion.fedoraproject.org (bastion01.iad2.fedoraproject.org [10.3.163.31])
    (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits))
    (No client certificate requested)
    by mx1.redhat.com (Postfix) with ESMTPS id 72AD83082A6F
    for <vondruch@redhat.com>; Fri, 12 Mar 2021 10:51:19 +0000 (UTC)
Received: from notifs-backend01.iad2.fedoraproject.org (notifs-backend01.iad2.fedoraproject.org [10.3.163.63])
    (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits))
    (Client did not present a certificate)
    by bastion01.iad2.fedoraproject.org (Postfix) with ESMTPS id 88C0930E196A
    for <vondruch@redhat.com>; Fri, 12 Mar 2021 10:47:47 +0000 (GMT)
DKIM-Filter: OpenDKIM Filter v2.11.0 bastion01.iad2.fedoraproject.org 88C0930E196A
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fedoraproject.org;
    s=bastion-iad; t=1615546067;
    bh=gt74ncoU4yV5IV6W3bwUjDok9BQF+GZRqxhzrG3nERY=;
    h=From:To:Subject:Date:From;
    b=uuodWHEAFmGfUq17cLLIxlJ4SmstLLerzfdGS366xP8eCtZUfwIxlQzfDKZIVS0WT
     c4e3mtPKnJtFAXDCW5crdyights9J1leyrkJV9Kj4fikoHxTAcfYdyxMrhRlVF+jdF
     r3vf9/2QpmvDIlhnggeMj6qkibOWyLwqv+loQBb0=
Precedence: Bulk
Auto-Submitted: auto-generated
From: notifications@fedoraproject.org
To: vondruch@redhat.com
X-Fedmsg-Topic: org.fedoraproject.prod.koschei.package.state.change
X-Fedmsg-Category: koschei
X-Fedmsg-Package: rubygem-activestorage
X-Fedmsg-Num-Packages: 1
Subject: rubygem-activestorage's builds started to fail in Fedora rawhide
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
Message-Id: <20210312104747.88C0930E196A@bastion01.iad2.fedoraproject.org>
Date: Fri, 12 Mar 2021 10:47:47 +0000 (GMT)
X-Greylist: Sender DNS name whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.45]); Fri, 12 Mar 2021 10:51:19 +0000 (UTC)
X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.45]); Fri, 12 Mar 2021 10:51:19 +0000 (UTC) for IP:'10.3.163.31' DOMAIN:'bastion01.iad2.fedoraproject.org' HELO:'bastion.fedoraproject.org' FROM:'notifications@fedoraproject.org' RCPT:''
X-RedHat-Spam-Score: -1.35  (ALL_TRUSTED,DKIMWL_WL_HIGH,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU) 10.3.163.31 bastion01.iad2.fedoraproject.org 10.3.163.31 bastion01.iad2.fedoraproject.org <notifications@fedoraproject.org>
X-Scanned-By: MIMEDefang 2.84 on 10.5.110.45
X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16

When do you need this to be done by? (YYYY/MM/DD)



It's just FMN. It got backloged recently by:

  • having to be restarted because it was leaking memory. It takes it ~1hour to restart.

  • There's been a lot of messages lately. I am not sure sure from what.

It's slowly catching up... and I am not sure there's much more we can do.

@vondruch do you still see this?

I'm not sure there is much we can do here :(

Metadata Update from @smooge:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: low-gain, low-trouble, ops

3 years ago

There is still/again ~1h delay

http://koji.fedoraproject.org/koji/buildinfo?buildID=1723968

Return-Path: notifications@fedoraproject.org
Received: from zmta04.collab.prod.int.phx2.redhat.com (LHLO
 zmta04.collab.prod.int.phx2.redhat.com) (10.5.81.11) by
 zmail26.collab.prod.int.phx2.redhat.com with LMTP; Tue, 16 Mar 2021
 11:05:10 -0400 (EDT)
Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23])
    by zmta04.collab.prod.int.phx2.redhat.com (Postfix) with ESMTP id A70C8DA25F
    for <vondruch@mail.corp.redhat.com>; Tue, 16 Mar 2021 11:05:10 -0400 (EDT)
Received: by smtp.corp.redhat.com (Postfix)
    id A13A019D7D; Tue, 16 Mar 2021 15:05:10 +0000 (UTC)
Delivered-To: vondruch@redhat.com
Received: from mx1.redhat.com (ext-mx20.extmail.prod.ext.phx2.redhat.com [10.5.110.49])
    by smtp.corp.redhat.com (Postfix) with ESMTPS id 9871519D61
    for <vondruch@redhat.com>; Tue, 16 Mar 2021 15:05:10 +0000 (UTC)
Received: from bastion.fedoraproject.org (bastion01.iad2.fedoraproject.org [10.3.163.31])
    (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits))
    (No client certificate requested)
    by mx1.redhat.com (Postfix) with ESMTPS id F3F6E30C3B7D
    for <vondruch@redhat.com>; Tue, 16 Mar 2021 15:05:01 +0000 (UTC)
Received: from notifs-backend01.iad2.fedoraproject.org (notifs-backend01.iad2.fedoraproject.org [10.3.163.63])
    (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits))
    (Client did not present a certificate)
    by bastion01.iad2.fedoraproject.org (Postfix) with ESMTPS id A9029304C5F7
    for <vondruch@redhat.com>; Tue, 16 Mar 2021 14:59:47 +0000 (GMT)
DKIM-Filter: OpenDKIM Filter v2.11.0 bastion01.iad2.fedoraproject.org A9029304C5F7
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fedoraproject.org;
    s=bastion-iad; t=1615906787;
    bh=Q4x+tokMqG870GUnWlfqOjdFKqY+9IpPtBUyupzRyD4=;
    h=From:To:Subject:Date:From;
    b=K9FWkCqBuVQBK7tlHH1HXgWpUWoNzjh0U5jXD0LQ05S7i6bo35mhURKD5hiXwbvFd
     1jUBYD1WgpyZN8sAfCIgd19tw8WaFSL4EaQ9fwJK9HpgFrJ9/FkdxCqm4N2HPvY8wz
     x5ykQ0ufLtZCMDKE6L/u/bWH0mKOR9HwHm1BNvnI=
Precedence: Bulk
Auto-Submitted: auto-generated
From: notifications@fedoraproject.org
To: vondruch@redhat.com
X-Fedmsg-Topic: org.fedoraproject.prod.buildsys.build.state.change
X-Fedmsg-Category: buildsys
X-Fedmsg-Username: vondruch
X-Fedmsg-Package: rubygem-builder
X-Fedmsg-Num-Packages: 1
Subject: vondruch's rubygem-builder-3.2.4-4.fc34 completed
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
Message-Id: <20210316145947.A9029304C5F7@bastion01.iad2.fedoraproject.org>
Date: Tue, 16 Mar 2021 14:59:47 +0000 (GMT)
X-Greylist: Sender DNS name whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.49]); Tue, 16 Mar 2021 15:05:02 +0000 (UTC)
X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.49]); Tue, 16 Mar 2021 15:05:02 +0000 (UTC) for IP:'10.3.163.31' DOMAIN:'bastion01.iad2.fedoraproject.org' HELO:'bastion.fedoraproject.org' FROM:'notifications@fedoraproject.org' RCPT:''
X-RedHat-Spam-Score: -1.35  (ALL_TRUSTED,DKIMWL_WL_HIGH,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU) 10.3.163.31 bastion01.iad2.fedoraproject.org 10.3.163.31 bastion01.iad2.fedoraproject.org <notifications@fedoraproject.org>
X-Scanned-By: MIMEDefang 2.84 on 10.5.110.49
X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23

So, sadly there's nothing we can do here until we re-write notifs.

At times when there's a flood of messages (like when we tag beta packages or the like) it just gets behind because the way it's implemented. It has to look at each message, then check each list of users to see if they want to be notified on it and this is really slow. ;(

This is definitely something we want to fix in the new version... but not much we can do right now. ;(

Metadata Update from @kevin:
- Issue close_status updated to: Will Not/Can Not fix
- Issue status updated to: Closed (was: Open)

2 years ago

Login to comment on this ticket.

Metadata
Boards 1
ops Status: Done