#5450 No longer receiving email notifications from notifications@fedoraproject.org
Closed: Fixed 7 years ago Opened 7 years ago by orion.

= bug description =

No longer receiving email notifications from notifications@fedoraproject.org

= bug analysis =

I'm setup for digest notifications. Haven't had any for several weeks.

fas: orion
email: orion@cora.nwra.com


Adding pingou here for comment. He hopefully can take a look...

Nothing striking with a quick glance at the settings :-s

Anything else I can provide? This is seriously hindering my Fedora work. We had DNS issues for a bit on nwra.com so perhaps something unsubscribed/disable me?

As another datapoint: if you set it back to immediate (ie, no digests) do they start coming in?

That would help us isolate it to the digest processor or not.

Switched back to immediate and I'm seeing notifications.

An update here: it is a problem with digests. I switched to them and no longer am getting anything either.

It looks like this might be similar to something we saw a while back and hotfixed, but the hotfix didn't go upstream or get setup right in ansible so a version update may have messed it up.

We are digging more, hopefully we can sort it out soon... sorry about this. ;(

so we had an issue a while back with digest email where the digest was generating emails too large to be sent.
We made it so that in these case the email would be dropped.

So this could explain the issue seen here.
To fix this we would have to rework sensibly the digest feature so that it can split into multiple emails depending on the size.

Update: Patrick added a bunch of debugging and tracked this down.

Basically somehow the latest upstream release doesn't handle the case of the 'android' backend. We had this backend enabled for a short time a number of years ago, but disabled it. For now we have cleared those settings/rules and restarted the digest processor.

It's processing through the IRC backlog now and then will process the email backlog, then will start sending digests.

Sorry for all this delay...

Further update: It processed some messages and then melted down again.

We are working now on seperating the digest and non digest processing. The non digest processing should be running as normal right now. The digest processing we will start back up as soon as we have it seperated out. Then it's going to need some additional work to try and find out failure conditions (email digest too large, irc message too large).

The digest producer should now be fixed, and is processing through the backlog.
It will take a little time before it's fully up-to-date, and some too big emails might not arrive, but the rest of the digests should arrive somewhere today.

ok, this should finally be fixed. ;( Sorry for the delay in getting it fixed.

I also sent a email to the devel-announce list with some details.
A copy of that is below.

'''
The Fedora Notifications System, available at:
https://apps.fedoraproject.org/notifications/
Is a way for interested parties to be notified when events occur in
Fedora. This system currently allows for IRC and/or Email notifications
of events, as well as "digesting" a specified number of messages or any
messages in a specified time.

Recently the digest processing had a number of issues:

  • Some users had selected very large time frames or number of messages
    and the resulting digest messages were too large to be accepted by
    our mail server.

  • Over sized digests also resulted in no processing of additional
    digests in the queue.

  • Digest processing was interrelated with the non digest processing,
    making it difficult to debug or test changes.

  • When the digest queue was very large, the IRC handler couldn't get
    enough processing time to join the network.

  • When the queue was very large testing fixes required processing all
    the queue and only then producing digests.

This resulted in no digests being sent at all.

We have taken the following steps:

  • Digests that are 'too large' are split into chunks that are
    acceptable to our mail server. If they are still too large for some
    users email providers, they will be dropped after retries.

  • Digest processing is now separated from non digest processing
    allowing for debugging and fixes that don't affect the real time
    notifications, as well as not blocking IRC workers from joining the
    network.

Unfortunately, in landing our fixes a bug was introduced that split
digests too soon, resulting in incomplete digests after a few
characters. This has been corrected and any new digest notifications
should be complete.

We are sorry for any inconvenience these issues may have caused.

Please report any issues you find with the notification system to
https://pagure.io/fedora-infrastructure (our config or deployment)
or
https://github.com/fedora-infra/fmn (upstream bugs/issues).

Thanks,
'''

@kevin changed the status to Fixed

7 years ago

Login to comment on this ticket.

Metadata