#10705 Missing emails from src.fp.o and pagure.io
Closed: Fixed 2 years ago by kevin. Opened 2 years ago by churchyard.

Describe what you would like us to do:


I'd like your help with receiving emails from src.fedoraproject.org pull requests. I seem to be missing a lot of them. Not sure if this is a Fedora problem or a Gmail/Red Hat problem.

E.g. here: https://src.fedoraproject.org/rpms/python3-docs/pull-request/72

  • PR was opened: no email
  • Zuul comment: got the email
  • PR was merged: no email

Or here: https://src.fedoraproject.org/rpms/python3.9/pull-request/113

  • PR was opened: got the email
  • The recheck comment: got the email
  • Zuul comment: no email
  • PR was merged: no email

I have more examples of PRs where I only got some of the emails, but there might even be some that I am not even aware of :/

When do you need this to be done by? (YYYY/MM/DD)

No idea. This is pretty bad for my workflow, but if it only happens to me, it's probably not urgent.


This is happening also to me although different emails are lost.

E.g. here: https://src.fedoraproject.org/rpms/python3-docs/pull-request/72

  • PR was opened: no email
  • Zuul comment: got the email
  • PR was merged: got the email

Or here: https://src.fedoraproject.org/rpms/python3.9/pull-request/113

  • PR was opened: got the email
  • The recheck comment: no email
  • Zuul comment: no email
  • PR was merged: no email

The same happens on pagure.io issues, I had no email about the previous comment. Also in this ticket: https://pagure.io/fesco/issue/2795 I only got the second comment email, not the new ticket email.

Metadata Update from @phsmoura:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: high-gain, high-trouble, ops

2 years ago

And I am missing Fedora devel emails. From this thread, I have received just the last email

I have created RH internal ticket. If somebody of RH folks is interested, let me know and I'll add you on CC

It looks like Red Hat's filters stacked up email several times over the weekend. The nagios queue fired off lots of messages this weekend about backlog to the Red Hat server.

@smooge Could you provide the details in INC2221068 internally, please? They asked us for the missing emails, but we don't have them, quite obviously.

Reminds me of the internet meme where the seller asks for the picture of the parcel that was not received.

So, (one problem) appears to be the tls connection reuse that we setup to try and accomodate redhat.com connection limits. :(
I swear I had it working at one point, but it's not been working at all.

I have disabled it for now and set it to not use tls for connections to mx2.redhat.com (These are internal machine to another internal machine, so really tls is kind of pointless anyhow)

The queue on our side is drained now, can you all confirm you got your list posts?

The one outstanding issue I know of is the case of 'redhat.com address mails fedoraproject.org address that is backed by a gmail account' which is likely still failing due to redhat.com's SPF policy.

The queue on our side is drained now, can you all confirm you got your list posts?

I have looked at the originally reported pull requests and I still haven't received the missing emails.

I traced out the orig email from that first PR... but I think it was dropped because it was in the queue for 5 days. ;( I see it as late as this morning:

May 24 07:38:28 bastion01 postfix/qmgr[3078694]: 457A031625C6: from=<pagure@pkgs.fedoraproject.org>, size=1950, nrcpt=1 (queue active)
May 24 07:38:31 bastion01 postfix/error[3316305]: 457A031625C6: to=<mhroncok@redhat.com>, orig_to=<churchyard@fedoraproject.org>, relay=none, delay=428800, delays=428797/3.5/0/0, dsn=4.7.0, status=deferred (delivery temporarily suspended: host mx2.redhat.com[10.11.203.6] refused to talk to me: 421 4.7.0 smtp.corp.redhat.com Error: too many connections from 10.3.163.31)

I would expect postfix to log dropping it, but it doesn't seem to list anything by queue id. ;(
So I fear things older than 5 days ago just got dropped. ;(

Do you have any examples more recent I can trace out?

So I fear things older than 5 days ago just got dropped. ;(

:(

Do you have any examples more recent I can trace out?

I don't think I do. The rest indeed seems OK. However, it is quite hard to be sure.

And I am missing Fedora devel emails. From this thread, I have received just the last email

I have still not received the email. So as far as I understand, there is no hope to receive it :/ Other than that, I can't see any gaps recently

There were some emails that were over 5 days old that timed out in the queue and lost I fear. Very sorry about that... but I can't think of much we can do about it now.

The SPF policy thing is being tracked in another ticket.

I don't think there's anything more to do here... please re-open if you need us to do further things that we can do.

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

2 years ago

Login to comment on this ticket.

Metadata
Boards 1
ops Status: Backlog