#9250 Mails from copr-fe-dev.aws.fedoraproject.org being marked as spam
Opened a month ago by frostyx. Modified 14 days ago

Describe what you would like us to do:

We had an issue with sending mails from copr-frontend instances, that was fixed as #9233. Mails are now successfully being sent from

copr-fe-dev.aws.fedoraproject.org
copr-fe.aws.fedoraproject.org

but they are being marked as spam. Is there something else that needs to be done for us being to able to send emails from copr-frontend instances?

I am attaching an email with all its headers, that came to my spam folder.

Delivered-To: jkadlcik@gapps.redhat.com
Received: by 2002:a67:c589:0:0:0:0:0 with SMTP id h9csp315739vsk;
        Wed, 19 Aug 2020 04:11:47 -0700 (PDT)
X-Google-Smtp-Source: ABdhPJxWT/Srce+SinTEl6Sj/9spOklKagorBjs4wTcyAnAQ1v2hzFtE2A4dGbxZtiAO+I8YlW2p
X-Received: by 2002:a17:906:5ad8:: with SMTP id x24mr23534810ejs.329.1597835507727;
        Wed, 19 Aug 2020 04:11:47 -0700 (PDT)
ARC-Seal: i=1; a=rsa-sha256; t=1597835507; cv=none;
        d=google.com; s=arc-20160816;
        b=keH5St/qISoXNQTXj9iK0JWpzXcfwSQHu4K1+z/8LJYmFVibbuwwlo52qlCE+grdas
         CJK0dB0NVAOgN5od3qtw7qAHh3HeQ1nke1/2KLNrz59YbyhDG+w5ziI/3Cf6Pt8k6Qcw
         evRC1jcdtfk4gytqzAG4iV4SnRH4DDlTGxEXsZYh6qL4Utap/oG9dxTxOZ3gkkHIR1bg
         F5c83RJulESp+g+7dOVWjTviNdqHA/iCCgLaBVgKSgeGwA6exd6FtVo/O560xWoyCi0b
         IRXwOJbjLgAhSeWiJiN32vsf6J7T8TuuwwRZan3lbjpuZwFJgABz6r3BKi7EEBzBHEL8
         Yu4g==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816;
        h=content-transfer-encoding:date:message-id:to:from:subject
         :mime-version:dkim-filter:delivered-to;
        bh=bRQ5/pT+F+u2bNZymFd1J4LP9gJfeJ2TSeVCnPQor0k=;
        b=tRCJapntiyp61ERLAzFKzeZ3D1IKbCsJ05pM79FhOattD3wod17wGGF787aMEuikTz
         oUIEhzF7m/EOvkTA8gSdlSxAc2fvsX34e8ocSibJSqIhiNo29pW2ZD/IoGAq4Xfz6Cs8
         7J1OSb3gHt/1qJkhcNpI0BRvSG6tVSgCknf1nr/AZ6wepjFGMCvWGjnLDluGRy+DNot7
         BQXhGYY6Aq1PqHjSAaxOV26kAwwoYL1tUc6ahYPFJ5+S8ALOaF+0b3BYTrh48A7JIHpF
         SK14J20uOJTqLFY4xzkEf8ypgi0/N8lx6HlXENHVYHb/DzFiYhYTFjU/hLl8Cpmj6hu1
         6sHA==
ARC-Authentication-Results: i=1; mx.google.com;
       spf=neutral (google.com: 35.153.70.58 is neither permitted nor denied by best guess record for domain of root@copr-fe-dev.aws.fedoraproject.org) smtp.mailfrom=root@copr-fe-dev.aws.fedoraproject.org
Return-Path: <root@copr-fe-dev.aws.fedoraproject.org>
Received: from us-smtp-1.mimecast.com (us-smtp-delivery-1.mimecast.com. [207.211.31.120])
        by mx.google.com with ESMTPS id r25si14070939ejx.486.2020.08.19.04.11.47
        for <jkadlcik@gapps.redhat.com>
        (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128);
        Wed, 19 Aug 2020 04:11:47 -0700 (PDT)
Received-SPF: neutral (google.com: 35.153.70.58 is neither permitted nor denied by best guess record for domain of root@copr-fe-dev.aws.fedoraproject.org) client-ip=35.153.70.58;
Authentication-Results: mx.google.com;
       spf=neutral (google.com: 35.153.70.58 is neither permitted nor denied by best guess record for domain of root@copr-fe-dev.aws.fedoraproject.org) smtp.mailfrom=root@copr-fe-dev.aws.fedoraproject.org
Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-141-IWN-Aq3jOsC05KUdrt_5pA-1; Wed, 19 Aug 2020 07:11:44 -0400
X-MC-Unique: IWN-Aq3jOsC05KUdrt_5pA-1
Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id D8C78807332 for <jkadlcik@gapps.redhat.com>; Wed, 19 Aug 2020 11:11:43 +0000 (UTC)
Received: by smtp.corp.redhat.com (Postfix) id D4ED77B90C; Wed, 19 Aug 2020 11:11:43 +0000 (UTC)
Delivered-To: jkadlcik@redhat.com
Received: from mx1.redhat.com (ext-mx11.extmail.prod.ext.phx2.redhat.com [10.5.110.40]) by smtp.corp.redhat.com (Postfix) with ESMTPS id D00ED7BE8F for <jkadlcik@redhat.com>; Wed, 19 Aug 2020 11:11:43 +0000 (UTC)
Received: from bastion.fedoraproject.org (bastion01.iad2.fedoraproject.org [10.3.163.31]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 3BB6D3084286 for <jkadlcik@redhat.com>; Wed, 19 Aug 2020 11:11:35 +0000 (UTC)
Received: from copr-fe-dev.aws.fedoraproject.org (ec2-35-153-70-58.compute-1.amazonaws.com [35.153.70.58]) by bastion01.iad2.fedoraproject.org (Postfix) with ESMTP id 8D3F630C6B3E for <jkadlcik@redhat.com>; Wed, 19 Aug 2020 10:53:32 +0000 (GMT)
DKIM-Filter: OpenDKIM Filter v2.11.0 bastion01.iad2.fedoraproject.org 8D3F630C6B3E
Received: from ec2-35-153-70-58.compute-1.amazonaws.com (localhost [IPv6:::1]) by copr-fe-dev.aws.fedoraproject.org (Postfix) with ESMTP id 78DCD40704 for <jkadlcik@redhat.com>; Wed, 19 Aug 2020 10:53:32 +0000 (UTC)
MIME-Version: 1.0
Subject: Testing mails from copr-fe-dev
From: root@copr-fe-dev.aws.fedoraproject.org
To: jkadlcik@redhat.com
Message-Id: <20200819105332.78DCD40704@copr-fe-dev.aws.fedoraproject.org>
Date: Wed, 19 Aug 2020 10:53:32 +0000 (UTC)
X-Greylist: Sender DNS name whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.40]); Wed, 19 Aug 2020 11:11:35 +0000 (UTC)
X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.40]); Wed, 19 Aug 2020 11:11:35 +0000 (UTC) for IP:'10.3.163.31' DOMAIN:'bastion01.iad2.fedoraproject.org' HELO:'bastion.fedoraproject.org' FROM:'root@copr-fe-dev.aws.fedoraproject.org' RCPT:''
X-RedHat-Spam-Score: 0.765
  (KHOP_HELO_FCRDNS,PDS_RDNS_DYNAMIC_FP,RDNS_DYNAMIC,SPF_HELO_NONE,TO_NO_BRKTS_DYNIP) 10.3.163.31 bastion01.iad2.fedoraproject.org 10.3.163.31 bastion01.iad2.fedoraproject.org <root@copr-fe-dev.aws.fedoraproject.org>
X-Scanned-By: MIMEDefang 2.84 on 10.5.110.40
X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11
X-Mimecast-Spam-Score: 0.0
X-Mimecast-Originator: copr-fe-dev.aws.fedoraproject.org
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit

Just testing mails from copr-fe-dev

When do you need this to be done by? (YYYY/MM/DD)

The sooner the better. We cannot properly notify users about important events.


Metadata Update from @mohanboddu:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: groomed, medium-gain, medium-trouble, ops

a month ago

@praiskup copr doesn't send mail directly it sends it through bastion and those are what deliver it to the world. As suck I believe the dkim for bastion.fedoraproject.org is what is to be used.

SPF entry for bastion.fedoraproject.org is also not set because when I run dig bastion.fedoraproject.org txt +multiline +noall +answer it returns no answer. bastion.fedoraproject.org has only A record. Can you create an SPF record and DKIM record please? Should I open a new issue because of it?
Without it, we will constantly send users spam emails and we don't want that.
I also think that an SPF record for fedoraproject.org is incorrect, because when you run dig fedoraproject.org txt +multiline +noall +answer you can see the answer as
fedoraproject.org. 231 IN TXT "v=spf1 a a:mailers.fedoraproject.org ipv4:38.145.60.11 ipv4:38.145.60.12 ?all" but it should be just ip4 not ipv4. Validators also don't recognize the syntax (https://toolbox.googleapps.com/apps/checkmx/check?domain=fedoraproject.org&dkim_selector=)

@schlupov Thanks for the fix on the spf1. I don't know where I got that syntax on ipv4. I have fixed that part.

Bastion lost its spf record when we had 2 bastions in phx2 and iad2 and I didn't cleanly rename things.. that has been fixed.

I see a DKIM domain key in the zone for bastion. but no other records. NBot sure what else is supposed to be there.

I tried to send an email to myself from the Copr dev instance and the email ended up in spam

Delivered-To: schlupov@gapps.redhat.com
Received: by 2002:a67:b305:0:0:0:0:0 with SMTP id a5csp305355vsm;
    Wed, 2 Sep 2020 05:49:03 -0700 (PDT)
X-Google-Smtp-Source: ABdhPJxaZv/7RPiG2hBuK2yXuT8J7de2NXdbgW6UKmv1tL37xomCM+R1YFRdacDyiVGVrHkHRzpp
X-Received: by 2002:a25:5741:: with SMTP id l62mr9969248ybb.299.1599050943308;
    Wed, 02 Sep 2020 05:49:03 -0700 (PDT)
ARC-Seal: i=1; a=rsa-sha256; t=1599050943; cv=none;
    d=google.com; s=arc-20160816;
    b=tXLqnoU7wXC8cDaK/5KCYDKpLsu0y1VZafZPqXLxTlCSJaPIzOA66ftCLmyib4pzG8
     3Las52F+9flwZtsJEUAW6ZHunp2bVDCVGj2PCJbrf1EhzMkSpCnNt6VgTANF2Hw6M3tf
     2uRWt0EHoGTYNpSsknMUrKWGoi8dztSqsdwFGSJhZErq72b+vd4FtSlvqXQXy3Njw8fk
     nt/CncMu7eLvf3I+QJxHoSxPJok2P8n+Cmz2R3mnbBU77SOYiK9txiG1/XnRFbrZWgrh
     onb2yhHK8UUYZHFXx2cCKqWDApdN1Aqk+v809l7vEn1ZA5JdWEQ098yZNA7CPyYnxJK6
     GFvw==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816;
    h=content-transfer-encoding:date:message-id:to:from:subject
     :mime-version:dkim-filter:delivered-to;
    bh=6h8spjCSGvcuawNpSpJKQ11Rnr2tlchkBf/qBJmMyAc=;
    b=EcA1DtrHgJM0t7tidsOjzwKAvc6Zu9OS8IXX64a5SXy43fIWo6gCndR28hn/OyNjH+
     mN/hMrvZDCMp3bVHPaTk0ULkjll5GaCz+XdXmvmYI2IEm0mS2N69kuZdGaa+V7vvy1rL
     zyawLphqoN6XNc87thEPs/YLPPzaHt/kbrQJ+B4oNt1d1uGQnkfY2bWAYQcYPgaN7ct+
     yxqlec/41XWUQUyGy6U3mOd0x6taUy1VX0QM+iSuIZGGv/CnvZyUX98JU3EpjhCNFivR
     3IKtWtsD9N0mPizTW8MPxGFlOmdAaKg6/IL0AjkwGqdk+kFodoPSJmiavzkflujLPwnC
     mDoA==
ARC-Authentication-Results: i=1; mx.google.com;
   spf=neutral (google.com: 35.153.70.58 is neither permitted nor denied by best guess record for domain of root@copr-fe-dev.aws.fedoraproject.org) smtp.mailfrom=root@copr-fe-dev.aws.fedoraproject.org
Return-Path: <root@copr-fe-dev.aws.fedoraproject.org>
Received: from us-smtp-1.mimecast.com (us-smtp-delivery-1.mimecast.com. [205.139.110.120])
    by mx.google.com with ESMTPS id t186si4013313ybf.203.2020.09.02.05.49.03
    for <schlupov@gapps.redhat.com>
    (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128);
    Wed, 02 Sep 2020 05:49:03 -0700 (PDT)
Received-SPF: neutral (google.com: 35.153.70.58 is neither permitted nor denied by best guess record for domain of root@copr-fe-dev.aws.fedoraproject.org) client-ip=35.153.70.58;
Authentication-Results: mx.google.com;
   spf=neutral (google.com: 35.153.70.58 is neither permitted nor denied by best guess record for domain of root@copr-fe-dev.aws.fedoraproject.org) smtp.mailfrom=root@copr-fe-dev.aws.fedoraproject.org
    Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-318-wA6AV2BHMHO6qYKMUSqoYg-1; Wed, 02 Sep 2020 08:49:01 -0400
X-MC-Unique: wA6AV2BHMHO6qYKMUSqoYg-1
Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 3651910082E8 for <schlupov@gapps.redhat.com>; Wed,
  2 Sep 2020 12:49:00 +0000 (UTC)
Received: by smtp.corp.redhat.com (Postfix) id 315AE9F63; Wed,
  2 Sep 2020 12:49:00 +0000 (UTC)
Delivered-To: schlupov@redhat.com
Received: from mx1.redhat.com (ext-mx16.extmail.prod.ext.phx2.redhat.com [10.5.110.45]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 2B2E86198E for <schlupov@redhat.com>; Wed,
2 Sep 2020 12:49:00 +0000 (UTC)
Received: from bastion.fedoraproject.org (bastion01.iad2.fedoraproject.org [10.3.163.31]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 853953082B1A for <schlupov@redhat.com>; Wed,
  2 Sep 2020 12:48:51 +0000 (UTC)
Received: from copr-fe-dev.aws.fedoraproject.org (ec2-35-153-70-58.compute-1.amazonaws.com [35.153.70.58]) by bastion01.iad2.fedoraproject.org (Postfix) with ESMTP id CA45630BDEA5 for <schlupov@redhat.com>; Wed,
  2 Sep 2020 12:48:45 +0000 (GMT)
DKIM-Filter: OpenDKIM Filter v2.11.0 bastion01.iad2.fedoraproject.org CA45630BDEA5
Received: from ec2-35-153-70-58.compute-1.amazonaws.com (localhost [IPv6:::1]) by copr-fe-dev.aws.fedoraproject.org (Postfix) with ESMTP id BC9BB40728 for <schlupov@redhat.com>; Wed,
2 Sep 2020 12:48:45 +0000 (UTC)
MIME-Version: 1.0
Subject: Email from Copr
From: root@copr-fe-dev.aws.fedoraproject.org
To: schlupov@redhat.com
Message-Id: <20200902124845.BC9BB40728@copr-fe-dev.aws.fedoraproject.org>
Date: Wed,
  2 Sep 2020 12:48:45 +0000 (UTC)
X-Greylist: Sender DNS name whitelisted, not delayed by milter-greylist-4.5.16  (mx1.redhat.com [10.5.110.45]); Wed, 02 Sep 2020 12:48:51 +0000 (UTC)
X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.45]); Wed, 02 Sep 2020 12:48:51 +0000 (UTC) for IP:'10.3.163.31' DOMAIN:'bastion01.iad2.fedoraproject.org' HELO:'bastion.fedoraproject.org' FROM:'root@copr-fe-dev.aws.fedoraproject.org' RCPT:''
X-RedHat-Spam-Score: 0.763
  (KHOP_HELO_FCRDNS,RDNS_DYNAMIC,SPF_HELO_NONE) 10.3.163.31  bastion01.iad2.fedoraproject.org 10.3.163.31 bastion01.iad2.fedoraproject.org <root@copr-fe-dev.aws.fedoraproject.org>
X-Scanned-By: MIMEDefang 2.84 on 10.5.110.45
X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12
X-Mimecast-Spam-Score: 0.0
X-Mimecast-Originator: copr-fe-dev.aws.fedoraproject.org
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit

Testing email from Copr

SPF is neutral, shouldn't be copr-fe-dev.aws.fedoraproject.org permitted sender? You should add include:_spf.google.com (or ip4:_spf.google.com since we need to use ipv4) into SPF record. SPF must allow Google servers to send mail on behalf of your domain, it's the reason why you can see spf=neutral. Also, I would expect in header Authentication-Results
Authentication-Results: mx.google.com;
dkim=pass...
spf=pass ...

It looks like the DKIM signature is not added to emails, I can't see the DKIM-Signature header.

OK I have made some changes to our DNS.

aws.fedoraproject.org now has an SPF1 record
fedoraproject.org has a fixed dkim record.

Reading through the docs I do not see why it would be reasonable to put ina ip4:_spf.google.com because we do not use them as a sender (Red Hat does in its routing but that is outside of fedoraproject.org servers).

At this point I am going on 2+ weeks of PTO. I am removing myself from 'owning' the ticket as I don't want you to think I can fix anything until I get back. Hopefully someone else can help.

@smooge thank you for your help :)
Unfortunately, this issue has still not been resolved. Emails still end up in the spam box. Any help will be very appreciated.

Login to comment on this ticket.

Metadata
Boards 1
ops Status: Backlog
Attachments 1