I downloaded the patches (emails) as a .mbox file in neomutt, e.g. https://lists.fedoraproject.org/archives/list/kexec@lists.fedoraproject.org/thread/ROKGAPNNISSV6NHO5KZEHLCSVNPVE55V/ and git am would fail because of trailing "^M"s. Here are some observations about this issue, 1. The problematic patches were sent by other people. If I download the patches sent by myself, they are good. 2. The problematic patches are base64 encoded whereas the patches sent by myself are in plain text.
FYI, this is the analysis from one of my colleagues: "From my side, your email went through rh internal -> bastion.fedoraproject.org -> (some other fedora server) -> Me. I guess maybe still some routing magic here, but the issue is related to mail list setting for sure".
2021/05/01
Its seems a tiny thing but if it requires upgrading mailman, then it will be a difficult one.
Metadata Update from @mohanboddu: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: low-gain, low-trouble, ops
Usually the inclusion of ^M means that the file was sent in either Mac format or Windows. The control of this inclusion of being in base64 etc is a client side item way before it gets to our systems. [I don't see anything on our side which convert the email attachments into base64 by our tools. However I am not ruling this out.]
The usual 'solution' recommended at various sites is to have
git config --global core.autocrlf true
so that git does the right conversion for you. However, I do not know what that might break in your workflow.
Metadata Update from @smooge: - Issue untagged with: low-gain, low-trouble, ops - Issue priority set to: Needs Review (was: Waiting on Assignee)
Metadata Update from @smooge: - Issue tagged with: lists, low-gain, low-trouble, ops
Here is another observation about this issue: The emails sent to the mailing list by other people are not always bas64-encoded. For example, I received this email https://lists.fedoraproject.org/archives/list/kexec@lists.fedoraproject.org/message/T7NZTVUMWYN37MPJNM42WEBTDIDUF3OJ/ in the same thread as plain text. A difference I notice is that it didn't go through bastion.fedoraproject.org. I used another mailbox to subscribe to kexec@lists.fedoraproject.org and the same email was base64-encoded and went through kexec@lists.fedoraproject.org. So maybe we can narrow down the issue to bastion.fedoraproject.org.
Actually I use neomutt on Linux. And if my client encodes the email in base64, I should always receive my email as base64-encoded. But this is not true. So we can rule this out.
The usual 'solution' recommended at various sites is to have git config --global core.autocrlf true so that git does the right conversion for you. However, I do not know what that might break in your workflow.
The usual 'solution' recommended at various sites is to have git config --global core.autocrlf true
Thanks for the tip:)
Can you include the complete email with headers so I can see. The email in the mbox does not seem to have all the headers in it for this analysis. [I need to see one that went through bastion and where it went through bastion.]
Metadata Update from @smooge: - Issue priority set to: Waiting on Assignee (was: Needs Review)
Sure, here's the complete email of https://lists.fedoraproject.org/archives/list/kexec@lists.fedoraproject.org/thread/ROKGAPNNISSV6NHO5KZEHLCSVNPVE55V/,
From kexec-bounces@lists.fedoraproject.org Thu Mar 18 13:18:14 2021 Delivered-To: coxu@gapps.redhat.com Received: by 2002:aa7:c606:0:0:0:0:0 with SMTP id h6csp194165edq; Wed, 17 Mar 2021 22:18:14 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxuOoSAd181H1wjTFJCV+XK08m/9sfpOvw71A5FMnFXf6H1KSLIfo+EEZSAEiv48aNezdhs X-Received: by 2002:a17:906:aada:: with SMTP id kt26mr38170391ejb.137.1616044694457; Wed, 17 Mar 2021 22:18:14 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1616044694; cv=none; d=google.com; s=arc-20160816; b=KhcHM8bG9ldm6DvH/Qs9xBk4ZV2r36fWzDtgB8bruuBjupuYbKhXXOfShgp5ldTh1h 0YyDA4g04QsACZMcGDAK8bAPqGfcp2mZUU6r0MgqMtK3JQK9F87dTS+JyYBLXSCWJxoQ 21WsO9/oq984RGsJAyj5h6jOn4I+UY0tQRTkDONicx5vX/pvAJDTpDVcOLY99gCiqXvf VvJ7RxbA9WPZv9hnYH9IFCSkumIqwV4Kxw/k2XsJf5T4vBB/5w0xh6F2cH4vPuUhnlvM 9KYnf+MxH+RUObo7kpJ/hWX6EAFITYkzJon/YKk40jT/eYRZkm65IbtVwLYDWqsj+35F ll3A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:list-unsubscribe:list-subscribe:list-post :list-help:list-archive:archived-at:list-id:precedence:cc :message-id-hash:mime-version:message-id:date:subject:to:from :dkim-filter:delivered-to; bh=XJbwwGDoAjLEGV35AIAECvfMaPFMkNdM87fTzWoW5s0=; b=QfkgUzjYPUEBcoUY5cQRzctD11uGGv/mXcmqYj5qvHbmyWkWexnnzG7+5Yf3UDPUgN yMPpsPEve7lQD1Ex4jSGGrkFIBGwuyH1zZjzO6cuFzK8Pvkml2kUX97uY8bKYBA6nIve +mR+IX2M1UQbCf6oyQA2htZLh80TjcgM8phIY2ExoU46ToDNo/GXr/HcRZT5U1kIN1kN rS5cAOENiTPCA66tUgiBeL70VAe7+JVFdqZ3qhettBohMnw+iaQiHGIyopgjMxLbuNb8 wQ1btRrA210Rw/gXv53oN00DvOBqUbSwdMr5a3KYLE/2zScQsSSHzmUc0BNn6QL99v8a jNog== ARC-Authentication-Results: i=1; mx.google.com; spf=softfail (google.com: domain of transitioning kexec-bounces@lists.fedoraproject.org does not designate 170.10.133.124 as permitted sender) smtp.mailfrom=kexec-bounces@lists.fedoraproject.org Return-Path: <kexec-bounces@lists.fedoraproject.org> Received: from us-smtp-1.mimecast.com (us-smtp-delivery-1.mimecast.com. [207.211.31.120]) by mx.google.com with ESMTPS id bt16si684886edb.440.2021.03.17.22.18.14 for <coxu@gapps.redhat.com> (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 17 Mar 2021 22:18:14 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning kexec-bounces@lists.fedoraproject.org does not designate 170.10.133.124 as permitted sender) client-ip=170.10.133.124; Authentication-Results: mx.google.com; spf=softfail (google.com: domain of transitioning kexec-bounces@lists.fedoraproject.org does not designate 170.10.133.124 as permitted sender) smtp.mailfrom=kexec-bounces@lists.fedoraproject.org Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-295-uqOC2s1NN7CGW2R7NHQhuQ-1; Thu, 18 Mar 2021 01:18:11 -0400 X-MC-Unique: uqOC2s1NN7CGW2R7NHQhuQ-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 40425801817 for <coxu@gapps.redhat.com>; Thu, 18 Mar 2021 05:18:10 +0000 (UTC) Received: by smtp.corp.redhat.com (Postfix) id 317371F41E; Thu, 18 Mar 2021 05:18:10 +0000 (UTC) Delivered-To: coxu@redhat.com Received: from mx1.redhat.com (ext-mx12.extmail.prod.ext.phx2.redhat.com [10.5.110.41]) by smtp.corp.redhat.com (Postfix) with ESMTPS id D6EDC1F403; Thu, 18 Mar 2021 05:17:47 +0000 (UTC) Received: from bastion.fedoraproject.org (bastion01.iad2.fedoraproject.org [10.3.163.31]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 45DC5308A981; Thu, 18 Mar 2021 05:17:40 +0000 (UTC) Received: from mailman01.iad2.fedoraproject.org (mailman01.iad2.fedoraproject.org [10.3.163.57]) by bastion01.iad2.fedoraproject.org (Postfix) with ESMTP id C35533072625; Thu, 18 Mar 2021 05:12:45 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 bastion01.iad2.fedoraproject.org C35533072625 Received: from mailman01.iad2.fedoraproject.org (localhost [IPv6:::1]) by mailman01.iad2.fedoraproject.org (Postfix) with ESMTP id BCE697705FAFF; Thu, 18 Mar 2021 05:12:45 +0000 (UTC) Received: by mailman01.iad2.fedoraproject.org (Postfix, from userid 991) id B6F2B7705FB03; Thu, 18 Mar 2021 05:12:43 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on mailman01.iad2.fedoraproject.org X-Spam-Level: X-Spam-Status: No, score=-1.0 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,RCVD_IN_DNSWL_LOW,RCVD_IN_MSPIKE_H4, RCVD_IN_MSPIKE_WL,SPF_HELO_NONE autolearn=disabled version=3.4.0 Received: from smtp-mm-cc-rdu01.fedoraproject.org (smtp-mm-cc-rdu01.vpn.fedoraproject.org [192.168.1.55]) by mailman01.iad2.fedoraproject.org (Postfix) with ESMTP id 91F507705FAE2 for <kexec@lists.fedoraproject.org>; Thu, 18 Mar 2021 05:12:43 +0000 (UTC) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.mimecast.com", Issuer "DigiCert TLS RSA SHA256 2020 CA1" (verified OK)) by smtp-mm-cc-rdu01.fedoraproject.org (Postfix) with ESMTPS id 789B8306A3B1 for <kexec@lists.fedoraproject.org>; Thu, 18 Mar 2021 05:12:43 +0000 (UTC) Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-154-gOb4vuAgNk-DSIUV-gvmUQ-1; Thu, 18 Mar 2021 01:12:40 -0400 X-MC-Unique: gOb4vuAgNk-DSIUV-gvmUQ-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id A6F45801817 for <kexec@lists.fedoraproject.org>; Thu, 18 Mar 2021 05:12:39 +0000 (UTC) Received: from kasong-rh-laptop.intra.hackret.com (ovpn-13-157.pek2.redhat.com [10.72.13.157]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8F7DD18A69; Thu, 18 Mar 2021 05:12:38 +0000 (UTC) From: Kairui Song <kasong@redhat.com> To: kexec@lists.fedoraproject.org Subject: [PATCH 0/3] Use a standalone kdump emergency shell Date: Thu, 18 Mar 2021 13:12:28 +0800 Message-Id: <20210318051231.1805180-1-kasong@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Message-ID-Hash: ROKGAPNNISSV6NHO5KZEHLCSVNPVE55V X-Message-ID-Hash: ROKGAPNNISSV6NHO5KZEHLCSVNPVE55V X-MailFrom: kasong@redhat.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-config-1; header-match-config-2; header-match-config-3; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; suspicious-header CC: Kairui Song <kasong@redhat.com> X-Mailman-Version: 3.1.1 Precedence: list List-Id: List for discussing kexec/kdump issues in Fedora <kexec.lists.fedoraproject.org> Archived-At: <https://lists.fedoraproject.org/archives/list/kexec@lists.fedoraproject.org/message/ROKGAPNNISSV6NHO5KZEHLCSVNPVE55V/> List-Archive: <https://lists.fedoraproject.org/archives/list/kexec@lists.fedoraproject.org/> List-Help: <mailto:kexec-request@lists.fedoraproject.org?subject=help> List-Post: <mailto:kexec@lists.fedoraproject.org> List-Subscribe: <mailto:kexec-join@lists.fedoraproject.org> List-Unsubscribe: <mailto:kexec-leave@lists.fedoraproject.org> X-Greylist: Sender DNS name whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.41]); Thu, 18 Mar 2021 05:17:40 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.41]); Thu, 18 Mar 2021 05:17:40 +0000 (UTC) for IP:'10.3.163.31' DOMAIN:'bastion01.iad2.fedoraproject.org' HELO:'bastion.fedoraproject.org' FROM:'kexec-bounces@lists.fedoraproject.org' RCPT:'' X-RedHat-Spam-Score: -1.447 (HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,RCVD_IN_DNSWL_LOW,RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE) 10.3.163.31 bastion01.iad2.fedoraproject.org 10.3.163.31 bastion01.iad2.fedoraproject.org <kexec-bounces@lists.fedoraproject.org> X-Scanned-By: MIMEDefang 2.84 on 10.5.110.41 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: lists.fedoraproject.org Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 X-Keywords: Status: RO Content-Length: 2348 Q3VycmVudGx5IGtkdW1wIHVzZSBzb21lIHdyYXBwZXIgYXJvdW5kIGRyYWN1dCBlbWVyZ2VuY3kg c2VydmljZSBhbmQNCmVtZXJnZW5jeSBzaGVsbCwgdGhpcyBoYXZlIG1hbnkgcHJvYmxlbXM6DQoN CiAgLSBJZiBkcmFjdXQtaW5pdHF1ZXVlIGZhaWxlZCBiYWNrIHRvIGVtZXJnZW5jeSBtb2RlIGR1 ZSB0byB0aW1lb3V0LA0KICAgIGFuZCBmYWl1cmUgYWN0aW9uIGlzIHNldCB0byBkdW1wX3RvX3Jv b3Rmcywga2R1bXAgd2lsbCB0cnkgc3RhcnQNCiAgICBpbml0cXVldWUgYWdhaW4sIGxlYWQgdG8g ZG91YmxlIHRpbWVvdXQgZXJyb3IuDQogIC0gRHJhY3V0J3MgZW1lcmdlbmN5IHNoZWxsIGhhdmUg bWFueSBidWlsdGluIGFjdGlvbnMsIGxpa2UNCiAgICBwZXJmb3JtIGRyYWN1dCBlbWVyZ2VuY3lf YWN0aW9uLCBhc2sgZm9yIHJvb3QgcGFzc3dvcmQsIGdlbmVyYXRlDQogICAgcmRzb3NyZXBvcnQg ZXRjLg0KDQpUaGlzIHNlcmllcyByZW1vdmUgdGhlIGVtZXJnZW5jeSB3cmFwcGVyLCBhbmQgdXNl IGEgc3RhbmRhbG9uZSBrZHVtcA0KZW1lcmdlbmN5IHNoZWxsLiBLZHVtcCB3aWxsIGFsd2F5cyBw ZXJmb3JtIGZpbmFsX2FjdGlvbiBhZnRlciBrZHVtcA0Kc2hlbGwsIHNvIHNpbXBsaWZpZWQgdmVy c2lvbiB3b3JrcyBmaW5lLg0KDQpLYWlydWkgU29uZyAoMyk6DQogIERvbidzIHRyeSB0byByZXN0 YXJ0IGRyYWN1dC1pbml0cXVldWUgaWYgaXQncyBhbHJlYWR5IHRoZXJlDQogIFJlbW92ZSB0aGUg a2R1bXAgZXJyb3IgaGFuZGxlciBpc29sYXRpb24gd3JhcHBlcg0KICBVc2UgYSBjdXN0b21pemVk IGVtZXJnZW5jeSBzaGVsbA0KDQogZHJhY3V0LWtkdW1wLWVtZXJnZW5jeS5zZXJ2aWNlICAgICB8 IDI1ICsrKysrKysrKy0tLS0tLS0tLS0NCiBkcmFjdXQta2R1bXAtZXJyb3ItaGFuZGxlci5zZXJ2 aWNlIHwgMzMgLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tDQogZHJhY3V0LW1vZHVsZS1zZXR1cC5z aCAgICAgICAgICAgICB8ICAxIC0NCiBrZHVtcC1saWItaW5pdHJhbWZzLnNoICAgICAgICAgICAg IHwgNDAgKysrKysrKysrKysrKysrKysrKysrKysrKy0tLS0tDQoga2V4ZWMtdG9vbHMuc3BlYyAg ICAgICAgICAgICAgICAgICB8ICAyIC0tDQogNSBmaWxlcyBjaGFuZ2VkLCA0NiBpbnNlcnRpb25z KCspLCA1NSBkZWxldGlvbnMoLSkNCiBkZWxldGUgbW9kZSAxMDA2NDQgZHJhY3V0LWtkdW1wLWVy cm9yLWhhbmRsZXIuc2VydmljZQ0KDQotLSANCjIuMjkuMg0KX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX18Ka2V4ZWMgbWFpbGluZyBsaXN0IC0tIGtleGVjQGxp c3RzLmZlZG9yYXByb2plY3Qub3JnClRvIHVuc3Vic2NyaWJlIHNlbmQgYW4gZW1haWwgdG8ga2V4 ZWMtbGVhdmVAbGlzdHMuZmVkb3JhcHJvamVjdC5vcmcKRmVkb3JhIENvZGUgb2YgQ29uZHVjdDog aHR0cHM6Ly9kb2NzLmZlZG9yYXByb2plY3Qub3JnL2VuLVVTL3Byb2plY3QvY29kZS1vZi1jb25k dWN0LwpMaXN0IEd1aWRlbGluZXM6IGh0dHBzOi8vZmVkb3JhcHJvamVjdC5vcmcvd2lraS9NYWls aW5nX2xpc3RfZ3VpZGVsaW5lcwpMaXN0IEFyY2hpdmVzOiBodHRwczovL2xpc3RzLmZlZG9yYXBy b2plY3Qub3JnL2FyY2hpdmVzL2xpc3Qva2V4ZWNAbGlzdHMuZmVkb3JhcHJvamVjdC5vcmcKRG8g bm90IHJlcGx5IHRvIHNwYW0gb24gdGhlIGxpc3QsIHJlcG9ydCBpdDogaHR0cHM6Ly9wYWd1cmUu aW8vZmVkb3JhLWluZnJhc3RydWN0dXJlCg==
Usually the inclusion of ^M means that the file was sent in either Mac format or Windows. The control of this inclusion of being in base64 etc is a client side item way before it gets to our systems. [I don't see anything on our side which convert the email attachments into base64 by our tools. However I am not ruling this out.] The usual 'solution' recommended at various sites is to have git config --global core.autocrlf true so that git does the right conversion for you. However, I do not know what that might break in your workflow.
Unfortunately, "git config --global core.autocrlf true" doesn't work. It even automatically add ^M for me when I checkout a branch on Linux.
So, the only thing bastion might be doing is adding a DKIM signature... which shouldn't change the contents of the message any. ;(
I do see above:
X-Scanned-By: MIMEDefang 2.84 on 10.5.110.41 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23
Could those internal hosts be doing something here?
Does running 'dos2unix' on the emails covert them back to where you can apply them?
@coiby is there any update here
So, the only thing bastion might be doing is adding a DKIM signature... which shouldn't change the contents of the message any. ;( I do see above: X-Scanned-By: MIMEDefang 2.84 on 10.5.110.41 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Could those internal hosts be doing something here? Does running 'dos2unix' on the emails covert them back to where you can apply them?
Sorry, somehow I didn't notice the previous reply.
dos2unix doesn't work maybe because the email body is base64-decoded. However if I extract the email body and decode it, I notice the trailing ^Ms get removed after running dos2unix.
This isn't an exact replication of the problem, but this is what I did in experiment.
To get the ^Ms:
wl-paste | base64 -d > foo.txt
To eliminate the ^Ms:
wl-paste | base64 -d | dos2unix > foo2.txt
@eddiejennings Thanks for sharing your experiment!
Based on a clue shared by my college, I did more investigation and think this is the problem of "git am".
According to rfc 2045 [1], it's correct for mail server covert line breaks into "^M" (i.e. CRLF sequences) when doing base64-content transfer.
6.8. Base64 Content-Transfer-Encoding ... Care must be taken to use the proper octets for line breaks if base64 encoding is applied directly to text material that has not been converted to canonical form. In particular, text line breaks must be converted into CRLF sequences prior to base64 encoding. The important thing to note is that this may be done directly by the encoder rather than in a prior canonicalization step in some implementations.
And it's the email client's responsibility to convert these CRLF sequences into proper line breaks when doing base64 decoding . For example, mutt's decode+copy (alt+shift+c) could successfully decode these based64-encoded emails and remove the traiming ^Ms. However "git am" as an email client doesn't convert these CRLF sequences after doing base64 decoding. This issue seems to have been there for a few years, 1. "git-am doesn't strip CRLF line endings when the mbox is base64-encoded - George Dunlap" https://lore.kernel.org/git/c44c3958-b0eb-22bd-bc35-04982706162f@citrix.com/) 2. "1469098 – git am doesn't apply for base64 encoded mail" https://bugzilla.redhat.com/show_bug.cgi?id=1469098)
It seems git developer somehow doesn't want to fix this seemingly easy problem.
[1] https://bugs.kde.org/show_bug.cgi?id=55624
I'm not sure there's much more we can do here. ;(
It's very hard to tell if upstream git has addressed this or not... (all git bugs go to a mailing list and I wasn't able to find a clear report from a few minutes of searching).
I'd suggest you try 'git bugreport' and see if upstream is willing to address it?
Metadata Update from @kevin: - Issue close_status updated to: Will Not/Can Not fix - Issue status updated to: Closed (was: Open)
@kevin Today I happened to read the text "Send patches in base64 attachments" on [1] which made me realize bastion or the internal hosts may choose to base64-encode the emails. Because if I "git send-email" the patches to my maibox directly, they are still plain text.
[1] http://www.kroah.com/log/linux/maintainer.html
Login to comment on this ticket.