Hi,
I am trying to run git clone https://src.fedoraproject.org/rpms/kernel.git fedora-kernel command which is failing:
git clone https://src.fedoraproject.org/rpms/kernel.git fedora-kernel
$ git --version git version 2.30.1 $ git clone https://src.fedoraproject.org/rpms/kernel.git fedora-kernel Cloning into 'fedora-kernel'... remote: Enumerating objects: 82940, done. remote: Counting objects: 100% (82940/82940), done. remote: Compressing objects: 100% (22693/22693), done. fetch-pack: unexpected disconnect while reading sideband packet fatal: early EOF fatal: index-pack failed
$ git --version git version 2.29.2 $ git clone https://src.fedoraproject.org/rpms/kernel.git fedora-kernel Cloning into 'fedora-kernel'... remote: Enumerating objects: 82940, done. remote: Counting objects: 100% (82940/82940), done. remote: Compressing objects: 100% (22693/22693), done. fatal: the remote end hung up unexpectedly19 MiB | 723.00 KiB/s fatal: early EOF fatal: index-pack failed
As you can see I tried different git versions on different machines. So I would rule out git problem on my end, including any connection problem on my side as well.
We will need to know: 1. what are the ips of these systems 2. when these connections were not working.
Let's start with 89.0.255.101 which is still failing.
Last attempt: 2021-02-18 16:50 UTC.
Thanks. that helps. I can see your connections and 'everything' logged in the apache logs looks right with no errors. I was able to clone this repository and was getting 18.9 Mib/s while you were getting 723.00 KiB/s. I am expecting that the Atlantic network links are saturated at the moment and that is causing your timeouts and closes.
[ssmoogen@localhost Fedora]$ fedpkg clone kernel Cloning into 'kernel'... remote: Enumerating objects: 82940, done. remote: Counting objects: 100% (82940/82940), done. remote: Compressing objects: 100% (22693/22693), done. remote: Total 82940 (delta 50690), reused 78049 (delta 46686), pack-reused 0 Receiving objects: 100% (82940/82940), 132.40 MiB | 13.77 MiB/s, done. Resolving deltas: 100% (50690/50690), done. [ssmoogen@localhost Fedora]$ git clone https://src.fedoraproject.org/rpms/kernel.git fedora-kernel Cloning into 'fedora-kernel'... remote: Enumerating objects: 82940, done. remote: Counting objects: 100% (82940/82940), done. remote: Compressing objects: 100% (22693/22693), done. remote: Total 82940 (delta 50694), reused 78048 (delta 46686), pack-reused 0 Receiving objects: 100% (82940/82940), 132.47 MiB | 18.90 MiB/s, done. Resolving deltas: 100% (50694/50694), done. [ssmoogen@localhost Fedora]$ git --version git version 2.29.2
I will see if our networking people can point to any thing we can do to help here.
Metadata Update from @smooge: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: medium-gain, medium-trouble, ops, src.fp.o
Metadata Update from @smooge: - Issue untagged with: medium-gain, medium-trouble - Issue tagged with: low-gain, low-trouble
Is this problem still happening?
Yes, still happening, just tried from 89.1.54.154.
Can you give us the output of:
GIT_CURL_VERBOSE=1 git clone https://src.fedoraproject.org/rpms/kernel.git fedora-kernel
(or at least just the last 100's of lines)
First attempt was successful but failed the same on 2nd attempt:
GIT_CURL_VERBOSE=1 git clone https://src.fedoraproject.org/rpms/kernel.git fedora-kernel Cloning into 'fedora-kernel'... 22:16:04.978398 http.c:756 == Info: Couldn't find host src.fedoraproject.org in the .netrc file; using defaults 22:16:05.082196 http.c:756 == Info: Trying 38.145.60.21:443... 22:16:05.179267 http.c:756 == Info: Connected to src.fedoraproject.org (38.145.60.21) port 443 (#0) 22:16:05.180076 http.c:756 == Info: ALPN, offering h2 22:16:05.180104 http.c:756 == Info: ALPN, offering http/1.1 22:16:05.194123 http.c:756 == Info: successfully set certificate verify locations: 22:16:05.194145 http.c:756 == Info: CAfile: /etc/ssl/certs/ca-certificates.crt 22:16:05.194152 http.c:756 == Info: CApath: /etc/ssl/certs 22:16:05.194581 http.c:756 == Info: TLSv1.3 (OUT), TLS handshake, Client hello (1): 22:16:05.295058 http.c:756 == Info: TLSv1.3 (IN), TLS handshake, Server hello (2): 22:16:05.295524 http.c:756 == Info: TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): 22:16:05.295645 http.c:756 == Info: TLSv1.3 (IN), TLS handshake, Certificate (11): 22:16:05.296509 http.c:756 == Info: TLSv1.3 (IN), TLS handshake, CERT verify (15): 22:16:05.296903 http.c:756 == Info: TLSv1.3 (IN), TLS handshake, Finished (20): 22:16:05.296998 http.c:756 == Info: TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): 22:16:05.297109 http.c:756 == Info: TLSv1.3 (OUT), TLS handshake, Finished (20): 22:16:05.297248 http.c:756 == Info: SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 22:16:05.297269 http.c:756 == Info: ALPN, server accepted to use http/1.1 22:16:05.297290 http.c:756 == Info: Server certificate: 22:16:05.297337 http.c:756 == Info: subject: C=US; ST=North Carolina; L=Raleigh; O=Red Hat, Inc.; CN=*.fedoraproject.org 22:16:05.297365 http.c:756 == Info: start date: Feb 27 00:00:00 2020 GMT 22:16:05.297388 http.c:756 == Info: expire date: Mar 2 12:00:00 2022 GMT 22:16:05.297427 http.c:756 == Info: subjectAltName: host "src.fedoraproject.org" matched cert's "*.fedoraproject.org" 22:16:05.297474 http.c:756 == Info: issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 High Assurance Server CA 22:16:05.297495 http.c:756 == Info: SSL certificate verify ok. 22:16:05.297609 http.c:703 => Send header, 0000000241 bytes (0x000000f1) 22:16:05.297630 http.c:715 => Send header: GET /rpms/kernel.git/info/refs?service=git-upload-pack HTTP/1.1 22:16:05.297641 http.c:715 => Send header: Host: src.fedoraproject.org 22:16:05.297652 http.c:715 => Send header: User-Agent: git/2.30.1 22:16:05.297663 http.c:715 => Send header: Accept: */* 22:16:05.297673 http.c:715 => Send header: Accept-Encoding: deflate, gzip 22:16:05.297683 http.c:715 => Send header: Accept-Language: en-US, *;q=0.9 22:16:05.297693 http.c:715 => Send header: Pragma: no-cache 22:16:05.297702 http.c:715 => Send header: Git-Protocol: version=2 22:16:05.297712 http.c:715 => Send header: 22:16:05.391702 http.c:756 == Info: TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): 22:16:05.391854 http.c:756 == Info: TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): 22:16:05.391922 http.c:756 == Info: old SSL session ID is stale, removing 22:16:05.580805 http.c:756 == Info: Mark bundle as not supporting multiuse 22:16:05.580854 http.c:703 <= Recv header, 0000000017 bytes (0x00000011) 22:16:05.580865 http.c:715 <= Recv header: HTTP/1.1 200 OK 22:16:05.580873 http.c:703 <= Recv header, 0000000037 bytes (0x00000025) 22:16:05.580880 http.c:715 <= Recv header: Date: Mon, 22 Feb 2021 21:16:31 GMT 22:16:05.580893 http.c:703 <= Recv header, 0000000016 bytes (0x00000010) 22:16:05.580905 http.c:715 <= Recv header: Server: Apache 22:16:05.580920 http.c:703 <= Recv header, 0000000073 bytes (0x00000049) 22:16:05.580934 http.c:715 <= Recv header: Strict-Transport-Security: max-age=31536000; includeSubDomains; preload 22:16:05.580952 http.c:703 <= Recv header, 0000000029 bytes (0x0000001d) 22:16:05.580977 http.c:715 <= Recv header: X-Frame-Options: SAMEORIGIN 22:16:05.580997 http.c:703 <= Recv header, 0000000033 bytes (0x00000021) 22:16:05.581008 http.c:715 <= Recv header: X-Xss-Protection: 1; mode=block 22:16:05.581019 http.c:703 <= Recv header, 0000000033 bytes (0x00000021) 22:16:05.581032 http.c:715 <= Recv header: X-Content-Type-Options: nosniff 22:16:05.581046 http.c:703 <= Recv header, 0000000030 bytes (0x0000001e) 22:16:05.581057 http.c:715 <= Recv header: Referrer-Policy: same-origin 22:16:05.581069 http.c:703 <= Recv header, 0000000040 bytes (0x00000028) 22:16:05.581080 http.c:715 <= Recv header: expires: Tue, 01 Jan 1980 00:00:00 GMT 22:16:05.581092 http.c:703 <= Recv header, 0000000018 bytes (0x00000012) 22:16:05.581104 http.c:715 <= Recv header: pragma: no-cache 22:16:05.581118 http.c:703 <= Recv header, 0000000053 bytes (0x00000035) 22:16:05.581128 http.c:715 <= Recv header: cache-control: no-cache, max-age=0, must-revalidate 22:16:05.581144 http.c:703 <= Recv header, 0000000386 bytes (0x00000182) 22:16:05.581158 http.c:715 <= Recv header: content-security-policy: default-src 'self'; script-src 'self' 'nonce-yNKh6lQZzJkPmtfcsjUvnLbbx' https://apps.fedoraproject.org https://mdapi.fedoraproject.org; style-src 'self' 'nonce-yNKh6lQZzJkPmtfcsjUvnLbbx'; object-src 'none'; base-uri 'self'; img-src 'self' https:; connect-src 'self' https://pdc.fedoraproject.org https://apps.fedoraproject.org https://mdapi.fedoraproject.org; 22:16:05.581180 http.c:703 <= Recv header, 0000000019 bytes (0x00000013) 22:16:05.581200 http.c:715 <= Recv header: apptime: D=108251 22:16:05.581215 http.c:703 <= Recv header, 0000000051 bytes (0x00000033) 22:16:05.581227 http.c:715 <= Recv header: x-fedora-appserver: pkgs01.iad2.fedoraproject.org 22:16:05.581246 http.c:703 <= Recv header, 0000000059 bytes (0x0000003b) 22:16:05.581261 http.c:715 <= Recv header: content-type: application/x-git-upload-pack-advertisement 22:16:05.581282 http.c:703 <= Recv header, 0000000228 bytes (0x000000e4) 22:16:05.581301 http.c:715 <= Recv header: set-cookie: disgit_pagure=eyJfcGVybWFuZW50Ijp0cnVlLCJjc3JmX3Rva2VuIjoiZTk2MTc0ODYwMTFkNWNmMzhjZmU1NzM5N2ZiNjdhYWRmNjYwYWQ3MiJ9.ExWwrw.6jDw4881JerXc3vvL1hWNXLPRhQ; Expires=Thu, 25-Mar-2021 21:16:31 GMT; Secure; HttpOnly; Path=/ 22:16:05.581319 http.c:703 <= Recv header, 0000000054 bytes (0x00000036) 22:16:05.581331 http.c:715 <= Recv header: X-Fedora-ProxyServer: proxy10.iad2.fedoraproject.org 22:16:05.581348 http.c:703 <= Recv header, 0000000049 bytes (0x00000031) 22:16:05.581360 http.c:715 <= Recv header: X-Fedora-RequestID: YDQfL27mWIUbBjM2F91@3wAACAU 22:16:05.581376 http.c:703 <= Recv header, 0000000028 bytes (0x0000001c) 22:16:05.581388 http.c:715 <= Recv header: Transfer-Encoding: chunked 22:16:05.581401 http.c:703 <= Recv header, 0000000002 bytes (0x00000002) 22:16:05.581413 http.c:715 <= Recv header: 22:16:05.581467 http.c:756 == Info: Connection #0 to host src.fedoraproject.org left intact 22:16:05.588570 http.c:756 == Info: Couldn't find host src.fedoraproject.org in the .netrc file; using defaults 22:16:05.588615 http.c:756 == Info: Found bundle for host src.fedoraproject.org: 0x564084669850 [serially] 22:16:05.588630 http.c:756 == Info: Can not multiplex, even if we wanted to! 22:16:05.588668 http.c:756 == Info: Re-using existing connection! (#0) with host src.fedoraproject.org 22:16:05.588687 http.c:756 == Info: Connected to src.fedoraproject.org (38.145.60.21) port 443 (#0) 22:16:05.588915 http.c:703 => Send header, 0000000280 bytes (0x00000118) 22:16:05.588939 http.c:715 => Send header: POST /rpms/kernel.git/git-upload-pack HTTP/1.1 22:16:05.588955 http.c:715 => Send header: Host: src.fedoraproject.org 22:16:05.588969 http.c:715 => Send header: User-Agent: git/2.30.1 22:16:05.588985 http.c:715 => Send header: Accept-Encoding: deflate, gzip 22:16:05.588996 http.c:715 => Send header: Content-Type: application/x-git-upload-pack-request 22:16:05.589005 http.c:715 => Send header: Accept: application/x-git-upload-pack-result 22:16:05.589021 http.c:715 => Send header: Content-Encoding: gzip 22:16:05.589032 http.c:715 => Send header: Content-Length: 1271 22:16:05.589041 http.c:715 => Send header: 22:16:08.039022 http.c:756 == Info: Mark bundle as not supporting multiuse 22:16:08.039083 http.c:703 <= Recv header, 0000000017 bytes (0x00000011) 22:16:08.039099 http.c:715 <= Recv header: HTTP/1.1 200 OK 22:16:08.039108 http.c:703 <= Recv header, 0000000037 bytes (0x00000025) 22:16:08.039118 http.c:715 <= Recv header: Date: Mon, 22 Feb 2021 21:16:31 GMT 22:16:08.039132 http.c:703 <= Recv header, 0000000016 bytes (0x00000010) 22:16:08.039144 http.c:715 <= Recv header: Server: Apache 22:16:08.039157 http.c:703 <= Recv header, 0000000073 bytes (0x00000049) 22:16:08.039174 http.c:715 <= Recv header: Strict-Transport-Security: max-age=31536000; includeSubDomains; preload 22:16:08.039189 http.c:703 <= Recv header, 0000000029 bytes (0x0000001d) 22:16:08.039202 http.c:715 <= Recv header: X-Frame-Options: SAMEORIGIN 22:16:08.039219 http.c:703 <= Recv header, 0000000033 bytes (0x00000021) 22:16:08.039233 http.c:715 <= Recv header: X-Xss-Protection: 1; mode=block 22:16:08.039248 http.c:703 <= Recv header, 0000000033 bytes (0x00000021) 22:16:08.039259 http.c:715 <= Recv header: X-Content-Type-Options: nosniff 22:16:08.039271 http.c:703 <= Recv header, 0000000030 bytes (0x0000001e) 22:16:08.039281 http.c:715 <= Recv header: Referrer-Policy: same-origin 22:16:08.039293 http.c:703 <= Recv header, 0000000040 bytes (0x00000028) 22:16:08.039304 http.c:715 <= Recv header: expires: Tue, 01 Jan 1980 00:00:00 GMT 22:16:08.039317 http.c:703 <= Recv header, 0000000018 bytes (0x00000012) 22:16:08.039328 http.c:715 <= Recv header: pragma: no-cache 22:16:08.039357 http.c:703 <= Recv header, 0000000053 bytes (0x00000035) 22:16:08.039370 http.c:715 <= Recv header: cache-control: no-cache, max-age=0, must-revalidate 22:16:08.039386 http.c:703 <= Recv header, 0000000386 bytes (0x00000182) 22:16:08.039406 http.c:715 <= Recv header: content-security-policy: default-src 'self'; script-src 'self' 'nonce-bZxAcBriGsGh72cvAj4az0V41' https://apps.fedoraproject.org https://mdapi.fedoraproject.org; style-src 'self' 'nonce-bZxAcBriGsGh72cvAj4az0V41'; object-src 'none'; base-uri 'self'; img-src 'self' https:; connect-src 'self' https://pdc.fedoraproject.org https://apps.fedoraproject.org https://mdapi.fedoraproject.org; 22:16:08.039425 http.c:703 <= Recv header, 0000000020 bytes (0x00000014) 22:16:08.039436 http.c:715 <= Recv header: apptime: D=2333353 22:16:08.039449 http.c:703 <= Recv header, 0000000051 bytes (0x00000033) 22:16:08.039461 http.c:715 <= Recv header: x-fedora-appserver: pkgs01.iad2.fedoraproject.org 22:16:08.039476 http.c:703 <= Recv header, 0000000052 bytes (0x00000034) 22:16:08.039488 http.c:715 <= Recv header: content-type: application/x-git-upload-pack-result 22:16:08.039504 http.c:703 <= Recv header, 0000000228 bytes (0x000000e4) 22:16:08.039521 http.c:715 <= Recv header: set-cookie: disgit_pagure=eyJfcGVybWFuZW50Ijp0cnVlLCJjc3JmX3Rva2VuIjoiZTgyZDAzOWVmZWQ5OWMyNmU4ZGMyNjE4ZWE2MzI4M2IzMjMyMmE3OSJ9.ExWwrw.ZKurxV2FCr9Nbr7e7-UlfRkW2vw; Expires=Thu, 25-Mar-2021 21:16:31 GMT; Secure; HttpOnly; Path=/ 22:16:08.039540 http.c:703 <= Recv header, 0000000054 bytes (0x00000036) 22:16:08.039554 http.c:715 <= Recv header: X-Fedora-ProxyServer: proxy10.iad2.fedoraproject.org 22:16:08.039572 http.c:703 <= Recv header, 0000000049 bytes (0x00000031) 22:16:08.039584 http.c:715 <= Recv header: X-Fedora-RequestID: YDQfL27mWIUbBjM2F91@5QAACBA 22:16:08.039600 http.c:703 <= Recv header, 0000000028 bytes (0x0000001c) 22:16:08.039622 http.c:715 <= Recv header: Transfer-Encoding: chunked 22:16:08.039639 http.c:703 <= Recv header, 0000000002 bytes (0x00000002) 22:16:08.039655 http.c:715 <= Recv header: remote: Enumerating objects: 82970, done. remote: Counting objects: 100% (82970/82970), done. remote: Compressing objects: 100% (22723/22723), done. 22:16:25.998726 http.c:756 == Info: Connection #0 to host src.fedoraproject.org left intact fetch-pack: unexpected disconnect while reading sideband packet fatal: early EOF fatal: index-pack failed
89.1.54.154 - - [22/Feb/2021:21:00:07 +0000] "GET /rpms/kernel.git/info/refs?service=git-upload-pack HTTP/1.1" 200 6722 "-" "git/2.30.1" 89.1.54.154 - - [22/Feb/2021:21:00:08 +0000] "POST /rpms/kernel.git/git-upload-pack HTTP/1.1" 200 50872320 "-" "git/2.30.1" 89.1.54.154 - - [22/Feb/2021:21:14:20 +0000] "GET /rpms/kernel.git/info/refs?service=git-upload-pack HTTP/1.1" 200 6722 "-" "git/2.30.1" 89.1.54.154 - - [22/Feb/2021:21:14:20 +0000] "POST /rpms/kernel.git/git-upload-pack HTTP/1.1" 200 139019434 "-" "git/2.30.1" 89.1.54.154 - - [22/Feb/2021:21:16:31 +0000] "GET /rpms/kernel.git/info/refs?service=git-upload-pack HTTP/1.1" 200 6722 "-" "git/2.30.1" 89.1.54.154 - - [22/Feb/2021:21:16:31 +0000] "POST /rpms/kernel.git/git-upload-pack HTTP/1.1" 200 36757504 "-" "git/2.30.1"
everything on our side looked 'complete'.
Does:
git config --global http.postBuffer 524288000
help any?
Are you behind a proxy?
Getting the failure also with http.postBuffer.
http.postBuffer
I am not behind a proxy. The IP addresses I shared with you are from my local ISP (NetCologne). But I also tested from dedicated servers at Hetzner and CDN77 and got the same failures (not that often like I am seeing the failure via my private VDSL dial-up connection but it is failing sometimes).
Never experienced similar problems with other git endpoints yet.
odd. Do:
git config --global core.compression 0
or
git clone --depth 1 <repo_URI>
@tmz any ideas here? :)
git config --global core.compression 0 has no impact. --depth 1 always succeeded (but of course, just 11MB vs 140MB)
--depth 1
Login to comment on this ticket.