fedora-infrastructure

Clone
Source Code
GIT

#9677 Unable to clone src.fedoraproject.org/rpms/kernel.git: fetch-pack: unexpected disconnect while reading sideband packet
Opened 9 days ago by whissi. Modified 5 days ago

Open
Close issue as:
Duplicate Fixed Insufficient data Invalid Spam Upstream Will Not/Can Not fix Initiative Worthy

Hi,

I am trying to run git clone https://src.fedoraproject.org/rpms/kernel.git fedora-kernel command which is failing:

$ git --version
git version 2.30.1

$ git clone https://src.fedoraproject.org/rpms/kernel.git fedora-kernel
Cloning into 'fedora-kernel'...
remote: Enumerating objects: 82940, done.
remote: Counting objects: 100% (82940/82940), done.
remote: Compressing objects: 100% (22693/22693), done.
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: index-pack failed

$ git --version
git version 2.29.2

$ git clone https://src.fedoraproject.org/rpms/kernel.git fedora-kernel
Cloning into 'fedora-kernel'...
remote: Enumerating objects: 82940, done.
remote: Counting objects: 100% (82940/82940), done.
remote: Compressing objects: 100% (22693/22693), done.
fatal: the remote end hung up unexpectedly19 MiB | 723.00 KiB/s 
fatal: early EOF
fatal: index-pack failed

As you can see I tried different git versions on different machines. So I would rule out git problem on my end, including any connection problem on my side as well.

We will need to know:
1. what are the ips of these systems
2. when these connections were not working.

Let's start with 89.0.255.101 which is still failing.

Last attempt: 2021-02-18 16:50 UTC.

Thanks. that helps. I can see your connections and 'everything' logged in the apache logs looks right with no errors. I was able to clone this repository and was getting 18.9 Mib/s while you were getting 723.00 KiB/s. I am expecting that the Atlantic network links are saturated at the moment and that is causing your timeouts and closes. 

[ssmoogen@localhost Fedora]$ fedpkg clone kernel
Cloning into 'kernel'...
remote: Enumerating objects: 82940, done.
remote: Counting objects: 100% (82940/82940), done.
remote: Compressing objects: 100% (22693/22693), done.
remote: Total 82940 (delta 50690), reused 78049 (delta 46686), pack-reused 0
Receiving objects: 100% (82940/82940), 132.40 MiB | 13.77 MiB/s, done.
Resolving deltas: 100% (50690/50690), done.
[ssmoogen@localhost Fedora]$ git clone https://src.fedoraproject.org/rpms/kernel.git fedora-kernel
Cloning into 'fedora-kernel'...
remote: Enumerating objects: 82940, done.
remote: Counting objects: 100% (82940/82940), done.
remote: Compressing objects: 100% (22693/22693), done.
remote: Total 82940 (delta 50694), reused 78048 (delta 46686), pack-reused 0
Receiving objects: 100% (82940/82940), 132.47 MiB | 18.90 MiB/s, done.
Resolving deltas: 100% (50694/50694), done.
[ssmoogen@localhost Fedora]$ git --version
git version 2.29.2

I will see if our networking people can point to any thing we can do to help here.

Metadata Update from @smooge:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: medium-gain, medium-trouble, ops, src.fp.o
9 days ago

Metadata Update from @smooge:
- Issue untagged with: medium-gain, medium-trouble
- Issue tagged with: low-gain, low-trouble
9 days ago

Is this problem still happening?

Yes, still happening, just tried from 89.1.54.154.

Can you give us the output of:

GIT_CURL_VERBOSE=1 git clone https://src.fedoraproject.org/rpms/kernel.git fedora-kernel

(or at least just the last 100's of lines)

First attempt was successful but failed the same on 2nd attempt:

GIT_CURL_VERBOSE=1 git clone https://src.fedoraproject.org/rpms/kernel.git fedora-kernel
Cloning into 'fedora-kernel'...
22:16:04.978398 http.c:756              == Info: Couldn't find host src.fedoraproject.org in the .netrc file; using defaults
22:16:05.082196 http.c:756              == Info:   Trying 38.145.60.21:443...
22:16:05.179267 http.c:756              == Info: Connected to src.fedoraproject.org (38.145.60.21) port 443 (#0)
22:16:05.180076 http.c:756              == Info: ALPN, offering h2
22:16:05.180104 http.c:756              == Info: ALPN, offering http/1.1
22:16:05.194123 http.c:756              == Info: successfully set certificate verify locations:
22:16:05.194145 http.c:756              == Info:  CAfile: /etc/ssl/certs/ca-certificates.crt
22:16:05.194152 http.c:756              == Info:  CApath: /etc/ssl/certs
22:16:05.194581 http.c:756              == Info: TLSv1.3 (OUT), TLS handshake, Client hello (1):
22:16:05.295058 http.c:756              == Info: TLSv1.3 (IN), TLS handshake, Server hello (2):
22:16:05.295524 http.c:756              == Info: TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
22:16:05.295645 http.c:756              == Info: TLSv1.3 (IN), TLS handshake, Certificate (11):
22:16:05.296509 http.c:756              == Info: TLSv1.3 (IN), TLS handshake, CERT verify (15):
22:16:05.296903 http.c:756              == Info: TLSv1.3 (IN), TLS handshake, Finished (20):
22:16:05.296998 http.c:756              == Info: TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
22:16:05.297109 http.c:756              == Info: TLSv1.3 (OUT), TLS handshake, Finished (20):
22:16:05.297248 http.c:756              == Info: SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
22:16:05.297269 http.c:756              == Info: ALPN, server accepted to use http/1.1
22:16:05.297290 http.c:756              == Info: Server certificate:
22:16:05.297337 http.c:756              == Info:  subject: C=US; ST=North Carolina; L=Raleigh; O=Red Hat, Inc.; CN=*.fedoraproject.org
22:16:05.297365 http.c:756              == Info:  start date: Feb 27 00:00:00 2020 GMT
22:16:05.297388 http.c:756              == Info:  expire date: Mar  2 12:00:00 2022 GMT
22:16:05.297427 http.c:756              == Info:  subjectAltName: host "src.fedoraproject.org" matched cert's "*.fedoraproject.org"
22:16:05.297474 http.c:756              == Info:  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 High Assurance Server CA
22:16:05.297495 http.c:756              == Info:  SSL certificate verify ok.
22:16:05.297609 http.c:703              => Send header, 0000000241 bytes (0x000000f1)
22:16:05.297630 http.c:715              => Send header: GET /rpms/kernel.git/info/refs?service=git-upload-pack HTTP/1.1
22:16:05.297641 http.c:715              => Send header: Host: src.fedoraproject.org
22:16:05.297652 http.c:715              => Send header: User-Agent: git/2.30.1
22:16:05.297663 http.c:715              => Send header: Accept: */*
22:16:05.297673 http.c:715              => Send header: Accept-Encoding: deflate, gzip
22:16:05.297683 http.c:715              => Send header: Accept-Language: en-US, *;q=0.9
22:16:05.297693 http.c:715              => Send header: Pragma: no-cache
22:16:05.297702 http.c:715              => Send header: Git-Protocol: version=2
22:16:05.297712 http.c:715              => Send header:
22:16:05.391702 http.c:756              == Info: TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
22:16:05.391854 http.c:756              == Info: TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
22:16:05.391922 http.c:756              == Info: old SSL session ID is stale, removing
22:16:05.580805 http.c:756              == Info: Mark bundle as not supporting multiuse
22:16:05.580854 http.c:703              <= Recv header, 0000000017 bytes (0x00000011)
22:16:05.580865 http.c:715              <= Recv header: HTTP/1.1 200 OK
22:16:05.580873 http.c:703              <= Recv header, 0000000037 bytes (0x00000025)
22:16:05.580880 http.c:715              <= Recv header: Date: Mon, 22 Feb 2021 21:16:31 GMT
22:16:05.580893 http.c:703              <= Recv header, 0000000016 bytes (0x00000010)
22:16:05.580905 http.c:715              <= Recv header: Server: Apache
22:16:05.580920 http.c:703              <= Recv header, 0000000073 bytes (0x00000049)
22:16:05.580934 http.c:715              <= Recv header: Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
22:16:05.580952 http.c:703              <= Recv header, 0000000029 bytes (0x0000001d)
22:16:05.580977 http.c:715              <= Recv header: X-Frame-Options: SAMEORIGIN
22:16:05.580997 http.c:703              <= Recv header, 0000000033 bytes (0x00000021)
22:16:05.581008 http.c:715              <= Recv header: X-Xss-Protection: 1; mode=block
22:16:05.581019 http.c:703              <= Recv header, 0000000033 bytes (0x00000021)
22:16:05.581032 http.c:715              <= Recv header: X-Content-Type-Options: nosniff
22:16:05.581046 http.c:703              <= Recv header, 0000000030 bytes (0x0000001e)
22:16:05.581057 http.c:715              <= Recv header: Referrer-Policy: same-origin
22:16:05.581069 http.c:703              <= Recv header, 0000000040 bytes (0x00000028)
22:16:05.581080 http.c:715              <= Recv header: expires: Tue, 01 Jan 1980 00:00:00 GMT
22:16:05.581092 http.c:703              <= Recv header, 0000000018 bytes (0x00000012)
22:16:05.581104 http.c:715              <= Recv header: pragma: no-cache
22:16:05.581118 http.c:703              <= Recv header, 0000000053 bytes (0x00000035)
22:16:05.581128 http.c:715              <= Recv header: cache-control: no-cache, max-age=0, must-revalidate
22:16:05.581144 http.c:703              <= Recv header, 0000000386 bytes (0x00000182)
22:16:05.581158 http.c:715              <= Recv header: content-security-policy: default-src 'self'; script-src 'self' 'nonce-yNKh6lQZzJkPmtfcsjUvnLbbx' https://apps.fedoraproject.org https://mdapi.fedoraproject.org; style-src 'self' 'nonce-yNKh6lQZzJkPmtfcsjUvnLbbx'; object-src 'none'; base-uri 'self'; img-src 'self' https:; connect-src 'self' https://pdc.fedoraproject.org https://apps.fedoraproject.org https://mdapi.fedoraproject.org;
22:16:05.581180 http.c:703              <= Recv header, 0000000019 bytes (0x00000013)
22:16:05.581200 http.c:715              <= Recv header: apptime: D=108251
22:16:05.581215 http.c:703              <= Recv header, 0000000051 bytes (0x00000033)
22:16:05.581227 http.c:715              <= Recv header: x-fedora-appserver: pkgs01.iad2.fedoraproject.org
22:16:05.581246 http.c:703              <= Recv header, 0000000059 bytes (0x0000003b)
22:16:05.581261 http.c:715              <= Recv header: content-type: application/x-git-upload-pack-advertisement
22:16:05.581282 http.c:703              <= Recv header, 0000000228 bytes (0x000000e4)
22:16:05.581301 http.c:715              <= Recv header: set-cookie: disgit_pagure=eyJfcGVybWFuZW50Ijp0cnVlLCJjc3JmX3Rva2VuIjoiZTk2MTc0ODYwMTFkNWNmMzhjZmU1NzM5N2ZiNjdhYWRmNjYwYWQ3MiJ9.ExWwrw.6jDw4881JerXc3vvL1hWNXLPRhQ; Expires=Thu, 25-Mar-2021 21:16:31 GMT; Secure; HttpOnly; Path=/
22:16:05.581319 http.c:703              <= Recv header, 0000000054 bytes (0x00000036)
22:16:05.581331 http.c:715              <= Recv header: X-Fedora-ProxyServer: proxy10.iad2.fedoraproject.org
22:16:05.581348 http.c:703              <= Recv header, 0000000049 bytes (0x00000031)
22:16:05.581360 http.c:715              <= Recv header: X-Fedora-RequestID: YDQfL27mWIUbBjM2F91@3wAACAU
22:16:05.581376 http.c:703              <= Recv header, 0000000028 bytes (0x0000001c)
22:16:05.581388 http.c:715              <= Recv header: Transfer-Encoding: chunked
22:16:05.581401 http.c:703              <= Recv header, 0000000002 bytes (0x00000002)
22:16:05.581413 http.c:715              <= Recv header:
22:16:05.581467 http.c:756              == Info: Connection #0 to host src.fedoraproject.org left intact
22:16:05.588570 http.c:756              == Info: Couldn't find host src.fedoraproject.org in the .netrc file; using defaults
22:16:05.588615 http.c:756              == Info: Found bundle for host src.fedoraproject.org: 0x564084669850 [serially]
22:16:05.588630 http.c:756              == Info: Can not multiplex, even if we wanted to!
22:16:05.588668 http.c:756              == Info: Re-using existing connection! (#0) with host src.fedoraproject.org
22:16:05.588687 http.c:756              == Info: Connected to src.fedoraproject.org (38.145.60.21) port 443 (#0)
22:16:05.588915 http.c:703              => Send header, 0000000280 bytes (0x00000118)
22:16:05.588939 http.c:715              => Send header: POST /rpms/kernel.git/git-upload-pack HTTP/1.1
22:16:05.588955 http.c:715              => Send header: Host: src.fedoraproject.org
22:16:05.588969 http.c:715              => Send header: User-Agent: git/2.30.1
22:16:05.588985 http.c:715              => Send header: Accept-Encoding: deflate, gzip
22:16:05.588996 http.c:715              => Send header: Content-Type: application/x-git-upload-pack-request
22:16:05.589005 http.c:715              => Send header: Accept: application/x-git-upload-pack-result
22:16:05.589021 http.c:715              => Send header: Content-Encoding: gzip
22:16:05.589032 http.c:715              => Send header: Content-Length: 1271
22:16:05.589041 http.c:715              => Send header:
22:16:08.039022 http.c:756              == Info: Mark bundle as not supporting multiuse
22:16:08.039083 http.c:703              <= Recv header, 0000000017 bytes (0x00000011)
22:16:08.039099 http.c:715              <= Recv header: HTTP/1.1 200 OK
22:16:08.039108 http.c:703              <= Recv header, 0000000037 bytes (0x00000025)
22:16:08.039118 http.c:715              <= Recv header: Date: Mon, 22 Feb 2021 21:16:31 GMT
22:16:08.039132 http.c:703              <= Recv header, 0000000016 bytes (0x00000010)
22:16:08.039144 http.c:715              <= Recv header: Server: Apache
22:16:08.039157 http.c:703              <= Recv header, 0000000073 bytes (0x00000049)
22:16:08.039174 http.c:715              <= Recv header: Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
22:16:08.039189 http.c:703              <= Recv header, 0000000029 bytes (0x0000001d)
22:16:08.039202 http.c:715              <= Recv header: X-Frame-Options: SAMEORIGIN
22:16:08.039219 http.c:703              <= Recv header, 0000000033 bytes (0x00000021)
22:16:08.039233 http.c:715              <= Recv header: X-Xss-Protection: 1; mode=block
22:16:08.039248 http.c:703              <= Recv header, 0000000033 bytes (0x00000021)
22:16:08.039259 http.c:715              <= Recv header: X-Content-Type-Options: nosniff
22:16:08.039271 http.c:703              <= Recv header, 0000000030 bytes (0x0000001e)
22:16:08.039281 http.c:715              <= Recv header: Referrer-Policy: same-origin
22:16:08.039293 http.c:703              <= Recv header, 0000000040 bytes (0x00000028)
22:16:08.039304 http.c:715              <= Recv header: expires: Tue, 01 Jan 1980 00:00:00 GMT
22:16:08.039317 http.c:703              <= Recv header, 0000000018 bytes (0x00000012)
22:16:08.039328 http.c:715              <= Recv header: pragma: no-cache
22:16:08.039357 http.c:703              <= Recv header, 0000000053 bytes (0x00000035)
22:16:08.039370 http.c:715              <= Recv header: cache-control: no-cache, max-age=0, must-revalidate
22:16:08.039386 http.c:703              <= Recv header, 0000000386 bytes (0x00000182)
22:16:08.039406 http.c:715              <= Recv header: content-security-policy: default-src 'self'; script-src 'self' 'nonce-bZxAcBriGsGh72cvAj4az0V41' https://apps.fedoraproject.org https://mdapi.fedoraproject.org; style-src 'self' 'nonce-bZxAcBriGsGh72cvAj4az0V41'; object-src 'none'; base-uri 'self'; img-src 'self' https:; connect-src 'self' https://pdc.fedoraproject.org https://apps.fedoraproject.org https://mdapi.fedoraproject.org;
22:16:08.039425 http.c:703              <= Recv header, 0000000020 bytes (0x00000014)
22:16:08.039436 http.c:715              <= Recv header: apptime: D=2333353
22:16:08.039449 http.c:703              <= Recv header, 0000000051 bytes (0x00000033)
22:16:08.039461 http.c:715              <= Recv header: x-fedora-appserver: pkgs01.iad2.fedoraproject.org
22:16:08.039476 http.c:703              <= Recv header, 0000000052 bytes (0x00000034)
22:16:08.039488 http.c:715              <= Recv header: content-type: application/x-git-upload-pack-result
22:16:08.039504 http.c:703              <= Recv header, 0000000228 bytes (0x000000e4)
22:16:08.039521 http.c:715              <= Recv header: set-cookie: disgit_pagure=eyJfcGVybWFuZW50Ijp0cnVlLCJjc3JmX3Rva2VuIjoiZTgyZDAzOWVmZWQ5OWMyNmU4ZGMyNjE4ZWE2MzI4M2IzMjMyMmE3OSJ9.ExWwrw.ZKurxV2FCr9Nbr7e7-UlfRkW2vw; Expires=Thu, 25-Mar-2021 21:16:31 GMT; Secure; HttpOnly; Path=/
22:16:08.039540 http.c:703              <= Recv header, 0000000054 bytes (0x00000036)
22:16:08.039554 http.c:715              <= Recv header: X-Fedora-ProxyServer: proxy10.iad2.fedoraproject.org
22:16:08.039572 http.c:703              <= Recv header, 0000000049 bytes (0x00000031)
22:16:08.039584 http.c:715              <= Recv header: X-Fedora-RequestID: YDQfL27mWIUbBjM2F91@5QAACBA
22:16:08.039600 http.c:703              <= Recv header, 0000000028 bytes (0x0000001c)
22:16:08.039622 http.c:715              <= Recv header: Transfer-Encoding: chunked
22:16:08.039639 http.c:703              <= Recv header, 0000000002 bytes (0x00000002)
22:16:08.039655 http.c:715              <= Recv header:
remote: Enumerating objects: 82970, done.
remote: Counting objects: 100% (82970/82970), done.
remote: Compressing objects: 100% (22723/22723), done.
22:16:25.998726 http.c:756              == Info: Connection #0 to host src.fedoraproject.org left intact
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: index-pack failed

89.1.54.154 - - [22/Feb/2021:21:00:07 +0000] "GET /rpms/kernel.git/info/refs?service=git-upload-pack HTTP/1.1" 200 6722 "-" "git/2.30.1"
89.1.54.154 - - [22/Feb/2021:21:00:08 +0000] "POST /rpms/kernel.git/git-upload-pack HTTP/1.1" 200 50872320 "-" "git/2.30.1"
89.1.54.154 - - [22/Feb/2021:21:14:20 +0000] "GET /rpms/kernel.git/info/refs?service=git-upload-pack HTTP/1.1" 200 6722 "-" "git/2.30.1"
89.1.54.154 - - [22/Feb/2021:21:14:20 +0000] "POST /rpms/kernel.git/git-upload-pack HTTP/1.1" 200 139019434 "-" "git/2.30.1"
89.1.54.154 - - [22/Feb/2021:21:16:31 +0000] "GET /rpms/kernel.git/info/refs?service=git-upload-pack HTTP/1.1" 200 6722 "-" "git/2.30.1"
89.1.54.154 - - [22/Feb/2021:21:16:31 +0000] "POST /rpms/kernel.git/git-upload-pack HTTP/1.1" 200 36757504 "-" "git/2.30.1"

everything on our side looked 'complete'.

Does:

git config --global http.postBuffer 524288000

help any?

Are you behind a proxy?

Getting the failure also with http.postBuffer.

I am not behind a proxy. The IP addresses I shared with you are from my local ISP (NetCologne).
But I also tested from dedicated servers at Hetzner and CDN77 and got the same failures (not that often like I am seeing the failure via my private VDSL dial-up connection but it is failing sometimes).

Never experienced similar problems with other git endpoints yet.

odd. Do:

git config --global core.compression 0

or

git clone --depth 1 <repo_URI>

help any?

@tmz any ideas here? :)

git config --global core.compression 0 has no impact.
--depth 1 always succeeded (but of course, just 11MB vs 140MB)

Login to comment on this ticket.

Metadata
None

src.fp.o ops low-trouble low-gain

None
None
Waiting on Assignee
Boards 1
ops Status: Backlog
Powered by Pagure 5.13.2
Documentation • File an Issue • About • SSH Hostkey/Fingerprint
© Red Hat, Inc. and others.