fedora-infrastructure

Clone
Source Code
GIT

#9677 Unable to clone src.fedoraproject.org/rpms/kernel.git: fetch-pack: unexpected disconnect while reading sideband packet
Opened 3 months ago by whissi. Modified a month ago

Open
Close issue as:
Duplicate Fixed Insufficient data Invalid Spam Upstream Will Not/Can Not fix Initiative Worthy

Hi,

I am trying to run git clone https://src.fedoraproject.org/rpms/kernel.git fedora-kernel command which is failing:

$ git --version
git version 2.30.1

$ git clone https://src.fedoraproject.org/rpms/kernel.git fedora-kernel
Cloning into 'fedora-kernel'...
remote: Enumerating objects: 82940, done.
remote: Counting objects: 100% (82940/82940), done.
remote: Compressing objects: 100% (22693/22693), done.
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: index-pack failed

$ git --version
git version 2.29.2

$ git clone https://src.fedoraproject.org/rpms/kernel.git fedora-kernel
Cloning into 'fedora-kernel'...
remote: Enumerating objects: 82940, done.
remote: Counting objects: 100% (82940/82940), done.
remote: Compressing objects: 100% (22693/22693), done.
fatal: the remote end hung up unexpectedly19 MiB | 723.00 KiB/s 
fatal: early EOF
fatal: index-pack failed

As you can see I tried different git versions on different machines. So I would rule out git problem on my end, including any connection problem on my side as well.

We will need to know:
1. what are the ips of these systems
2. when these connections were not working.

Let's start with 89.0.255.101 which is still failing.

Last attempt: 2021-02-18 16:50 UTC.

Thanks. that helps. I can see your connections and 'everything' logged in the apache logs looks right with no errors. I was able to clone this repository and was getting 18.9 Mib/s while you were getting 723.00 KiB/s. I am expecting that the Atlantic network links are saturated at the moment and that is causing your timeouts and closes. 

[ssmoogen@localhost Fedora]$ fedpkg clone kernel
Cloning into 'kernel'...
remote: Enumerating objects: 82940, done.
remote: Counting objects: 100% (82940/82940), done.
remote: Compressing objects: 100% (22693/22693), done.
remote: Total 82940 (delta 50690), reused 78049 (delta 46686), pack-reused 0
Receiving objects: 100% (82940/82940), 132.40 MiB | 13.77 MiB/s, done.
Resolving deltas: 100% (50690/50690), done.
[ssmoogen@localhost Fedora]$ git clone https://src.fedoraproject.org/rpms/kernel.git fedora-kernel
Cloning into 'fedora-kernel'...
remote: Enumerating objects: 82940, done.
remote: Counting objects: 100% (82940/82940), done.
remote: Compressing objects: 100% (22693/22693), done.
remote: Total 82940 (delta 50694), reused 78048 (delta 46686), pack-reused 0
Receiving objects: 100% (82940/82940), 132.47 MiB | 18.90 MiB/s, done.
Resolving deltas: 100% (50694/50694), done.
[ssmoogen@localhost Fedora]$ git --version
git version 2.29.2

I will see if our networking people can point to any thing we can do to help here.

Metadata Update from @smooge:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: medium-gain, medium-trouble, ops, src.fp.o
3 months ago

Metadata Update from @smooge:
- Issue untagged with: medium-gain, medium-trouble
- Issue tagged with: low-gain, low-trouble
3 months ago

Is this problem still happening?

Yes, still happening, just tried from 89.1.54.154.

Can you give us the output of:

GIT_CURL_VERBOSE=1 git clone https://src.fedoraproject.org/rpms/kernel.git fedora-kernel

(or at least just the last 100's of lines)

First attempt was successful but failed the same on 2nd attempt:

GIT_CURL_VERBOSE=1 git clone https://src.fedoraproject.org/rpms/kernel.git fedora-kernel
Cloning into 'fedora-kernel'...
22:16:04.978398 http.c:756              == Info: Couldn't find host src.fedoraproject.org in the .netrc file; using defaults
22:16:05.082196 http.c:756              == Info:   Trying 38.145.60.21:443...
22:16:05.179267 http.c:756              == Info: Connected to src.fedoraproject.org (38.145.60.21) port 443 (#0)
22:16:05.180076 http.c:756              == Info: ALPN, offering h2
22:16:05.180104 http.c:756              == Info: ALPN, offering http/1.1
22:16:05.194123 http.c:756              == Info: successfully set certificate verify locations:
22:16:05.194145 http.c:756              == Info:  CAfile: /etc/ssl/certs/ca-certificates.crt
22:16:05.194152 http.c:756              == Info:  CApath: /etc/ssl/certs
22:16:05.194581 http.c:756              == Info: TLSv1.3 (OUT), TLS handshake, Client hello (1):
22:16:05.295058 http.c:756              == Info: TLSv1.3 (IN), TLS handshake, Server hello (2):
22:16:05.295524 http.c:756              == Info: TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
22:16:05.295645 http.c:756              == Info: TLSv1.3 (IN), TLS handshake, Certificate (11):
22:16:05.296509 http.c:756              == Info: TLSv1.3 (IN), TLS handshake, CERT verify (15):
22:16:05.296903 http.c:756              == Info: TLSv1.3 (IN), TLS handshake, Finished (20):
22:16:05.296998 http.c:756              == Info: TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
22:16:05.297109 http.c:756              == Info: TLSv1.3 (OUT), TLS handshake, Finished (20):
22:16:05.297248 http.c:756              == Info: SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
22:16:05.297269 http.c:756              == Info: ALPN, server accepted to use http/1.1
22:16:05.297290 http.c:756              == Info: Server certificate:
22:16:05.297337 http.c:756              == Info:  subject: C=US; ST=North Carolina; L=Raleigh; O=Red Hat, Inc.; CN=*.fedoraproject.org
22:16:05.297365 http.c:756              == Info:  start date: Feb 27 00:00:00 2020 GMT
22:16:05.297388 http.c:756              == Info:  expire date: Mar  2 12:00:00 2022 GMT
22:16:05.297427 http.c:756              == Info:  subjectAltName: host "src.fedoraproject.org" matched cert's "*.fedoraproject.org"
22:16:05.297474 http.c:756              == Info:  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 High Assurance Server CA
22:16:05.297495 http.c:756              == Info:  SSL certificate verify ok.
22:16:05.297609 http.c:703              => Send header, 0000000241 bytes (0x000000f1)
22:16:05.297630 http.c:715              => Send header: GET /rpms/kernel.git/info/refs?service=git-upload-pack HTTP/1.1
22:16:05.297641 http.c:715              => Send header: Host: src.fedoraproject.org
22:16:05.297652 http.c:715              => Send header: User-Agent: git/2.30.1
22:16:05.297663 http.c:715              => Send header: Accept: */*
22:16:05.297673 http.c:715              => Send header: Accept-Encoding: deflate, gzip
22:16:05.297683 http.c:715              => Send header: Accept-Language: en-US, *;q=0.9
22:16:05.297693 http.c:715              => Send header: Pragma: no-cache
22:16:05.297702 http.c:715              => Send header: Git-Protocol: version=2
22:16:05.297712 http.c:715              => Send header:
22:16:05.391702 http.c:756              == Info: TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
22:16:05.391854 http.c:756              == Info: TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
22:16:05.391922 http.c:756              == Info: old SSL session ID is stale, removing
22:16:05.580805 http.c:756              == Info: Mark bundle as not supporting multiuse
22:16:05.580854 http.c:703              <= Recv header, 0000000017 bytes (0x00000011)
22:16:05.580865 http.c:715              <= Recv header: HTTP/1.1 200 OK
22:16:05.580873 http.c:703              <= Recv header, 0000000037 bytes (0x00000025)
22:16:05.580880 http.c:715              <= Recv header: Date: Mon, 22 Feb 2021 21:16:31 GMT
22:16:05.580893 http.c:703              <= Recv header, 0000000016 bytes (0x00000010)
22:16:05.580905 http.c:715              <= Recv header: Server: Apache
22:16:05.580920 http.c:703              <= Recv header, 0000000073 bytes (0x00000049)
22:16:05.580934 http.c:715              <= Recv header: Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
22:16:05.580952 http.c:703              <= Recv header, 0000000029 bytes (0x0000001d)
22:16:05.580977 http.c:715              <= Recv header: X-Frame-Options: SAMEORIGIN
22:16:05.580997 http.c:703              <= Recv header, 0000000033 bytes (0x00000021)
22:16:05.581008 http.c:715              <= Recv header: X-Xss-Protection: 1; mode=block
22:16:05.581019 http.c:703              <= Recv header, 0000000033 bytes (0x00000021)
22:16:05.581032 http.c:715              <= Recv header: X-Content-Type-Options: nosniff
22:16:05.581046 http.c:703              <= Recv header, 0000000030 bytes (0x0000001e)
22:16:05.581057 http.c:715              <= Recv header: Referrer-Policy: same-origin
22:16:05.581069 http.c:703              <= Recv header, 0000000040 bytes (0x00000028)
22:16:05.581080 http.c:715              <= Recv header: expires: Tue, 01 Jan 1980 00:00:00 GMT
22:16:05.581092 http.c:703              <= Recv header, 0000000018 bytes (0x00000012)
22:16:05.581104 http.c:715              <= Recv header: pragma: no-cache
22:16:05.581118 http.c:703              <= Recv header, 0000000053 bytes (0x00000035)
22:16:05.581128 http.c:715              <= Recv header: cache-control: no-cache, max-age=0, must-revalidate
22:16:05.581144 http.c:703              <= Recv header, 0000000386 bytes (0x00000182)
22:16:05.581158 http.c:715              <= Recv header: content-security-policy: default-src 'self'; script-src 'self' 'nonce-yNKh6lQZzJkPmtfcsjUvnLbbx' https://apps.fedoraproject.org https://mdapi.fedoraproject.org; style-src 'self' 'nonce-yNKh6lQZzJkPmtfcsjUvnLbbx'; object-src 'none'; base-uri 'self'; img-src 'self' https:; connect-src 'self' https://pdc.fedoraproject.org https://apps.fedoraproject.org https://mdapi.fedoraproject.org;
22:16:05.581180 http.c:703              <= Recv header, 0000000019 bytes (0x00000013)
22:16:05.581200 http.c:715              <= Recv header: apptime: D=108251
22:16:05.581215 http.c:703              <= Recv header, 0000000051 bytes (0x00000033)
22:16:05.581227 http.c:715              <= Recv header: x-fedora-appserver: pkgs01.iad2.fedoraproject.org
22:16:05.581246 http.c:703              <= Recv header, 0000000059 bytes (0x0000003b)
22:16:05.581261 http.c:715              <= Recv header: content-type: application/x-git-upload-pack-advertisement
22:16:05.581282 http.c:703              <= Recv header, 0000000228 bytes (0x000000e4)
22:16:05.581301 http.c:715              <= Recv header: set-cookie: disgit_pagure=eyJfcGVybWFuZW50Ijp0cnVlLCJjc3JmX3Rva2VuIjoiZTk2MTc0ODYwMTFkNWNmMzhjZmU1NzM5N2ZiNjdhYWRmNjYwYWQ3MiJ9.ExWwrw.6jDw4881JerXc3vvL1hWNXLPRhQ; Expires=Thu, 25-Mar-2021 21:16:31 GMT; Secure; HttpOnly; Path=/
22:16:05.581319 http.c:703              <= Recv header, 0000000054 bytes (0x00000036)
22:16:05.581331 http.c:715              <= Recv header: X-Fedora-ProxyServer: proxy10.iad2.fedoraproject.org
22:16:05.581348 http.c:703              <= Recv header, 0000000049 bytes (0x00000031)
22:16:05.581360 http.c:715              <= Recv header: X-Fedora-RequestID: YDQfL27mWIUbBjM2F91@3wAACAU
22:16:05.581376 http.c:703              <= Recv header, 0000000028 bytes (0x0000001c)
22:16:05.581388 http.c:715              <= Recv header: Transfer-Encoding: chunked
22:16:05.581401 http.c:703              <= Recv header, 0000000002 bytes (0x00000002)
22:16:05.581413 http.c:715              <= Recv header:
22:16:05.581467 http.c:756              == Info: Connection #0 to host src.fedoraproject.org left intact
22:16:05.588570 http.c:756              == Info: Couldn't find host src.fedoraproject.org in the .netrc file; using defaults
22:16:05.588615 http.c:756              == Info: Found bundle for host src.fedoraproject.org: 0x564084669850 [serially]
22:16:05.588630 http.c:756              == Info: Can not multiplex, even if we wanted to!
22:16:05.588668 http.c:756              == Info: Re-using existing connection! (#0) with host src.fedoraproject.org
22:16:05.588687 http.c:756              == Info: Connected to src.fedoraproject.org (38.145.60.21) port 443 (#0)
22:16:05.588915 http.c:703              => Send header, 0000000280 bytes (0x00000118)
22:16:05.588939 http.c:715              => Send header: POST /rpms/kernel.git/git-upload-pack HTTP/1.1
22:16:05.588955 http.c:715              => Send header: Host: src.fedoraproject.org
22:16:05.588969 http.c:715              => Send header: User-Agent: git/2.30.1
22:16:05.588985 http.c:715              => Send header: Accept-Encoding: deflate, gzip
22:16:05.588996 http.c:715              => Send header: Content-Type: application/x-git-upload-pack-request
22:16:05.589005 http.c:715              => Send header: Accept: application/x-git-upload-pack-result
22:16:05.589021 http.c:715              => Send header: Content-Encoding: gzip
22:16:05.589032 http.c:715              => Send header: Content-Length: 1271
22:16:05.589041 http.c:715              => Send header:
22:16:08.039022 http.c:756              == Info: Mark bundle as not supporting multiuse
22:16:08.039083 http.c:703              <= Recv header, 0000000017 bytes (0x00000011)
22:16:08.039099 http.c:715              <= Recv header: HTTP/1.1 200 OK
22:16:08.039108 http.c:703              <= Recv header, 0000000037 bytes (0x00000025)
22:16:08.039118 http.c:715              <= Recv header: Date: Mon, 22 Feb 2021 21:16:31 GMT
22:16:08.039132 http.c:703              <= Recv header, 0000000016 bytes (0x00000010)
22:16:08.039144 http.c:715              <= Recv header: Server: Apache
22:16:08.039157 http.c:703              <= Recv header, 0000000073 bytes (0x00000049)
22:16:08.039174 http.c:715              <= Recv header: Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
22:16:08.039189 http.c:703              <= Recv header, 0000000029 bytes (0x0000001d)
22:16:08.039202 http.c:715              <= Recv header: X-Frame-Options: SAMEORIGIN
22:16:08.039219 http.c:703              <= Recv header, 0000000033 bytes (0x00000021)
22:16:08.039233 http.c:715              <= Recv header: X-Xss-Protection: 1; mode=block
22:16:08.039248 http.c:703              <= Recv header, 0000000033 bytes (0x00000021)
22:16:08.039259 http.c:715              <= Recv header: X-Content-Type-Options: nosniff
22:16:08.039271 http.c:703              <= Recv header, 0000000030 bytes (0x0000001e)
22:16:08.039281 http.c:715              <= Recv header: Referrer-Policy: same-origin
22:16:08.039293 http.c:703              <= Recv header, 0000000040 bytes (0x00000028)
22:16:08.039304 http.c:715              <= Recv header: expires: Tue, 01 Jan 1980 00:00:00 GMT
22:16:08.039317 http.c:703              <= Recv header, 0000000018 bytes (0x00000012)
22:16:08.039328 http.c:715              <= Recv header: pragma: no-cache
22:16:08.039357 http.c:703              <= Recv header, 0000000053 bytes (0x00000035)
22:16:08.039370 http.c:715              <= Recv header: cache-control: no-cache, max-age=0, must-revalidate
22:16:08.039386 http.c:703              <= Recv header, 0000000386 bytes (0x00000182)
22:16:08.039406 http.c:715              <= Recv header: content-security-policy: default-src 'self'; script-src 'self' 'nonce-bZxAcBriGsGh72cvAj4az0V41' https://apps.fedoraproject.org https://mdapi.fedoraproject.org; style-src 'self' 'nonce-bZxAcBriGsGh72cvAj4az0V41'; object-src 'none'; base-uri 'self'; img-src 'self' https:; connect-src 'self' https://pdc.fedoraproject.org https://apps.fedoraproject.org https://mdapi.fedoraproject.org;
22:16:08.039425 http.c:703              <= Recv header, 0000000020 bytes (0x00000014)
22:16:08.039436 http.c:715              <= Recv header: apptime: D=2333353
22:16:08.039449 http.c:703              <= Recv header, 0000000051 bytes (0x00000033)
22:16:08.039461 http.c:715              <= Recv header: x-fedora-appserver: pkgs01.iad2.fedoraproject.org
22:16:08.039476 http.c:703              <= Recv header, 0000000052 bytes (0x00000034)
22:16:08.039488 http.c:715              <= Recv header: content-type: application/x-git-upload-pack-result
22:16:08.039504 http.c:703              <= Recv header, 0000000228 bytes (0x000000e4)
22:16:08.039521 http.c:715              <= Recv header: set-cookie: disgit_pagure=eyJfcGVybWFuZW50Ijp0cnVlLCJjc3JmX3Rva2VuIjoiZTgyZDAzOWVmZWQ5OWMyNmU4ZGMyNjE4ZWE2MzI4M2IzMjMyMmE3OSJ9.ExWwrw.ZKurxV2FCr9Nbr7e7-UlfRkW2vw; Expires=Thu, 25-Mar-2021 21:16:31 GMT; Secure; HttpOnly; Path=/
22:16:08.039540 http.c:703              <= Recv header, 0000000054 bytes (0x00000036)
22:16:08.039554 http.c:715              <= Recv header: X-Fedora-ProxyServer: proxy10.iad2.fedoraproject.org
22:16:08.039572 http.c:703              <= Recv header, 0000000049 bytes (0x00000031)
22:16:08.039584 http.c:715              <= Recv header: X-Fedora-RequestID: YDQfL27mWIUbBjM2F91@5QAACBA
22:16:08.039600 http.c:703              <= Recv header, 0000000028 bytes (0x0000001c)
22:16:08.039622 http.c:715              <= Recv header: Transfer-Encoding: chunked
22:16:08.039639 http.c:703              <= Recv header, 0000000002 bytes (0x00000002)
22:16:08.039655 http.c:715              <= Recv header:
remote: Enumerating objects: 82970, done.
remote: Counting objects: 100% (82970/82970), done.
remote: Compressing objects: 100% (22723/22723), done.
22:16:25.998726 http.c:756              == Info: Connection #0 to host src.fedoraproject.org left intact
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: index-pack failed

89.1.54.154 - - [22/Feb/2021:21:00:07 +0000] "GET /rpms/kernel.git/info/refs?service=git-upload-pack HTTP/1.1" 200 6722 "-" "git/2.30.1"
89.1.54.154 - - [22/Feb/2021:21:00:08 +0000] "POST /rpms/kernel.git/git-upload-pack HTTP/1.1" 200 50872320 "-" "git/2.30.1"
89.1.54.154 - - [22/Feb/2021:21:14:20 +0000] "GET /rpms/kernel.git/info/refs?service=git-upload-pack HTTP/1.1" 200 6722 "-" "git/2.30.1"
89.1.54.154 - - [22/Feb/2021:21:14:20 +0000] "POST /rpms/kernel.git/git-upload-pack HTTP/1.1" 200 139019434 "-" "git/2.30.1"
89.1.54.154 - - [22/Feb/2021:21:16:31 +0000] "GET /rpms/kernel.git/info/refs?service=git-upload-pack HTTP/1.1" 200 6722 "-" "git/2.30.1"
89.1.54.154 - - [22/Feb/2021:21:16:31 +0000] "POST /rpms/kernel.git/git-upload-pack HTTP/1.1" 200 36757504 "-" "git/2.30.1"

everything on our side looked 'complete'.

Does:

git config --global http.postBuffer 524288000

help any?

Are you behind a proxy?

Getting the failure also with http.postBuffer.

I am not behind a proxy. The IP addresses I shared with you are from my local ISP (NetCologne).
But I also tested from dedicated servers at Hetzner and CDN77 and got the same failures (not that often like I am seeing the failure via my private VDSL dial-up connection but it is failing sometimes).

Never experienced similar problems with other git endpoints yet.

odd. Do:

git config --global core.compression 0

or

git clone --depth 1 <repo_URI>

help any?

@tmz any ideas here? :)

git config --global core.compression 0 has no impact.
--depth 1 always succeeded (but of course, just 11MB vs 140MB)

I'm not sure what to do here. As far as I know you are the only one reporting this, and I can't see anything obviously wrong with our config. ;(

Perhaps we could engage the upstream git developers? or the fedora/rhel ones?

Is this issue still happening?

I just wanted to report that it seems to be working now (had no problems last week) but just now when I did my last test it failed again like before :/

Can you try again now and see if you can get it to happen?

Done, failed on second attempt (from 78.34.106.145).

Login to comment on this ticket.

Metadata
None

src.fp.o ops low-trouble low-gain

None
None
Waiting on Assignee
Boards 1
ops Status: Backlog
Powered by Pagure 5.13.2
Documentation • File an Issue • About • SSH Hostkey/Fingerprint
© Red Hat, Inc. and others.