This is not something we can fix.
1. We don't run the COPR build system.. we just provide the network/power to it so it can keep going. I am asking the COPR people where this should be filed and will update the ticket.
2. Takes too long needs more data. A 2 minute push might be completely appropriate if the systems are overloaded, if the repository being pushed to is large, or a number of other factors.

Tomáš meant that he isn't able to push to https://pagure.io/copr/copr git repository; I am facing similar issues -> each git push takes about 2:18m.

I was asked by @pingou to fill this ticket today on #pagure:

<praiskup> copr doesn't seem to receive messages from pagure.io, is something broken now?
<praiskup> neither fedmsg grepper shows the messages: https://apps.fedoraproject.org/datagrepper/raw?category=pagure&delta=172800
* mikem (~mikem@bouncer.preoccupied.net) has joined
* hberaud (~hberaud@2a01:e0a:ff:e0e0:d20c:4ce7:fe8:ee13) has joined
* sgallagh (sid44138@gateway/web/irccloud.com/x-vzuypjtpxmhhahhc) has joined
<pingou> praiskup: does copr use something in /etc/hosts?
<praiskup> pingou, there used to be some hack on builders;  but not on infrastructure machines AFAIK
<praiskup> on affected box, there's just: 10.5.126.23 puppet.fedoraproject.org puppet puppet01 puppet01.phx2.fedoraproject.org
<praiskup> pingou, fwiw, now I noticed that pushes take extremely long
<praiskup> I mean, git pushes to pagure.io repos
* celebdor43503 (~celebdor@37.223.27.69) has joined
<pingou> praiskup: open an infra ticket
<pingou> pagure01 moved datacenter over the week-end
<pingou> there may be some fall over
<praiskup> for the messaging issue, or for the push problem, or both?
<pingou> both

This may have been due to waiting for certain fedmsg hooks to happen. We seem to have 'fixed' that as the messages being seen again. Please confirm and we can close this.

Seems to work fine, thank you.

The problem regressed back.

Can you provide some more info?

• Is it just pushes that are slow? Or pulls also?

• is it only the copr/copr repo that has the behavior? or any repos on pagure.io? (you could also test with stg.pagure.io)

• Are you using ssh or https?

• Does a new clone show the behavior?

Can you provide some more info?

Yes,

Is it just pushes that are slow? Or pulls also?

Pulls as well.

is it only the copr/copr repo that has the behavior? or any repos on
pagure.io? (you could also test with stg.pagure.io)

I tend to say any repos, I played coincidentally with stg.pagure.io, and it was
affected as well - from what I can tell.

Though the problem is solved now, so I'm unable to confirm.

Are you using ssh or https?

ssh

Does a new clone show the behavior?

Can't tell now anymore.

Hum, so all is working now? Or the problem persists... I am confused...

Now it seems to be working, but at the time of writing https://pagure.io/fedora-infrastructure/issue/7993#comment-582469 , it wasn't. I think this can be closed.

ok, makes sense. Glad it's working!

Hmm, again 2min+ so it is broken again; push/pull to/from copr/copr.git over ssh (https works fine). Push to staging (ssh://git@stg.pagure.io/copr-pagure-events-testing/copr.git) takes too long as well.

Pushing to where from where? I can't duplicate this and nothing is changing on our side. Please also include timestamps so I can try to see what is going on at that time.

Pushing to where from where?

From my box (and thrnciar's box), in Brno Red Hat office.

I can't duplicate this and nothing is changing on our side.

As far as I can tell, nothing is changing on our client side either.

At this moment, it doesn't work:

$ date ; time git pull
Wed 17 Jul 2019 01:54:27 PM CEST
Already up to date.

real    2m12.170s
user    0m0.090s
sys     0m0.046s

So please try now.

[smooge@smoogen-laptop Pagure_Repos]$ git clone https://pagure.io/copr/copr.git
Cloning into 'copr'...
remote: Counting objects: 46405, done.
remote: Compressing objects: 100% (11379/11379), done.
remote: Total 46405 (delta 34086), reused 46152 (delta 33869)
Receiving objects: 100% (46405/46405), 83.85 MiB | 2.22 MiB/s, done.
Resolving deltas: 100% (34086/34086), done.
[smooge@smoogen-laptop Pagure_Repos]$ cd copr/
[smooge@smoogen-laptop copr (master)]$ date ; time git pull
Wed Jul 17 09:01:44 EDT 2019
Already up-to-date.

real    0m0.989s
user    0m0.225s
sys 0m0.193s

checked with ssh

[smooge@smoogen-laptop Pagure_Repos]$ git clone ssh://git@pagure.io/copr/copr.git
Cloning into 'copr'...
remote: Counting objects: 46405, done.
remote: Compressing objects: 100% (11379/11379), done.
remote: Total 46405 (delta 34087), reused 46151 (delta 33869)
Receiving objects: 100% (46405/46405), 83.85 MiB | 4.35 MiB/s, done.
Resolving deltas: 100% (34087/34087), done.
[smooge@smoogen-laptop Pagure_Repos]$ cd copr
[smooge@smoogen-laptop copr (master)]$ date ; time git pull
Wed Jul 17 09:03:52 EDT 2019
Already up-to-date.

real    0m4.136s
user    0m0.121s
sys 0m0.112s

I think we need a traceroute or some debug to see what is going on because something is timing out in Brno that isn't happening fresh in US.

So a few more things to check..

1) could you do a

host pagure.io
host stg.pagure.io

2) if your git is newer than 2.16 see if an --ipv4 flag makes things faster?

3) is the repository ssh based? if it is could you see what the .ssh/config has for pagure and the authorized keys also?

Good catch, --ipv4 helps. So I suppose this is because we have broken ipv6 here in Brno, or on my box. It is weird that
- it flips from working to non-working status during the day, and
- that there's some 2minutes timeout, and fallback to ipv4 (if there's no --ipv* option)
.. but anyways, thanks a lot for this help!

I expect it is whatever orders dns lookups being helpful. It rotates through them over time. The other fix I saw was in .ssh/config to change:

AddressFamily any 

to

AddressFamily inet

which will make sure anything using ssh does not try to use ipv4. I am guessing that there may be a config tied to the old 140.211.x.y ip address also set somewhere in your configs which made sure it only went ipv4 to that one.. now that it is 8.43.85.75 it is not using ipv4 only.

OK for me to close this?

I can confirm that a no-op git pull (ssh:// remote) from Pagure takes > 2 minutes and with --ipv4 it's immediate. The same on a colleague's box. We're also in Brno's RH office. I guess we need to raise this internally.

I can confirm that a no-op git pull (ssh:// remote) from Pagure takes > 2 minutes and with --ipv4 it's immediate. The same on a colleague's box. We're also in Brno's RH office. I guess we need to raise this internally.

+1