I have seen the same thing this morning on rpkg:
https://jenkins.fedorainfracloud.org/job/pyrpkg/266/Builders=EL7/console

This also happens on the bodhi job.

ok. I think this is fixed now.

The disk on the el7 builder had some xfs errors, so I updated and rebooted it...

Please let us know if you still see any issues.

:paperclip:

Hi, it was working for some hours, but now getting the same error once again (only EL7) https://jenkins.fedorainfracloud.org/job/koji/label=EL7/271/console

I've been experiencing problems that seem related or identical to this still:

 > git -c core.askpass=true fetch --tags --progress https://github.com/fedora-infra/bodhi.git +refs/heads/*:refs/remotes/origin/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from https://github.com/fedora-infra/bodhi.git
        at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:763)
        at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1012)
        at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1043)
        at hudson.scm.SCM.checkout(SCM.java:485)
        at hudson.model.AbstractProject.checkout(AbstractProject.java:1269)
        at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:607)
        at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
        at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
        at hudson.model.Run.execute(Run.java:1738)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        at hudson.model.ResourceController.execute(ResourceController.java:98)
        at hudson.model.Executor.run(Executor.java:410)
Caused by: hudson.plugins.git.GitException: Command "git -c core.askpass=true fetch --tags --progress https://github.com/fedora-infra/bodhi.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: remote: Counting objects: 31278, done.        
error: cannot fork() for index-pack: Resource temporarily unavailable
fatal: fetch-pack: unable to fork off index-pack

Is this still happening?

Yes, tested now: https://jenkins.fedorainfracloud.org/job/koji/label=EL7/300/console

There were a bunch of redis-servers hanging around taking up memory/files on the el7 builder.

I've cleaned them up, can you fire another build and see if it's all working now?

@kevin The bodhi job seems to be back to normal (i.e., failing still, but not due to this issue ☺).

Koji also confirmed working, thanks!

:-( Didn't last too long... https://jenkins.fedorainfracloud.org/job/koji/308/label=EL7/console

So, there was another 2500+ redis instances again. I killed them.

I think there's some job thats starting them and not closing them out properly, but I am not sure how to identify this job. ;(

we should check if this isn't pagure

Yeah, can you see when pagure is about to do a run, check for any redis servers, run the tests, then check again?

Or alternately, add something at the end of pagure tests that kills any redis servers?

I have not seen the redis storm in a while now.

I also just updated the builder again and made sure swap was now working.

Looking back at history, it seems like it's working now?

Anyone still seeing issues? Could you link to failing builds?

I've reenabled EL7 for koji and no problems so far (2 builds only).

Hopefully this was caused by redis forking issues and is now fixed.

Please comment or re-open if anyone still sees it.

:credit_card: