I believe we're experiencing the same error described in #8312
I see warnings like this in the docsbuilding project in staging:
Search Line limits were exceeded, some search paths have been omitted, the applied search line is: docsbuilding.svc.cluster.local svc.cluster.local cluster.local phx2.fedoraproject.org vpn.fedoraproject.org fedoraproject.org
And a build is stuck (currently cron-1583384400-t574f).
cron-1583384400-t574f
No hard deadline
CC @jibecfed
Metadata Update from @smooge: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: OpenShift
[2020-03-05-14:01] <nirik> so, the warning in the ticket is harmless, just anoying.
Current fix for stuck builds is to restart docker but we need to investigate why this is happening.
We looked a bit more here... this might be related to nfs being full at one point or some nfs stale handle?
Needs more looking into...
So, it looks like one finished ok (but after 15 hours!) and then one after that failed, but it at least had an error in it and another one is building
How long should they take? Is it back to working as you expect now?
An easy way to see if it succeeded is the "Last build" information in the bottom of: https://docs.stg.fedoraproject.org/en-US/docs/ It says: Last build: 2020-01-02 11:32:49 UTC
Thanks for looking into this!
The build time seems odd, I'll look into that.
It's weird the site doesn't get updated even when the build succeeds. Is it possible the sync between OpenShift and the mirrors is not working? I checked the volume in OpenShift (via a console one one if the pods) and it does have the right content. But as @jibecfed points out, it doesn't propagate to the website.
Ah, this was a different problem. It was building fine, but not syncing out.
There was a stuck sync job on proxy01.stg. I killed that and ran a sync.
On doing that I noticed there's a 'fil' and 'fil.builder' dirs on the top level. The fil.building dir is... 4.5GB... which seems really large. Perhaps this is related to the long build times?
So, I think things should be back to normal, aside from the long build times.
Can you please re-open this or file a new ticket if there is anything more you need from our side?
Metadata Update from @kevin: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.