Xen2's nfs failed in a spectacular way this morning. Still looking on why and how to ensure it won't happen again in the future. This caused the whole buildsystem to fail.
We continue to see issues with this. Bugs have been opened but as of yet very little progress has been made:
Some plans early next week include building a dedicated nfs vm to mitigate this issue when it comes up, also we should make it RHEL4 as it seems RHEL5's nfs isn't ready for prime time yet.
We've had multiple failures. RHEL4 was working except that it did not support our extra large (10T) ext3 filesystem so we started getting IO errors. Switching back to EL5. During the next failure we'll try to get ahold of steved (nfs expert) to look at the box.
Since migrating to the xen guest this issue has seemingly disappeared. its been many months since we've had a lockup. Closing ticket.
to comment on this ticket.