#5397 Disk failures in the COPR build servers?
Closed: Fixed None Opened 4 years ago by openresty.

= bug description =

All my builds are always failing with the "Read-only file system" error tonight. They used to work yesterday. They now always fail with the following error in the import log:

{{{
rpkgError: Command git push returned code 1 with error: Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
remote: error: unable to create temporary file: Read-only file system
remote: fatal: failed to write object
error: unpack failed: unpack-objects abnormal exit
To ssh://copr-dist-git@localhost/openresty/openresty/openresty
! [remote rejected] f23 -> f23 (unpacker error)
error: failed to push some refs to 'ssh://copr-dist-git@localhost/openresty/openresty/openresty'
}}}

The link to one sample of the original build import logs is below:

http://copr-dist-git.fedorainfracloud.org/per-task-logs/384928-f23.log

See the build history of my "openresty" project below:

https://copr.fedorainfracloud.org/coprs/openresty/openresty/builds/

All my recent builds are red. Alas.

= bug analysis =

The "Read-only file system" error from git is usually a sign of fatal errors in the disks and the OS chose to mount them as read-only volumns.

= fix recommendation =

Drop off those bad machines or just replace the disks.


This had been a remnant of a failed maintenance window on the storage area network in the Fedora Infra Cloud.
This should now be resolved.

Login to comment on this ticket.

Metadata