• Describe what you need us to do:
  There were filesystem errors on both copr-dist-git and copr-backend production machines: dmesg from copr-dist-git:

[11548163.487614] EXT4-fs (vdc1): previous I/O error to superblock detected
[11548163.492348] Buffer I/O error on dev vdc1, logical block 0, lost sync page write
[11548163.497601] EXT4-fs error (device vdc1): ext4_find_entry:1436: inode #36702123: comm cgit: reading directory lblock 0
[11548163.502812] EXT4-fs (vdc1): previous I/O error to superblock detected
[11548163.506978] Buffer I/O error on dev vdc1, logical block 0, lost sync page write
[11548163.511598] EXT4-fs warning (device vdc1): htree_dirblock_to_tree:977: inode #36702123: lblock 0: comm cgit: error -5 reading directory block
[11548163.519660] EXT4-fs warning (device vdc1): htree_dirblock_to_tree:977: inode #45088769: lblock 2: comm cgit: error -5 reading directory block
[11548163.526799] EXT4-fs error (device vdc1): ext4_find_entry:1436: inode #36702123: comm cgit: reading directory lblock 0
[11548163.531846] EXT4-fs (vdc1): previous I/O error to superblock detected
[11548163.535852] Buffer I/O error on dev vdc1, logical block 0, lost sync page write
[11548163.539858] EXT4-fs error (device vdc1): ext4_find_entry:1436: inode #36702123: comm cgit: reading directory lblock 0
[11548163.545071] EXT4-fs (vdc1): previous I/O error to superblock detected
[11548163.549676] Buffer I/O error on dev vdc1, logical block 0, lost sync page write
[11548163.554224] EXT4-fs warning (device vdc1): htree_dirblock_to_tree:977: inode #36702123: lblock 0: comm cgit: error -5 reading directory block
[11548163.562884] EXT4-fs warning (device vdc1): htree_dirblock_to_tree:977: inode #45088769: lblock 2: comm cgit: error -5 reading directory block
[11548163.573022] EXT4-fs error (device vdc1): ext4_find_entry:1436: inode #36702123: comm cgit: reading directory lblock 0
[11548163.580313] EXT4-fs (vdc1): previous I/O error to superblock detected
[11548163.584373] Buffer I/O error on dev vdc1, logical block 0, lost sync page write
[11548163.591407] EXT4-fs error (device vdc1): ext4_find_entry:1436: inode #36702123: comm cgit: reading directory lblock 0
[11548163.596796] EXT4-fs (vdc1): previous I/O error to superblock detected
[11548163.601398] Buffer I/O error on dev vdc1, logical block 0, lost sync page write
[11548163.606140] EXT4-fs warning (device vdc1): htree_dirblock_to_tree:977: inode #36702123: lblock 0: comm cgit: error -5 reading directory block
[11548163.613421] EXT4-fs warning (device vdc1): htree_dirblock_to_tree:977: inode #45088769: lblock 2: comm cgit: error -5 reading directory block
[11548163.631674] Buffer I/O error on dev vdc1, logical block 0, lost sync page write
[11548163.637402] EXT4-fs warning (device vdc1): htree_dirblock_to_tree:977: inode #36702123: lblock 0: comm cgit: error -5 reading directory block
[11548164.568393] EXT4-fs error (device vdc1): ext4_journal_check_start:61: Detected aborted journal
[11582343.422541] EXT4-fs error (device vdc1): ext4_remount:5049: Abort forced by user

The bug describing the first time is https://bugzilla.redhat.com/show_bug.cgi?id=1579564, which states date: 2018-05-17 18:52:22 EDT

Given that this happened on two different machines. It might be worth to check if something didn't went off with the cloud storage system and whether or not it might happen again.

I ran fsck on /dev/vdc1 on both copr-backend and copr-dist-git machines and right now Copr is operating again.