fedora-infrastructure

#7780 copr: dist-git production fails to read storage on /dev/vdc1
Closed: Fixed by praiskup. Opened by praiskup.

Closed: Fixed
Reopen Issue

dmesg:

[88380.180719] EXT4-fs (vdd1): error count since last fsck: 5
[88380.180723] EXT4-fs (vdc1): error count since last fsck: 1869993
[88380.184677] EXT4-fs (vdd1): initial error at time 1539979275: ext4_journal_check_start:61
[88380.184681] EXT4-fs (vdd1): last error at time 1540020852: ext4_mb_release_inode_pa:3857
[88380.188815] EXT4-fs (vdc1): initial error at time 1539977708: ext4_find_entry:1437: inode 99746125
[88380.190428] EXT4-fs (vdc1): last error at time 1540020854: ext4_remount:5128: inode 98721927
[176460.491027] EXT4-fs (vdd1): error count since last fsck: 5
[176460.491042] EXT4-fs (vdc1): error count since last fsck: 1869993
[176460.492613] EXT4-fs (vdd1): initial error at time 1539979275: ext4_journal_check_start:61
[176460.493683] EXT4-fs (vdc1): initial error at time 1539977708: ext4_find_entry:1437: inode 99746125
[176460.495150] EXT4-fs (vdd1): last error at time 1540020852: ext4_mb_release_inode_pa:3857
[176460.496700] EXT4-fs (vdc1): last error at time 1540020854: ext4_remount:5128: inode 98721927
[264540.791616] EXT4-fs (vdd1): error count since last fsck: 5
[264540.791619] EXT4-fs (vdc1): error count since last fsck: 1869993

fixed by kevin

Metadata Update from @praiskup:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

This was due to the storage volume trying to grow and not having enough space, so it took the volume offline. ;(

I added a bit of space and deleted some snapshots and got it back online.

I think, that this outage damaged the /var/lib/dist-git/git/@kicad/kicad/kicad.git repository and now it is not possible to clone it (and therefore build any package into the Copr @kicad/kicad project.

[root@copr-dist-git kicad.git][PROD]# pwd
/var/lib/dist-git/git/@kicad/kicad/kicad.git
[root@copr-dist-git kicad.git][PROD]# git fsck --full 
error: object file ./objects/1f/7f0414aff1105f8afec1d52a8722cad1fe8194 is empty
error: unable to mmap ./objects/1f/7f0414aff1105f8afec1d52a8722cad1fe8194: No such file or directory
error: 1f7f0414aff1105f8afec1d52a8722cad1fe8194: object corrupt or missing: ./objects/1f/7f0414aff1105f8afec1d52a8722cad1fe8194
error: object file ./objects/6e/2fad3ad5c4940ae59169f897fad482eeb16ae7 is empty
error: unable to mmap ./objects/6e/2fad3ad5c4940ae59169f897fad482eeb16ae7: No such file or directory
error: 6e2fad3ad5c4940ae59169f897fad482eeb16ae7: object corrupt or missing: ./objects/6e/2fad3ad5c4940ae59169f897fad482eeb16ae7
error: object file ./objects/6f/3b6bb149f990be8bf8b35ce488cdb6427ee10f is empty
error: unable to mmap ./objects/6f/3b6bb149f990be8bf8b35ce488cdb6427ee10f: No such file or directory
error: 6f3b6bb149f990be8bf8b35ce488cdb6427ee10f: object corrupt or missing: ./objects/6f/3b6bb149f990be8bf8b35ce488cdb6427ee10f
error: object file ./objects/80/9c9d913aa03948f57d00c9b586ae15950f518e is empty
error: unable to mmap ./objects/80/9c9d913aa03948f57d00c9b586ae15950f518e: No such file or directory
error: 809c9d913aa03948f57d00c9b586ae15950f518e: object corrupt or missing: ./objects/80/9c9d913aa03948f57d00c9b586ae15950f518e
error: object file ./objects/90/f82bd8fcec71b497d13c3bd90dfba8f479650b is empty
error: unable to mmap ./objects/90/f82bd8fcec71b497d13c3bd90dfba8f479650b: No such file or directory
error: 90f82bd8fcec71b497d13c3bd90dfba8f479650b: object corrupt or missing: ./objects/90/f82bd8fcec71b497d13c3bd90dfba8f479650b
error: object file ./objects/b7/dcc58478862808214b58e84637e6e63acc58e9 is empty
error: unable to mmap ./objects/b7/dcc58478862808214b58e84637e6e63acc58e9: No such file or directory
error: b7dcc58478862808214b58e84637e6e63acc58e9: object corrupt or missing: ./objects/b7/dcc58478862808214b58e84637e6e63acc58e9
error: object file ./objects/da/e06c9ba81bb3fdebe481ed23cd8b26d9e57b9c is empty
error: unable to mmap ./objects/da/e06c9ba81bb3fdebe481ed23cd8b26d9e57b9c: No such file or directory
error: dae06c9ba81bb3fdebe481ed23cd8b26d9e57b9c: object corrupt or missing: ./objects/da/e06c9ba81bb3fdebe481ed23cd8b26d9e57b9c
error: object file ./objects/fa/78e944147fbf4506a4c141efe47c27655fffa9 is empty
error: unable to mmap ./objects/fa/78e944147fbf4506a4c141efe47c27655fffa9: No such file or directory
error: fa78e944147fbf4506a4c141efe47c27655fffa9: object corrupt or missing: ./objects/fa/78e944147fbf4506a4c141efe47c27655fffa9
Checking object directories: 100% (256/256), done.
error: object file ./objects/80/9c9d913aa03948f57d00c9b586ae15950f518e is empty
error: object file ./objects/80/9c9d913aa03948f57d00c9b586ae15950f518e is empty
fatal: loose object 809c9d913aa03948f57d00c9b586ae15950f518e (stored in ./objects/80/9c9d913aa03948f57d00c9b586ae15950f518e) is corrupt

@kevin, do we have any backups to restore the repository from, please?

no.

There might be a snapshot, but not really sure how to access it or if it would be worth all the trouble...

Possibly try git fsck, or so. Or simply remove the git repository (and let copr-dist-git re-create the repo), at least if kicad maintainer isn't against such solution. We don't promise dist-git persistence AFAIK, it's mostly proxy tool helping us with package builds at this point.

Possibly try git fsck, or so. Or simply remove the git repository (and let copr-dist-git re-create the repo), at least if kicad maintainer isn't against such solution

I've tried fsck and it wasn't as simple as I've expected. However, we have discussed it with kicad maintainer yesterday and he was for exactly what you are suggesting

Or simply remove the git repository (and let copr-dist-git re-create the repo), at least if kicad maintainer isn't against such solution.

And it worked well, so this issue is solved.
Metadata
None
None
None
None
Lowest
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.