fedora-infrastructure

#12872 flatpak-indexer is ENOSPC, not running
Closed: Fixed by yselkowitz. Opened by yselkowitz.

Closed: Fixed
Reopen Issue

Describe what you would like us to do:

The flatpak-indexer-redis-storage PVC is overfull, and the redis pod keeps failing with ENOSPC when saving snapshots, which then brings down the other pods. Without this running, flatpak updates are not made available to users, and there is a bunch of them right now in conjunction with F43 GA.

When do you need this to be done by? (YYYY/MM/DD)

ASAP

I grew the storage from 30G to 50G

Metadata Update from @kevin:
- Issue assigned to kevin
- Issue tagged with: low-trouble, medium-gain, ops

Why does this keep happening though? The database file itself isn't that large, and while there needs to be more space there for temporary files (before the main file is overwritten iiuc) it shouldn't keep growing like that. Are there volume snapshots happening that I can't see, and if so, are they truly necessary (or at least maybe not so many)?

Yes, there's a lot of snapshots:

ntap-rdu3-c02-fedora01 
         flatpak_indexer_redis_storage
                  snapmirror.fede3f22-fb5a-11ef-b799-d039eac1c452_2152509667.2025-07-01_111531 
                                                         60.80MB     0%    3%
                  weekly_sun_0230.2025-07-20_0230          476KB     0%    0%
                  weekly_sun_0230.2025-07-27_0230        856.6MB     2%   32%
                  weekly_sun_0230.2025-08-03_0230        875.6MB     2%   33%
                  weekly_sun_0230.2025-08-10_0230        901.8MB     2%   33%
                  weekly_sun_0230.2025-08-17_0230         1.12GB     2%   39%
                  weekly_sun_0230.2025-08-24_0230         1.07GB     2%   38%
                  weekly_sun_0230.2025-08-31_0230         1.02GB     2%   37%
                  weekly_sun_0230.2025-09-07_0230         1.04GB     2%   37%
                  weekly_sun_0230.2025-09-14_0230         1.05GB     2%   37%
                  weekly_sun_0230.2025-09-21_0230        963.9MB     2%   35%
                  weekly_sun_0230.2025-09-28_0230         1.09GB     2%   38%
                  weekly_sun_0230.2025-10-05_0230        899.6MB     2%   33%
                  weekly_sun_0230.2025-10-12_0230        898.1MB     2%   33%
                  daily_0230.2025-10-15_0230             951.6MB     2%   35%
                  daily_0230.2025-10-16_0230             941.8MB     2%   34%
                  daily_0230.2025-10-17_0230             982.7MB     2%   35%
                  daily_0230.2025-10-18_0230             962.4MB     2%   35%
                  weekly_sun_0230.2025-10-19_0230          180KB     0%    0%
                  daily_0230.2025-10-19_0230             936.0MB     2%   34%
                  daily_0230.2025-10-20_0230              1.05GB     2%   37%
                  daily_0230.2025-10-21_0230               340KB     0%    0%
                  daily_0230.2025-10-22_0230             898.4MB     2%   33%
                  daily_0230.2025-10-23_0230              1.15GB     2%   40%
                  daily_0230.2025-10-24_0230              1008MB     2%   36%
                  daily_0230.2025-10-25_0230             872.4MB     2%   33%
                  weekly_sun_0230.2025-10-26_0230          184KB     0%    0%
                  daily_0230.2025-10-26_0230              1.15GB     2%   40%
                  daily_0230.2025-10-27_0230             874.1MB     2%   33%
                  12hour_0630.2025-10-27_0630            881.4MB     2%   33%
                  12hour_0630.2025-10-27_1830            291.7MB     1%   14%
                  daily_0230.2025-10-28_0230             883.0MB     2%   33%
                  12hour_0630.2025-10-28_0630            888.3MB     2%   33%
                  snapmirror.46ae8357-e29c-11e4-b894-123478563412_2155595997.2025-10-28_175000 
                                                         956.3MB     2%   35%
                  12hour_0630.2025-10-28_1830             1.19GB     2%   40%
35 entries were displayed.

The snapshots are determined by the 'criticality' level of data by the storage folks. We don't have fine grained entries, so all our data is under this same level of snapshots normally. ;(

So, not easy to just change one volume.
I think it's easier to just make sure it's large enough to handle that.

Metadata Update from @kevin:
- Issue priority set to: Waiting on Assignee (was: Needs Review)

If you say so.

After rebuilding the indexer (the only way it seems I can kill an existing pod, it seems), the indexer is currently processing again. Let's see if it can get to the end without falling over again.

Metadata Update from @yselkowitz:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)
Metadata

low-trouble medium-gain ops

None
None
Normal
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.