New initiative: improve counting metrics backend scripts

What is this initiative about?

1. Improve intermediate format for DNF Countme statistics gathering to be more robust and consume less disk space.
2. Develop a similar system for the "old" statistics (unique daily IPs), and run that weekly as well.
3. Finally, shut down the current "old" statistics system (which is fragile and only known to Smooge).

See:

• https://pagure.io/mirrors-countme/issue/56
• https://pagure.io/fedora-infrastructure/issue/10443

Why this initiative?

1. Old IP/day-based mirror metrics continue to be useful even though we have the new system
   • coverage for still-running old Fedora Linux releases, plus EPEL 5 (!), 6, and 7.
   • it turns out to be useful to correlate for current releases
2. But (A) the old scripts are finicky and only @smooge really knows how to feed and water them
   • So the script needs to be updated and replaced
3. Plus (B) they need maintenance every time a new Fedora Linux or EL release happens
   • Again, basically only Smooge can do it
4. If we write a new script, it can generate output in a format similar to countme, which would solve B and presumably also A.
5. Smooge says that the current DNF Countme system is running into an inevitable disk space usage problem even if we don't add something else.
   * So we need to something!
   * And reworking the intermediate format should solve that.

Definition of success

What is the minimal outcome you would like to see from this initiative to be satisfied?

1. Not running out of disk space
2. Not dependent on Smooge, because the current setup is not fair to him.
3. Data for IP/day (without actual IPs, mind you!) alongside countme data in "long" form (file name suggestion dailyips.db or dailyips-week####.db if using the incremental approach)
4. IP/day scripts don't need annual and per-release (or even more frequent!) maintenance (no more need to add new columns, due to using long form)

What are your nice or really nice to have wishes?

• Suggested intermediate format is just one idea. It wouldn't need to be done that way as long as it resolves the above.
• also as suggested there, incremental final files countme-week####.db instead of one totals.db
• The last log info for a week (ending Sunday) comes in Monday a few minutes after midnight UTC. Right now, there's a lag until each entire Monday log gets to the processing system, which means that the data is crunched and available only Thursday morning. It would be nice to have this actually be Monday or Tuesday morning. This, however, requires restructuring how the raw logs are handled, and that's big and risky.

Area/community impacted

Fedora, EPEL, CentOS Stream

Dependencies

Do this initiative have any dependencies?

• Not really, but it's worth noting that the current system should be left running while this is developed. That way, we'll have continuity while the work happens, and also have a dataset to validate against. (The end result should be nearly identical for the DNF Countme data, and have reasonable correlation for the Unique Daily IPs count.)

Skills needed?

• Python programming. Familiarity with sqlite a plus but it's pretty easy.

Person who must or should be involved?

• It'd be both be nice for Smooge to be able to consult, and probably also a relief for him not to. :)

Other work that should be completed prior to this initiative?

• Nope

Deadline

Is this initiative under a time-constraint? Should it start or end before a certain date?

• Only the disk issue is a hard constraint.