#17 Improved Fedora Metrics: refactor Unique Daily IPs metrics backend scripts (plus improvements / tech-debt paydown on DNF Countme)
Closed 7 months ago by amoloney. Opened 2 years ago by mattdm.

New initiative: improve counting metrics backend scripts

What is this initiative about?

  1. Improve intermediate format for DNF Countme statistics gathering to be more robust and consume less disk space.
  2. Develop a similar system for the "old" statistics (unique daily IPs), and run that weekly as well.
  3. Finally, shut down the current "old" statistics system (which is fragile and only known to Smooge).

See:

Why this initiative?

  1. Old IP/day-based mirror metrics continue to be useful even though we have the new system
    • coverage for still-running old Fedora Linux releases, plus EPEL 5 (!), 6, and 7.
    • it turns out to be useful to correlate for current releases
  2. But (A) the old scripts are finicky and only @smooge really knows how to feed and water them
    • So the script needs to be updated and replaced
  3. Plus (B) they need maintenance every time a new Fedora Linux or EL release happens
    • Again, basically only Smooge can do it
  4. If we write a new script, it can generate output in a format similar to countme, which would solve B and presumably also A.
  5. Smooge says that the current DNF Countme system is running into an inevitable disk space usage problem even if we don't add something else.
    * So we need to something!
    * And reworking the intermediate format should solve that.

Definition of success

What is the minimal outcome you would like to see from this initiative to be satisfied?

  1. Not running out of disk space
  2. Not dependent on Smooge, because the current setup is not fair to him.
  3. Data for IP/day (without actual IPs, mind you!) alongside countme data in "long" form (file name suggestion dailyips.db or dailyips-week####.db if using the incremental approach)
  4. IP/day scripts don't need annual and per-release (or even more frequent!) maintenance (no more need to add new columns, due to using long form)

What are your nice or really nice to have wishes?

  • Suggested intermediate format is just one idea. It wouldn't need to be done that way as long as it resolves the above.
  • also as suggested there, incremental final files countme-week####.db instead of one totals.db
  • The last log info for a week (ending Sunday) comes in Monday a few minutes after midnight UTC. Right now, there's a lag until each entire Monday log gets to the processing system, which means that the data is crunched and available only Thursday morning. It would be nice to have this actually be Monday or Tuesday morning. This, however, requires restructuring how the raw logs are handled, and that's big and risky.

Area/community impacted

Fedora, EPEL, CentOS Stream

Dependencies

Do this initiative have any dependencies?

  • Not really, but it's worth noting that the current system should be left running while this is developed. That way, we'll have continuity while the work happens, and also have a dataset to validate against. (The end result should be nearly identical for the DNF Countme data, and have reasonable correlation for the Unique Daily IPs count.)

Skills needed?

  • Python programming. Familiarity with sqlite a plus but it's pretty easy.

Person who must or should be involved?

  • It'd be both be nice for Smooge to be able to consult, and probably also a relief for him not to. :)

Other work that should be completed prior to this initiative?

  • Nope

Deadline

Is this initiative under a time-constraint? Should it start or end before a certain date?

  • Only the disk issue is a hard constraint.

Note: I've made some edits to the original post for clarity.

Issue tagged with: Accepted

2 years ago

@smooge is adding himself to be reminded when this gets dealt with.

This initiative has now been scheduled for an investigation spike and is estimated to begin on April 20th 2022.
The investigation team will look at possible technical solutions to deliver this project to the success criteria specified in the ticket and recommend a particular course of action.
Once a technical solution has been proposed, the ticket will be marked as Ready and picked up as the next project to be actioned.

Cool. It's probably useful for me to have a high-bandwidth meeting with the investigation team, because I have thoughts. :)

Update: as I understand it, the team has addressed definition of success item 1 (the disk space issue) but not by refactoring as outlined https://pagure.io/mirrors-countme/issue/56. (As noted, that's okay — I really only care about the result. But I do think what I suggest has some ongoing benefits.)

The rest:

  • Not dependent on Smooge, because the current setup is not fair to him.
  • Data for IP/day (without actual IPs, mind you!) alongside countme data in "long" form
  • IP/day scripts don't need annual and per-release (or even more frequent!) maintenance

is still pending. Correct?

That is correct. We plan to action this work once communishift wraps up,
pending no other higher priority work request being submitted.

On Fri, Aug 5, 2022 at 2:38 PM Matthew Miller pagure@pagure.io wrote:

mattdm added a new comment to an issue you are following:
` Update: as I understand it, the team has addressed definition of success item1` (the disk space issue) but not by refactoring as outlined
https://pagure.io/mirrors-countme/issue/56. (As noted, that's okay =E2=80=
=94 I
really only care about the result. But I do think what I suggest has some
ongoing benefits.)

The rest:

  • Not dependent on Smooge, because the current setup is not fair to hi=
    m.
  • Data for IP/day (without actual IPs, mind you!) alongside countme
    data in "long" form
  • IP/day scripts don't need annual and per-release (or even more
    frequent!) maintenance

is still pending. Correct?

``

To reply, visit the link below or just reply to this email
https://pagure.io/cpe/initiatives-proposal/issue/17

--=20

Aoife Moloney

Product Owner

Community Platform Engineering Team

Red Hat EMEA https://www.redhat.com

Communications House

Cork Road

Waterford
https://www.redhat.com

An ARC (advance recon crew - small team who pre-scopes work for a request before a dev team takes it on) investigation has been done for this request https://fedora-arc.readthedocs.io/en/latest/mirrors-countme/index.html

Its ready to be scheduled and currently 'next' for CPE team to take on. Current estimated start date is Monday April 3rd 2023.

This project is now in flight and tracked here https://github.com/orgs/fedora-infra/projects/16/views/1

Est completion date is end of June 2023.

Metadata Update from @amoloney:
- Issue tagged with: In Progress, On Track

10 months ago

Metadata Update from @amoloney:
- Issue set to the milestone: Fedora Infra Enhancements

10 months ago

Login to comment on this ticket.

Metadata
Boards 1
2022 Status: Ready