This can be done using the google cloud platform console.
But it needs billing so as an organization we can do it but not as an individual.

We could possibly get AWS resources from Fedora Infrastructure. Or, we could use fedorapeople.org for storage rather than a cloud service.

I don't have aws credits and I don't find some documentation for pushing the files in the storage of fedorapeople.org. Is there any way to find a solution?

I am curious if it is possible to use GrimoireELK tool to help with this work. It seems like a lot of effort for Fedora to write our own software to handle caching when there is a larger upstream project doing similar work.

Is it possible to use a project like GrimoireELK in this context, has anyone looked into it?

@jflory7 I am researching about Grimoire Labs stack and I mentioned here a first approach to ideas of working over this stack :blush:

Documentation on the Fedora People space is here: https://fedoraproject.org/wiki/Infrastructure/fedorapeople.org

Justin, I'm not opposed to using GrimoireELK, but... really, Datanommer is already a database of cached messages. Having another type of database running doesn't solve the basic problem of making queries for a lot of individual messages going back for many years. Unless each person working on this is running their own copy of that locally, I think they'll have the same basic problem except we'll just be asking how to distribute a pre-filled GrimoireELK database.

(For that matter, another approach would be for people to run datagrepper and datanommer locally, and distribute a "seed" database dump of datanommer's postgresql store.)

@mattdm I'm sorry if I misunderstood, but why people would run datagrepper and datanommer locally?

@josseline Well, my program calls those things and then caches the result locally to avoid needing to hit the remote service so many times. If you were running a copy of that remote service locally, presumably it wouldn't be such an issue. But I think that would be a lot of extra work.

Yes, I know right now the project gets data from datagrepper to process the data and I'm aware that keep a replica of the datasources is a better choice to handle data more efficiently.

Do you suggest keep a datanommer replica in a relational database and then make a custom dashboard app? :blush:

Datanommer is a store of message bus messages in a relational database. (And datagrepper is a web api to make certain queries of that database.)

Actually, one possibility might be to ask CPE if we can have direct access to that database for a web dashboard. For reasons I'm not quite clear on (performance or security, or something else?) the rule is that everything has to go through datagrepper.

The existing code here is not pretty, but it has the advantage of needing no special access.

The ticket #9981 on Fedora Infrastructure is the request for resources for this project.

CC @mattdm