#281 Adding Prometheus /metrics endpoint and some data
Closed by gnaponie. Opened by gnaponie.
gnaponie/greenwave monitoring  into  master

Download 281.patch

Adding the Prometheus endpoint /metrics so that when one day we will
have a Prometheus instance running for our services the metrics will
already be available.
Added:
* basic metrics automatically provided by Prometheus
* counter for exceptions in the waiverdb consumer (exception receiving
a new waiver)
* counter for exceptions in the resultsdb consumer (exception receiving
a new result)
* counter for exceptions in the decision API
* latency for the decision API

You can also add python3-prometheus_client requirement to spec file.

Will this work as expected with multiple gunicorn workers?

I'm actually not sure about it... what can be another the solution? Some more complicated structure? Something like Redis maybe?
That's for sure the common way I saw around Prometheus docs/tutorials...

Probably store the objects with memcached (though these need to be pickled).

Hm, the problem with respect to multiple gunicorn workers also applies to multiple pods in openshift.

I think we should expect to (eventually) have an instance of prometheus running in openshift that knows how to collect metrics from the pods directly, so we don't need to solve the multi-pod problem in greenwave itself, but we do (unfortunately) need to solve the multi-worker problem.

One option could be to store the counters "on disk" in the ephemeral pod. No need for a persistent volume claim... but the workers could each use a file locking mechanism to reliably update the counter.

This seems like a problem other people must have hit. Is there some generic way to solve it?

I found one link that may provide some clues:
https://github.com/prometheus/client_python/issues/30

Pull-Request has been closed by gnaponie

Metadata