hckrnws

Ah, a shared solution to this problem is sorely needed in the Python ecosystem. When we developed the Quay[0] container registry, we had a bunch of gunicorn processes all with their own Prometheus client registries (the in-memory data structures keeping track of the metrics), but you can't just scrape this directly. We ended up having to adopt a push model and then even then it's tricky to make sure metrics are designed to aggregate properly.

[0]: https://github.com/quay/quay

In theory the push gateway is to be used for batch/one of jobs, in limited cases because it had drawbacks

But yeah that's what I'm trying to solve in a different way, registries in different processes are separated but with the "backend" architecture, if they are the same metric they will end up in the same place. Redis can be considered the shared memory for all the processes that together make a single service, when you scrape one of the processes, you get back the correct value. This is possible because redis is single-threaded & the float increment operations are atomic. I'm sure not everyone might want to use redis, so the library implements a "Backend Protocol" that you just have to respect the interface and so you can build a different solution with different toolings, it's pluggable! (This made it extremely easy for me to implement a different backend in rust that might also remove the dependency on the redis client in the python side :) )

Quick note that although I'm probably in the target audience (I have used and deployed Grafana using Prometheus and I'm generally interested in software performance), and I skimmed the readme, I don't actually know what a "Prometheus Client" is. Or perhaps I don't know what it's for. A block diagram of the whole monitoring system, with a "this thing goes here" label might be useful? Or a sentence like "This would be useful to someone building an XXX"?

I've probably cut the wrong word to make the title more concise, it's a prometheus client library. (https://prometheus.io/docs/instrumenting/clientlibs/)

This is the library to use in your python code to expose metrics so that Prometheus can scrape them. The diagram would definitely be useful, it started as something that "you already knew if you need it" but I noticed that there is more interest from people with no experience with these system, I was considering adding more information or making small guides/tutorial to ease into the why you need the monitoring, even for small applications since the price is small. Thanks!

The terminology is tricky here. Client implies it's using Prometheus, but as far as I can tell it's not. Calling it a Prometheus "server" then makes it sound like the Prometheus part.

I believe these are typically called "exporters". They implement servers that Prometheus scrapes. The quote "if you had issues while using multiprocess in python when collecting metrics this might interest you" suggests this is what this is. I've used [1] in the past, including for collecting across multiple processes with multiprocessing, and didn't really have any problems beyond my usual issues with Prometheus.

[1]: github.com/prometheus/client_python

The correct naming would be "prometheus client library" as per prometheus docs.

If you didn't have issues with multiprocessing that's great, but it seems like it's not uncommon for people to have issues with that approach, for example (the reason I started to experiment on this as a sideproject) in the k6 team at grafana we had the problem of too many files getting created and not properly getting cleaned up, we were forced to go into single process mode and with kubernetes that's fine. But if you are not running in kubernetes, I'm not sure you had a clean alternative :)

We ditched Prometheus in favour of Statsd before moving to Kubernetes. We had regular enough releases that we were losing significant data due to the scrape interval and felt a push model worked better for us. The push gateway had too many caveats and when we used it we felt we couldn’t trust the data coming from it.

This is so classic software development. "Here is a super simple readme that tells you everything you need to know (PS we expect you to already know everything.)

Well, I never claimed that it was a super simple readme that explained everything you need to know :)

The target was clear, people that had issues with multiprocessing with the current ecosystem.

But as I noticed the interest growing, I also noticed a lack of simple guides on how to go from zero to have graphs in grafana for your metrics. That's why I started the "tutorial" section in the docs (pythe.us) initially for FastAPI to try to fill that void. I'm also considering making a full video tutorial to complement it.

So to recap, here I'm sharing the existence of this project so I can hopefully gather some feedback for the future direction. For beginners right now there is the "Quickstart" section in the documentation that guides you through monitoring latency in a flask application. It does not cover how to setup prometheus yet but I'm sure the prometheus docs can help with that for now :)

I see that this uses a Redis backend. Does that mean every metric operation goes to network? Are those synchronous/blocking calls, and are they pipelined with redis? If not, it seems like this could be quite detrimental to app response times if it does many metric ops when handling a single request.

The other thing I noticed is the redis backend has a 1 hour TTL. Does that exist to clean up once a metric is no longer getting touched (presumably because a worker/pod is gone)?

Correct, for multiprocessing support it uses Redis, the idea is that you have a sidecar with your service that will sync metrics between all of them for scrapes.

The default are blocking operations and they are fast, they are pipelined for scrapes (retrieving all the metrics value is the slowest part sped-up with this).

If you are curious there are some benchmarks tests that I used to make sure the library was correct comparing with the official one, in my tests this approacher is faster than the mmap files one but they are testing a really limited scenario so take it with a grain of salt: https://github.com/Llandy3d/pytheus-bench#the-results-

Correct for the TTL, if a metric doesn't get scraped/incremented for more than ttl, it will be cleaned up. 1 hour is the current default value.

To go a little further into details, the Rust based backend has a separate thread for changing metrics value, so on your python code, wether sync or async, the operation is extremely fast. On the Rust side, the operations are collected and pipelined together asynchronously. From the tests I've made this is enough, but if someone has an insane amount of metrics to modify, it is possible to add support for multiple "writer" threads, the important bit is that operations for a single metric are done in order, but this can be easily achieved by hashing with the number of threads. I hope this answer your questions!