Nginx exporter used in kubernetes pods

[Mehrandev]

Hello and thanks for this great exporter.
I've got a situation here that needs help , and i hope it will help some one else too.

I've currently used the exporter in a container along with nginx in another container both in a pod and the metrics looks nice when we have only one pod , but the problem starts when we are going to scale pods to 2-3 and more.

We have a domain lets say abc.com and in the path of /metrics we are serving the exporter container , so it shows the metrics nicely , but when the load balancer starts to change the traffic between pods , we are getting different metrics each time and this results in a wrong data(each time its a different pod and exporter data ) , because the prometheus and exporter doesn't have any information about the pod and only know the instance and its same instance in all pods , but each pod got their own nginx and nginx exporter with a metrics path for their own pod.

We are using a share mount between containers in a pod and using that we can have access to nginx logs from the exporter.

I wonder if you have any idea on how to solve the issue , any help is appreciated.
Thanks

[Mehrandev]

@martin-helmich any help is appreciated.

[martin-helmich]

So, just to clarify on your architecture, before commenting further:

1. You have your NGINX app running in a (potentially scaled) deployment
2. The NGINX exporter is running as a sidecar within these Pods (same Pod, different container probably?), so that when you scale your app to n replicas, there are also n exporters running
3. The accesslog file of each Pod is shared between the containers of that Pod (NGINX and exporter) via a volume, but each Pod writes into its own access log.

Did I get that right so far?

[Mehrandev]

Yes , thats right, the architecture looks exactly like you say .

The NGINX exporter is running as a sidecar within these Pods same Pod, different container.

[martin-helmich]

I'm assuming that you're scraping your metrics from the Service endpoint itself (or the ingress, which uses the Service as backend), which will result in your scraping requests to the /metrics route to hit different Pods (and subsequently different exporters) in a round-robin manner. Since each Pod has its own access log file, each exporter will export different metrics (each generated from its own access log shard).

I would suggest scraping individual Pods, instead of the entire service. That way, you will retrieve a consistent set of metrics for each Pod. This allows you to inspect performance metrics for individual Pods (and if you don't care about single Pods, you can still aggregate all metrics to give you an aggregated view on the entire service).

Depending on how you're deploying Prometheus, this might either be done...

1. by using a kubernetes_sd_configs: [{role: pod}] configuration in your Prometheus configuration, when you are deploying Prometheus manually
2. by using a PodMonitor instead of a ServiceMonitor if you're using the Prometheus Operator.

[Mehrandev]

Thanks for sharing your idea,
I would rather scrap each pod data separately , so i can have RPS and other metrics for each pod.
Also i can have total RPS too by aggregating all metrics.
i'm gonna try your options then .
Thanks

[Mehrandev]

@martin-helmich
I just need to ask something else about your solution.
As i understood you mean i should scrap each pod individually , but if i do that, this will give me details about the pod itself not the nginx exporter i'm using.
How can i achieve that to get data of each exporter in each pod individualy?

Thanks

[martin-helmich]

Not sure I understand. If you scrape the /metrics endpoint of each Pod (at the port the prometheus-nginxlog-exporter is listening on), it'll still be the Prometheus Exporter answering responding with the same metrics as always.

[Mehrandev]

Yes, thats right , thanks a lot