Add documentation/tutorial on how to read Tetragon metrics

[mtardy]

From a discussion on the Cilium Slack:

Question:

How can I see the Prometheus metrics generated by Tetragon?

Answer:

If you run Tetragon with the helm chart, it should be enabled, you can read cat /var/run/tetragon/tetragon-info.json for the info but typically it listens on 2112.

If you run Tetragon standalone you can use: sudo ./tetragon --bpf-lib ./bpf/objs/ --metrics-server localhost:2112 to start with the metrics server (it's disabled by default)

Then to read the the metrics you can do curl localhost:2112/metrics.

[YTGhost]

Hi, I am interested in this job. Can you assign it to me?

[prateek041]

Hello @mtardy

I see that the PR Linked up is not on par with what was expected. I believe YTGhost is still working on it. In the meanwhile, I would like to know what is expected outcome for solving this issue ? A little more description would be nice.

[YTGhost]

Hello @mtardy

I see that the PR Linked up is not on par with what was expected. I believe YTGhost is still working on it. In the meanwhile, I would like to know what is expected outcome for solving this issue ? A little more description would be nice.

Hi, @prateek041

I see that the PR Linked up is not on par with what was expected.

Yes, we need a more detailed tutorial with enough context. But I've been stuck in some other more urgent and busy things lately, so if you have time, you're more than welcome to do it!

[SumaiyaSafdar]

HELLO @mtardy if anyone's not working over the project I request you to assign it to me

[mtardy]

HELLO @mtardy if anyone's not working over the project I request you to assign it to me

I think prateek041 was explicitly asking to work on that very recently.

[prateek041]

@mtardy sorry for the 3 months delay ! I have been working on this issue. I can pick it up now, if it works ?

[mtardy]

@mtardy sorry for the 3 months delay ! I have been working on this issue. I can pick it up now, if it works ?

yes sure

[prateek041]

Ok, so would you mind briefing what the expected tutorial looks like ? what it should contain, what are some "good to have"(s), so I can kind of use it as an index to shape the entire tutorial.

@mtardy

[mtardy]

maybe @lambdanis can help here if she wants! :)

It was mostly, showing the flag to disable/enable Tetragon metrics, explaining what's the default for various installation, to get the JSON file where you have the port to fetch the metrics (in /var/run/tetragon file tetragon-info.json) and then show how to fetch it with curl. And then explain a few of the metrics.

[prateek041]

I am not super experienced and memory is a bit flaky but your message said this

If you run Tetragon with the helm chart, it should be enabled, you can read `cat /var/run/tetragon/tetragon-info.json` for the info but typically it listens on `2112`.

If you run Tetragon standalone you can use: `sudo ./tetragon --bpf-lib ./bpf/objs/ --metrics-server localhost:2112` to start with the metrics server (it's disabled by default)

Then to read the metrics you can do `curl localhost:2112/metrics`.

are these the only ways ?

also, I would greatly appreciate @lambdanis suggestions.

[mtardy]

indeed. the idea would be to just write a tutorial for all installations, standalone, container, Kubernetes. If you don't feel like it it's okay you can drop the issue. I think that would be a good first step with what I suggested, you can draft something.

[lambdanis]

👍 This sounds good.

I think the initial page can be simple - documenting how to enable the metrics server (via Helm and via agent flag), and what's the metrics endpoint.

There is also an in progress PR to add extra labels configuration: #1444 When it's merged we could extend the docs page.

A more advanced tutorial could help users get started with observability stack in Kubernetes: install kube-prometheus-stack, enable ServiceMonitor in Tetragon Helm chart, and maybe show an example Grafana dashboard. But for now starting simple will be good.

[prateek041]

Sure @lambdanis @mtardy let me write a draft.

[prateek041]

Raised a Draft for Kubernetes, working on Package and standalone. In the meantime, Please review if i'm doing it the right way..

cc @mtardy @lambdanis