Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: adopt Grafana for metrics visualization (#2364) #2378

Merged
merged 4 commits into from
Aug 15, 2024

Conversation

jonathanj-square
Copy link
Contributor

fixes: #2364

Introduces a Grafana / OTEL container for capturing, ephemerally storing, and visualizing OTEL metrics emitted by FTL. The collection frequency is also increased to once per second (from a default once per minute)

Screenshot 2024-08-14 at 7 58 39 PM

@jonathanj-square jonathanj-square requested review from a team and worstell and removed request for a team August 15, 2024 03:21
@ftl-robot ftl-robot mentioned this pull request Aug 15, 2024
.env Outdated
@@ -0,0 +1,2 @@
OTEL_GRPC_PORT=4317
OTEL_HTTP_PORT=4318
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use Hermit for managing these envars instead:

hermit env OTEL_GRPC_PORT 4317
hermit env OTEL_HTTP_PORT 4318

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks updated; didn't know of this option

grafana:
image: grafana/otel-lgtm
platform: linux/amd64
restart: unless-stopped
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add the infra profile here so it doesn't start by default. You'll also need to add that to the docker compose lines that bring it up/down.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Justfile Outdated
-p 55679:55679 \
otel/opentelemetry-collector:0.104.0 2>&1 | sed 's/\([A-Z].* #\)/\
\1/g'

# Stop the docker container running otel.
otelContainerID := `docker ps -f ancestor=otel/opentelemetry-collector:0.104.0 | tail -1 | cut -d " " -f1`
otel-container-id := `docker ps -f ancestor=otel/opentelemetry-collector:0.104.0 | tail -1 | cut -d " " -f1`
grafana-container-id := `docker ps -f ancestor=grafana/otel-lgtm | tail -1 | cut -d " " -f1`
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unless this is doing something I'm not aware of, couldn't we just use docker compose down instead of this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

@jonathanj-square jonathanj-square added this pull request to the merge queue Aug 15, 2024
@alecthomas alecthomas removed this pull request from the merge queue due to a manual request Aug 15, 2024
@alecthomas
Copy link
Collaborator

@jonathanj-square I pulled it out of the queue because there are a couple of changes that would be nice to make

@jonathanj-square jonathanj-square changed the title feat: adopt Grafana for metrics visualization feat: adopt Grafana for metrics visualization (#2364) Aug 15, 2024
@jonathanj-square jonathanj-square added this pull request to the merge queue Aug 15, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Aug 15, 2024
@jonathanj-square jonathanj-square added this pull request to the merge queue Aug 15, 2024
Merged via the queue into main with commit 7029173 Aug 15, 2024
17 checks passed
@jonathanj-square jonathanj-square deleted the jonathanj/grafana branch August 15, 2024 17:19
safeer pushed a commit that referenced this pull request Aug 19, 2024
fixes: #2364 

Introduces a Grafana / OTEL container for capturing, ephemerally
storing, and visualizing OTEL metrics emitted by FTL. The collection
frequency is also increased to once per second (from a default once per
minute)

<img width="1246" alt="Screenshot 2024-08-14 at 7 58 39 PM"
src="https://github.com/user-attachments/assets/bd21cd33-dc93-4a79-9b2a-ad124f2db296">
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Improve otel-stream usability for local metrics troubleshooting
2 participants