Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(metrics): histogram metric for loop latency #5812

Draft
wants to merge 8 commits into
base: main
Choose a base branch
from

Conversation

Hy3n4
Copy link

@Hy3n4 Hy3n4 commented May 20, 2024

This PR is supposed to add new histogram type of metric for the internal loop latency. This should be more useful when creating the SLIs.

Checklist

Copy link

stale bot commented Jul 30, 2024

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale All issues that are marked as stale due to inactivity label Jul 30, 2024
@Hy3n4 Hy3n4 force-pushed the feature/internal_loop_histogram_metrics branch from 99460e2 to 43e362a Compare August 6, 2024 08:55
@stale stale bot removed the stale All issues that are marked as stale due to inactivity label Aug 6, 2024
@Hy3n4
Copy link
Author

Hy3n4 commented Aug 6, 2024

Hey, I would like to revive this PR. I've found some time to test it out and it does exactly what I need from it to do. At least I think it does. I would appreciate another review and possibly the direction what else should I do to have this PR ready for merge 🙏 .

Copy link
Member

@JorTurFer JorTurFer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good, can you include the new metrics in the e2e tests?

@Hy3n4
Copy link
Author

Hy3n4 commented Aug 21, 2024

Sure, I'll try my best.

keda_internal_scale_loop_latency_seconds_bucket

Signed-off-by: Hy3n4 <[email protected]>
Copy link
Member

@JorTurFer JorTurFer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add the test coverage also to opentelemetry

Copy link
Member

@JorTurFer JorTurFer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good! Could you update the changelog with this improvement?

@JorTurFer
Copy link
Member

JorTurFer commented Aug 21, 2024

/run-e2e metric
Update: You can check the progress here

@JorTurFer
Copy link
Member

Could you update docs as well to include this metric?

@Hy3n4
Copy link
Author

Hy3n4 commented Aug 21, 2024

It's done. Although I'm not sure I chose the right place for it 😄

@JorTurFer
Copy link
Member

It's done. Although I'm not sure I chose the right place for it 😄

Could you send me the link? I can't find it :(

@Hy3n4
Copy link
Author

Hy3n4 commented Aug 21, 2024

It's done. Although I'm not sure I chose the right place for it 😄

Could you send me the link? I can't find it :(

Never mind, I mixed up docs and changelog. So I changed the changelog. I'll revert it and edit the docs. Should I add it to version 2.16?

@@ -904,6 +904,37 @@ func testScalableObjectMetrics(t *testing.T) {
}
}
assert.Equal(t, true, found)

val, ok = family["keda_internal_scale_loop_latency_seconds_bucket"]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not 100% sure, but I think that the name is keda_internal_scale_loop_latency_bucket here
e2e test has failed here
image

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, I see the problem with otel. There is no otel metric for this. The problem is I don't have enough expertise to create an otel histogram metric 🤔 So we have two options. Remove e2e test or make the otel metric somehow.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you have to call the the record function instead of using an observable approach

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aha, now I can see in the code that I tried something, but apparently it's not the right way to do it.

Copy link

stale bot commented Oct 28, 2024

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale All issues that are marked as stale due to inactivity label Oct 28, 2024
@stale stale bot removed the stale All issues that are marked as stale due to inactivity label Oct 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants