-
Notifications
You must be signed in to change notification settings - Fork 169
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Help on Latency SLO definition #506
Comments
The normal approach is to consider any request that doesn't meet latency target as a straight up error. This would mean you should use For your
Still drawbacks I see with such an approach are
|
Hi, great job on Sloth!
I think it would be very helpful if there was a clearer explanation of how to setup latency SLOs.
Specific
le
bucketsThe first and easiest way I think that works is (using Istio in this case but there should be a similar metric) using specific
le
buckets:The downside of this method is that we are limited by the pre-defined
le
buckets and we need to tweak SLO objective too in order to get closer to our latency limits given thele
pre-defined buckets.Using
histogram_quantile
In this case, we have the option of setting up whatever latency we want but the complexity of the query is overwhelming for me and the results are not as good as the method '1'.
I noticed that in this case we need to define the SLO target both inside the errorQuery and on the
objective
field.Comparison
I also noticed that this methodology does not graph well, I have a comparison of the same latency SLO 98% for less than 1000ms
I am using the queries mentioned above, with totalQueries the 'istio_request_duration_milliseconds_count' one for the
le
method and without the> 1000
on the histogram one. Naturally the 'histogram_quantile' is the one whose name is suffixed with-hist
.I am also want to show how the
histogram_quantile
method shows for 1200ms and 99% this time. It looks very spiky, it's like either 100% or dipping low (the actual latency indeed spikes, but I dont understand the 100%). Thele
method does not show 100% flat.Question
So can you please explain which is the right method and what maybe i could improve on my approach? I did spend quite a lot of time on this but still my statistics fail me.
One more "bonus" question: Why do we declare the 'ticket' alerts on 1day and 3days but both with "same" burn rate ?? I mean burn rate 3 for 1 day is the same budget consumption rate as burn rate 1 for 3 days, right? Is it possible to violate the burn rate 3 in one day and not violate the burn rate 1 for 3 days? Isn't this redundant and waste of recording rules (I see there's a significant toll on prometheus)?
The text was updated successfully, but these errors were encountered: