Adds support for global circuit budget #6013

davinci26 · 2023-12-12T23:07:10Z

Solves projectcontour#6001 Signed-off-by: Sotiris Nanopoulos <[email protected]>

codecov · 2023-12-12T23:10:25Z

Codecov Report

Attention: 1 lines in your changes are missing coverage. Please review.

Comparison is base (a579ec3) 78.70% compared to head (e4a5cbb) 78.77%.
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6013      +/-   ##
==========================================
+ Coverage   78.70%   78.77%   +0.07%     
==========================================
  Files         138      138              
  Lines       19717    19745      +28     
==========================================
+ Hits        15518    15555      +37     
+ Misses       3895     3887       -8     
+ Partials      304      303       -1

Files	Coverage Δ
cmd/contour/servecontext.go	`83.88% <100.00%> (+0.03%)`	⬆️
internal/dag/gatewayapi_processor.go	`93.74% <100.00%> (+<0.01%)`	⬆️
internal/dag/httpproxy_processor.go	`91.94% <100.00%> (+0.01%)`	⬆️
internal/dag/ingress_processor.go	`97.61% <100.00%> (+0.01%)`	⬆️
internal/dag/policy.go	`95.65% <100.00%> (+0.12%)`	⬆️
pkg/config/parameters.go	`86.39% <ø> (ø)`
cmd/contour/serve.go	`20.77% <75.00%> (+1.27%)`	⬆️

Signed-off-by: Sotiris Nanopoulos <[email protected]>

davinci26 · 2023-12-12T23:13:11Z

removing @clayton-gonsalves since he is on PAT leave (if you see this feel free to review anyway).

Adding @shadialtarsha and @seth-epps as well

apis/projectcontour/v1alpha1/contourconfig.go

Signed-off-by: Sotiris Nanopoulos <[email protected]>

shadialtarsha

I think the code makes sense to me.
I have 2 questions:

What will happen if not all GlobalCircuitBreakerDefaults attributes are defined? I think they will have the value 0 and we will default to Envoy settings again which makes sense but I would clarify this in the doc.
Wondering if we can write e2e tests for this feature. I know it might be tricky.

cmd/contour/servecontext.go

skriss

Thanks for the PR @davinci26! Overall this makes sense to me, added some comments.

apis/projectcontour/v1alpha1/contourconfig.go

pkg/config/parameters.go

cmd/contour/serve.go

skriss · 2023-12-13T22:46:55Z

internal/metrics/metrics.go

@@ -87,6 +88,8 @@ const (
 	statusUpdateConflict        = "contour_status_update_conflict_total"
 	statusUpdateNoop            = "contour_status_update_noop_total"
 	statusUpdateDurationSeconds = "contour_status_update_duration_seconds"
+
+	ContourCircuitBreakerSettings = "contour_dag_circuit_breaker_settings"


Could you explain why you want to expose this as a metric? We don't really have any other metrics like this, so it's a little unusual.

Could you explain why you want to expose this as a metric?

We want to alert and build graphs on this as a defence in depth. We have discovered during a deployment that we accidentally deleted those annotations and that caused issues and that was really painful to debug.

Basically it helps with the discoverability of what people are setting across the cluster.

Hm, I would expect something like this to be exposed via resource Status or something similar rather than a Contour xDS server metric

Also I believe we have an issue/PR to expand on cluster circuit breaker metrics: https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_stats#circuit-breakers-statistics (here: #5950)

I don't want to hold up the rest of the change, so maybe lets pull the metrics part out into another PR?

Hm, I would expect something like this to be exposed via resource Status or something similar rather than a Contour xDS server metric

Personally I take a more liberal approach to metrics in the sense that, if contour makes the decision to set a specific value it would be useful for operators to know the decision at runtime, especially if it affect things

I don't want to hold up the rest of the change, so maybe lets pull the metrics part out into another PR?

I won't die on this hill :D, So I can separate the changes into two PRs. But my general take is that adding the metric is pretty "cheap" and overall useful. I.e. there have been cases where I wish I had this metric tracked.

Signed-off-by: Sotiris Nanopoulos <[email protected]>

apis/projectcontour/v1alpha1/contourconfig.go

internal/dag/policy.go

Signed-off-by: Sotiris Nanopoulos <[email protected]>

davinci26 · 2023-12-18T19:32:55Z

@skriss this is ready for another review round

cmd/contour/serve.go

site/content/docs/main/configuration.md

changelogs/unreleased/6013-davinci26-small.md

Signed-off-by: Sotiris Nanopoulos <[email protected]>

davinci26 · 2023-12-18T21:34:40Z

@skriss unless there is an embarrassing bug somewhere this is ready

davinci26 · 2023-12-20T14:10:55Z

@skriss maybe we close/merge this today before the holiday break?

skriss · 2023-12-20T18:41:39Z

Looks like we got some merge conflicts, if you can resolve those I'll take another look

Signed-off-by: Sotiris Nanopoulos <[email protected]>

davinci26 · 2023-12-20T20:31:00Z

@skriss merge conflicts resolved

skriss

This LGTM, I'm approving with comments that need to be addressed but can be done in a follow-up PR if preferred. Will leave for another maintainer to take a look as well. Thanks @davinci26!

internal/metrics/metrics.go

Signed-off-by: Sotiris Nanopoulos <[email protected]>

davinci26 · 2023-12-21T17:29:55Z

@projectcontour/maintainers can someone PTAL as well

sunjayBhatia

i think everything LGTM minus the metric part, I wonder if we can pull that out into a separate PR so we can discuss a little more?

Signed-off-by: Sotiris Nanopoulos <[email protected]>

davinci26 · 2023-12-21T22:14:25Z

i think everything LGTM minus the metric part, I wonder if we can pull that out into a separate PR so we can discuss a little more?

Will do it in a separate PR

davinci26 · 2023-12-21T22:17:01Z

there is more cleanup to be made

Signed-off-by: Sotiris Nanopoulos <[email protected]>

davinci26 · 2023-12-21T22:20:16Z

I think I got all the references now

internal/metrics/metrics.go

site/content/docs/v1.9.0/annotations.md

Signed-off-by: Sotiris Nanopoulos <[email protected]>

Adds support for global circuit budget

46629af

Solves projectcontour#6001 Signed-off-by: Sotiris Nanopoulos <[email protected]>

davinci26 requested a review from a team as a code owner December 12, 2023 23:07

davinci26 requested review from tsaarni and stevesloka and removed request for a team December 12, 2023 23:07

sunjayBhatia requested review from a team, skriss and clayton-gonsalves and removed request for a team December 12, 2023 23:07

release note

589633f

Signed-off-by: Sotiris Nanopoulos <[email protected]>

davinci26 removed the request for review from clayton-gonsalves December 12, 2023 23:12

davinci26 added the release-note/small A small change that needs one line of explanation in the release notes. label Dec 12, 2023

davinci26 commented Dec 12, 2023

View reviewed changes

apis/projectcontour/v1alpha1/contourconfig.go Show resolved Hide resolved

davinci26 added 2 commits December 12, 2023 18:25

adds test and a docstring

4237680

Signed-off-by: Sotiris Nanopoulos <[email protected]>

test metrics

c42a27a

Signed-off-by: Sotiris Nanopoulos <[email protected]>

shadialtarsha reviewed Dec 13, 2023

View reviewed changes

cmd/contour/servecontext.go Outdated Show resolved Hide resolved

skriss requested changes Dec 13, 2023

View reviewed changes

remove reddit parameters

6c7e736

Signed-off-by: Sotiris Nanopoulos <[email protected]>

seth-epps reviewed Dec 14, 2023

View reviewed changes

apis/projectcontour/v1alpha1/contourconfig.go Outdated Show resolved Hide resolved

internal/dag/policy.go Show resolved Hide resolved

davinci26 added 3 commits December 15, 2023 09:44

PR comments

bdf1de3

Signed-off-by: Sotiris Nanopoulos <[email protected]>

revert change

8e354c4

Signed-off-by: Sotiris Nanopoulos <[email protected]>

regenerate

337431c

Signed-off-by: Sotiris Nanopoulos <[email protected]>

davinci26 requested review from seth-epps, skriss and shadialtarsha December 15, 2023 15:00

davinci26 mentioned this pull request Dec 15, 2023

internal/dag: Add support for per-host max connections #6016

Merged

skriss requested changes Dec 18, 2023

View reviewed changes

cmd/contour/serve.go Show resolved Hide resolved

site/content/docs/main/configuration.md Outdated Show resolved Hide resolved

changelogs/unreleased/6013-davinci26-small.md Outdated Show resolved Hide resolved

changelogs/unreleased/6013-davinci26-small.md Outdated Show resolved Hide resolved

fixes pr comments

4151035

Signed-off-by: Sotiris Nanopoulos <[email protected]>

davinci26 added release-note/minor A minor change that needs about a paragraph of explanation in the release notes. and removed release-note/small A small change that needs one line of explanation in the release notes. labels Dec 18, 2023

davinci26 requested a review from skriss December 18, 2023 21:33

Merge remote-tracking branch 'upstream/main' into retryBudget

47df638

Signed-off-by: Sotiris Nanopoulos <[email protected]>

skriss approved these changes Dec 21, 2023

View reviewed changes

internal/metrics/metrics.go Outdated Show resolved Hide resolved

internal/metrics/metrics.go Outdated Show resolved Hide resolved

davinci26 added 2 commits December 21, 2023 10:51

update metrics docs

567d012

Signed-off-by: Sotiris Nanopoulos <[email protected]>

kick ci

4f4825b

Signed-off-by: Sotiris Nanopoulos <[email protected]>

sunjayBhatia reviewed Dec 21, 2023

View reviewed changes

revert metric change

509d1ac

Signed-off-by: Sotiris Nanopoulos <[email protected]>

davinci26 requested a review from sunjayBhatia December 21, 2023 22:13

remove metrics part 2

f9edfc7

Signed-off-by: Sotiris Nanopoulos <[email protected]>

sunjayBhatia reviewed Dec 21, 2023

View reviewed changes

internal/metrics/metrics.go Outdated Show resolved Hide resolved

sunjayBhatia reviewed Dec 21, 2023

View reviewed changes

site/content/docs/v1.9.0/annotations.md Outdated Show resolved Hide resolved

davinci26 added 2 commits December 24, 2023 15:09

remove metrics part 3

378cee4

Signed-off-by: Sotiris Nanopoulos <[email protected]>

ws change

5531f35

Signed-off-by: Sotiris Nanopoulos <[email protected]>

davinci26 requested a review from sunjayBhatia December 24, 2023 20:11

davinci26 added 2 commits December 25, 2023 10:35

kick ci

5a2ada7

Signed-off-by: Sotiris Nanopoulos <[email protected]>

kick ci 2

e4a5cbb

Signed-off-by: Sotiris Nanopoulos <[email protected]>

sunjayBhatia approved these changes Jan 2, 2024

View reviewed changes

sunjayBhatia merged commit b474d10 into projectcontour:main Jan 2, 2024
26 checks passed

skriss mentioned this pull request Jan 5, 2024

internal/featuretests: fix test merge conflicts #6057

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds support for global circuit budget #6013

Adds support for global circuit budget #6013

davinci26 commented Dec 12, 2023

codecov bot commented Dec 12, 2023 •

edited

Loading

davinci26 commented Dec 12, 2023

shadialtarsha left a comment

skriss left a comment

skriss Dec 13, 2023

davinci26 Dec 13, 2023

sunjayBhatia Dec 21, 2023

davinci26 Dec 21, 2023

davinci26 Dec 21, 2023

davinci26 commented Dec 18, 2023

davinci26 commented Dec 18, 2023

davinci26 commented Dec 20, 2023

skriss commented Dec 20, 2023

davinci26 commented Dec 20, 2023

skriss left a comment

davinci26 commented Dec 21, 2023

sunjayBhatia left a comment

davinci26 commented Dec 21, 2023

davinci26 commented Dec 21, 2023

davinci26 commented Dec 21, 2023

Adds support for global circuit budget #6013

Adds support for global circuit budget #6013

Conversation

davinci26 commented Dec 12, 2023

codecov bot commented Dec 12, 2023 • edited Loading

Codecov Report

davinci26 commented Dec 12, 2023

shadialtarsha left a comment

Choose a reason for hiding this comment

skriss left a comment

Choose a reason for hiding this comment

skriss Dec 13, 2023

Choose a reason for hiding this comment

davinci26 Dec 13, 2023

Choose a reason for hiding this comment

sunjayBhatia Dec 21, 2023

Choose a reason for hiding this comment

davinci26 Dec 21, 2023

Choose a reason for hiding this comment

davinci26 Dec 21, 2023

Choose a reason for hiding this comment

davinci26 commented Dec 18, 2023

davinci26 commented Dec 18, 2023

davinci26 commented Dec 20, 2023

skriss commented Dec 20, 2023

davinci26 commented Dec 20, 2023

skriss left a comment

Choose a reason for hiding this comment

davinci26 commented Dec 21, 2023

sunjayBhatia left a comment

Choose a reason for hiding this comment

davinci26 commented Dec 21, 2023

davinci26 commented Dec 21, 2023

davinci26 commented Dec 21, 2023

codecov bot commented Dec 12, 2023 •

edited

Loading