Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Upgrade to google.golang.org/grpc v1.66.2 #9401

Draft
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

aknuds1
Copy link
Contributor

@aknuds1 aknuds1 commented Sep 25, 2024

What this PR does

Upgrade google.golang.org/grpc from v1.65.0 to v1.66.2. The performance regression referenced from dskit's reversal to v1.65.0 is fixed in v1.66.2, and the potential functional regression also referenced in said reversal is Loki specific.

What we found out is that v1.66 fixes a serious performance bottleneck in conjunction with compression; the decompress function is optimized through memory pooling, whereas v1.65.0 would use io.ReadAll and cause a lot of allocations during decompression. As a result, we saw ingester CPU usage increase by ~40% when enabling gRPC compression. The increase is now much more modest, maybe ~7% based on observations.

Also upgrade github.com/googleapis/enterprise-certificate-proxy to v0.3.4, to fix a checksum issue in v0.3.3 (v0.3.4 was released to take care of this). With v0.3.3, building on OS X fails:

✗ go mod tidy && go mod vendor
go: downloading github.com/googleapis/enterprise-certificate-proxy v0.3.3
verifying github.com/googleapis/[email protected]: checksum mismatch
	downloaded: h1:G6q7VHBoU74wQHXFsZSLMPl0rFw0ZDrlZ3rt6/aTBII=
	go.sum:     h1:QRje2j5GZimBzlbhGA2V2QlGNgL8G6e+wGo/+/2bWI0=

SECURITY ERROR
This download does NOT match an earlier download recorded in go.sum.
The bits may have been replaced on the origin server, or an attacker may
have intercepted the download attempt.

Which issue(s) this PR fixes or relates to

Checklist

  • Tests updated.
  • Documentation added.
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
  • about-versioning.md updated with experimental features.

@aknuds1 aknuds1 marked this pull request as ready for review September 25, 2024 07:54
@aknuds1 aknuds1 requested review from stevesg, grafanabot and a team as code owners September 25, 2024 07:54
@aknuds1 aknuds1 marked this pull request as draft September 25, 2024 07:54
@aknuds1 aknuds1 marked this pull request as ready for review September 25, 2024 07:56
@aknuds1
Copy link
Contributor Author

aknuds1 commented Sep 25, 2024

Some tests are breaking for this upgrade, have to fix them.

@aknuds1 aknuds1 marked this pull request as draft September 25, 2024 08:17
@aknuds1 aknuds1 changed the title Upgrade to google.golang.org/grpc v1.66.2 WIP: Upgrade to google.golang.org/grpc v1.66.2 Sep 25, 2024
@aknuds1
Copy link
Contributor Author

aknuds1 commented Sep 25, 2024

Tests currently pass, but that is through undoing some memory unsafe optimizations in the mimirpb package. We want to investigate whether the optimizations can safely be retained.

@aknuds1 aknuds1 force-pushed the arve/upgrade-grpc branch 7 times, most recently from 54d501e to d19fc2f Compare September 30, 2024 07:34
Also upgrade github.com/googleapis/enterprise-certificate-proxy to v0.3.4, to fix a checksum issue in v0.3.3.

Signed-off-by: Arve Knudsen <[email protected]>
Signed-off-by: Arve Knudsen <[email protected]>
Signed-off-by: Arve Knudsen <[email protected]>
Signed-off-by: Arve Knudsen <[email protected]>
Signed-off-by: Arve Knudsen <[email protected]>
@aknuds1 aknuds1 force-pushed the arve/upgrade-grpc branch 4 times, most recently from 1180cb6 to 554a41b Compare September 30, 2024 09:08
@aknuds1 aknuds1 force-pushed the arve/upgrade-grpc branch 4 times, most recently from 37eae97 to d79ea9f Compare September 30, 2024 11:30
Signed-off-by: Arve Knudsen <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant