-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Vector high memory utilization #20643
Comments
Can you share the configuration? |
Hi @jszwedko. I can't share the whole repo, but if there particular parts you're interested in, we can share those. |
Memory use is very workload and configuration dependent so this is likely to be difficult to track down without the configuration and at least a description of the workload (with as many specifics as possible). If you want, you could try to engage in some memory profiling using valgrind or a similar tool. |
That's fair. It'll probably be tricky to provide description of details of the pipeline, we have 100s VRLs. But on high level, it processes all of our telemetry before delivering it to Datadog. You might recall our pipeline. We've met on a video call last year. I work for a gaming company from U.S. We wrote the Vector We'll look into memory profiling and try to get some more details. Thanks Jesse |
A note for the community
Problem
Hi, we have one of the environments behaving strange. After each deployment on that env, memory usage of Vector spikes to dangerous values. It's a preprod environment with the same configuration as prod environments, but much lower traffic per Vector container. Some numbers:
Common for all envs:
Prod:
Preprod:
Below is an image of memory utilization of one container after deploy
My guess about what could be possibly happening is that Vector struggles with quick scaling of a single transformation. In our case vector has a lot of transformations that are specific to a single service. And preprod takes traffic from a small set of services. When looking at utilization of transformations after deploy, we see spikes but it looks relatively healthy and nothing else is different on that env, so I'm probably wrong.
Utilization of each transformation
Happy to take any suggestions what might be the issue hee
Configuration
No response
Version
vector 0.38.0 (aarch64-unknown-linux-musl ea0ec6f 2024-05-07 14:34:39.794027186)
Debug Output
No response
Example Data
No response
Additional Context
No response
References
No response
The text was updated successfully, but these errors were encountered: