Vector sink invalid value for component_sent_bytes_total metric

[nmiculinic]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

I am analysing network throughput for the vector pods. When querying prometheus for:

rate(vector_component_sent_bytes_total{pod="<xxx>"}[1m])

I get rates of 8MB/s.

However when I see kubernetes pod network transmit bandwidth, the rate are much much smaller around ~150kB/s.

I'm using https://vector.dev/docs/reference/configuration/sinks/vector with compression enabled, thus as per:

component_sent_bytes_total
counter
The number of raw bytes sent by this component to destination sinks.

I'd assume this are raw bytes sent on the network interface. This appears to be true (or feasible) for the HTTP sink https://vector.dev/docs/reference/configuration/sinks/http/ but not for vector one. It seems as if the value are pre-compression, and not post-compression

Configuration

sinks:
  log:
    # This will be best-effort delivery, and it will not block any other sink
    type: vector
    acknowledgements:
      enabled: false
    inputs: <>
    buffer:
      when_full: "drop_newest"
      max_events: 5000
    batch:
      max_bytes: 4000111  # < default gRPC message limit
      max_events: 100
    address: <>
    compression: true
    request:
      concurrency: "adaptive"
      adaptive_concurrency:
        initial_concurrency: 5

Version

0.38.0

Debug Output

No response

Example Data

No response

Additional Context

No response

References

No response

[jszwedko]

Hi @nmiculinic !

Thanks for opening this! I think there is some understandable confusion here. component_sent_bytes_total is not meant to actually match the network bytes written to the wire: instead it is meant to be the serialized event bytes prior prior to compression. I think compression is likely causing the bulk of the difference you are seeing. There are some more details here: https://github.com/vectordotdev/vector/blob/master/docs/specs/component.md#componentbytessent

We would like to implement a new metric that indicates the network bytes (described here), but this is not yet implemented.

Does that help clear things up?

[nmiculinic]

@jszwedko yes it does, since I do observe the same for kafka sink as well.

I'm reading the spec, and:

For HTTP-based protocols, the total number of bytes in the HTTP body, as represented by the Content-Length header.

given vector is gRPC, which is HTTP-based protocol, shouldn't the metric be after compression, same for HTTP? Similarly for kafka, it's TCP based protocol, and I'd assume it'd be the total number of bytes placed on the socket excluding the delimiter, that is after compression?

For clickhouse sink (TCP based protocol) I do see this as expected.

[jszwedko]

For HTTP-based protocols, the total number of bytes in the HTTP body, as represented by the Content-Length header.

given vector is gRPC, which is HTTP-based protocol, shouldn't the metric be after compression, same for HTTP? Similarly for kafka, it's TCP based protocol, and I'd assume it'd be the total number of bytes placed on the socket excluding the delimiter, that is after compression?

Ah, good catch. The spec contradicts itself there. I believe the bit about matching Content-Length header is not correct since it is intended to be prior to compression. I'll open a PR to adjust that.

[jszwedko]

Closing this since the spec has been updated. Let me know if you have any other questions!