Receiver: Send tenant by label instead of header

[farodin91]

We are using remote write capability of opentelemetry collector, to send all collected metrics to a receiver. These metrics contain a label which defines the tenant.

It is possible to pass the tenant identifier via a label instead of using a header?

[squat]

hi @farodin91 very good question.
TL;DR: this is not exactly possible today.

Longer version:
The tenant header is used for 3 things:

1. choosing which hashring within the hashring configuration should handle a request;
2. hashing the request to spread out the load; and
3. instantiating a new TSDB instance in each receiver for each tenant.

If your hashring configuration file only defines one default hashring, then 1 doesn't really matter, since this hashring will handle all requests anyways. If you have tenant-specific hashrings, then they will not receive any load.
2 does not really matter any more because your time series contain a tenant-specific label. Since labels are hashed to distribute load across receiver instances, this means that the tenant is implicitly considered as a hashing parameter.
3 is a problem; as far as the receiver knows, all the requests belong to the "default" tenant and will all end up in the same TSDB instance. Depending on your configuration and the type of load you handle, this may or may not be an issue, however we can say for sure that the Thanos receiver cannot provide the same guarantees.

I think that providing the possibility to define a tenant using a label would be a good feature. Furthermore, it would make our tenancy model much more Prometheus-native and extendable.