Skip to content

Commit

Permalink
Add topic and app for new otel collector
Browse files Browse the repository at this point in the history
The new tail sampler will:

* Read from the current `otel.otlp_spans` topic
* Produce to the new `otel.otlp_sampled_spans`

Later `tempo` will also be configure to read from the new topic and not
the old one, but we'll leave it alone for now.

Ticket: DENA-1126
  • Loading branch information
matthewhughes-uw committed Dec 12, 2024
1 parent d6a720d commit 94b5f07
Showing 1 changed file with 27 additions and 0 deletions.
27 changes: 27 additions & 0 deletions dev-aws/otel/otel.tf
Original file line number Diff line number Diff line change
Expand Up @@ -30,3 +30,30 @@ module "tempo_distributor" {
consume_groups = ["processor-tempo"]
cert_common_name = "otel/tempo-distributor"
}

resource "kafka_topic" "otlp_sampled_spans" {
name = "otel.otlp_sampled_spans"
replication_factor = 3
partitions = 200
config = {
# retain 5GB on each partition
"retention.bytes" = "5368709120"
# keep data for 12 hours
"retention.ms" = "43200000"
# allow max 128 MB for a message
"max.message.bytes" = "134217728"
# roll log at 3h max
"segment.ms" = "10800000"
# max log size of 250 MB
"segment.bytes" = "262144000"
"compression.type" = "zstd"
"cleanup.policy" = "delete"
}
}

module "otel_tail_sampling_collector" {
source = "../../modules/tls-app"
consume_topics = [kafka_topic.otlp_spans.name]
produce_topics = [kafka_topic.otlp_sampled_spans.name]
cert_common_name = "otel/tail-sampling-collector"
}

0 comments on commit 94b5f07

Please sign in to comment.