No such column attrs in table signoz_metrics.distributed_time_series_v4

[Ruppsn]

Bug description

When updating a 0.57.0 to 0.58.2 Version i get the following log with the new introduced clickhousemetricswritev2 exporter.

otelcol:

{"level":"info","ts":1732012204.8112547,"caller":"internal/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"metrics","name":"clickhousemetricswritev2","error":"code: 16, message: No such column attrs in table signoz_metrics.distributed_time_series_v4 (a541c577-bed4-4a95-84f7-315bc9cfbe0f)","interval":"6.398095484s"}
{"level":"info","ts":1732012206.004339,"caller":"internal/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"metrics","name":"clickhousemetricswritev2","error":"code: 16, message: No such column attrs in table signoz_metrics.distributed_time_series_v4 (a541c577-bed4-4a95-84f7-315bc9cfbe0f)","interval":"5.565095083s"}
{"level":"info","ts":1732012206.6314,"caller":"internal/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"metrics","name":"clickhousemetricswritev2","error":"code: 16, message: No such column attrs in table signoz_metrics.distributed_time_series_v4 (a541c577-bed4-4a95-84f7-315bc9cfbe0f)","interval":"7.117505655s"}
{"level":"info","ts":1732012209.7228727,"caller":"internal/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"metrics","name":"clickhousemetricswritev2","error":"code: 16, message: No such column attrs in table signoz_metrics.distributed_time_series_v4 (a541c577-bed4-4a95-84f7-315bc9cfbe0f)","interval":"3.23480418s"}
{"level":"info","ts":1732012211.3184712,"caller":"internal/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"metrics","name":"clickhousemetricswritev2","error":"code: 16, message: No such column attrs in table signoz_metrics.distributed_time_series_v4 (a541c577-bed4-4a95-84f7-315bc9cfbe0f)","interval":"8.474944602s"}

Is this ignorable?

Expected behavior

How to reproduce

1. Have 0.57.0 running
2. Upgrade to 0.58.2

Version information

• Signoz version: 0.58.2
• Your OS and version: RedHat 9
• Your CPU Architecture(ARM/Intel): Intel

[Ruppsn]

There are about 500k entries of this in 24h.

I am already reading about how to configure the clickhouse db internas but some help would be welcoming.

[srikanthccv]

Did the migrator run successfully? If not, please make sure you have the migration container part of the upgrade and run it. If it ran and didn't exit properly, please share the error.

[Ruppsn]

Thank you for your answer. It seems it is related to my other request. please close the other one if you think it is : #6467

I dont think the migrator is running successful but it completes without an error:

Inside the Logs i cant find anything:

 docker service logs signoz_otel-collector-migrator 
signoz_otel-collector-migrator.1.2r13rs72ngol@    | Signoz Schema Migrator
signoz_otel-collector-migrator.1.2r13rs72ngol@    | 
signoz_otel-collector-migrator.1.2r13rs72ngol@    | Usage:
signoz_otel-collector-migrator.1.2r13rs72ngol@    |   signoz-schema-migrator [command]
signoz_otel-collector-migrator.1.2r13rs72ngol@    | 
signoz_otel-collector-migrator.1.2r13rs72ngol@    | Available Commands:
signoz_otel-collector-migrator.1.2r13rs72ngol@    |   async       Run migrations in async mode
signoz_otel-collector-migrator.1.2r13rs72ngol@    |   completion  Generate the autocompletion script for the specified shell
signoz_otel-collector-migrator.1.2r13rs72ngol@    |   help        Help about any command
signoz_otel-collector-migrator.1.2r13rs72ngol@    |   sync        Run migrations in sync mode
signoz_otel-collector-migrator.1.2r13rs72ngol@    | 
signoz_otel-collector-migrator.1.2r13rs72ngol@    | Flags:
signoz_otel-collector-migrator.1.2r13rs72ngol@    |       --cluster-name string   Cluster name to use while running migrations (default "cluster")
signoz_otel-collector-migrator.1.2r13rs72ngol@    |       --dev                   Development mode
signoz_otel-collector-migrator.1.2r13rs72ngol@    |       --dsn string            Clickhouse DSN
signoz_otel-collector-migrator.1.2r13rs72ngol@    |   -h, --help                  help for signoz-schema-migrator
signoz_otel-collector-migrator.1.2r13rs72ngol@    |       --replication           Enable replication
signoz_otel-collector-migrator.1.2r13rs72ngol@    | 
signoz_otel-collector-migrator.1.2r13rs72ngol@    | Use "signoz-schema-migrator [command] --help" for more information about a command.

Is there more to do than a checkout to the new version and a redeploy?

[srikanthccv]

Seems like the command is not set up properly for swarm. Here is the correct values from docker deploy

signoz/deploy/docker/clickhouse-setup/docker-compose-minimal.yaml

Lines 215 to 228 in d7bd72e

	otel-collector-migrator-sync:
	image: signoz/signoz-schema-migrator:${OTELCOL_TAG:-0.111.5}
	container_name: otel-migrator-sync
	command:
	- "sync"
	- "--dsn=tcp://clickhouse:9000"
	- "--up="
	depends_on:
	clickhouse:
	condition: service_healthy
	# clickhouse-2:
	# condition: service_healthy
	# clickhouse-3:
	# condition: service_healthy

Please try with the same command args. Please feel free to create a pull request fixing the command.

[Ruppsn]

This did the trick :) Thank you very much. I will try to make a pull reqeuest.