skywalking=10.0.1 and banyandb=0.7.0-rc0, errors has occurred #12634

Spinestars · 2024-09-23T09:46:13Z

Spinestars
Sep 23, 2024

After running for two or three minutes, banyandb started reporting errors:

{"level":"error","module":"MEASURE","error":"cannot create data points in table: cannot create segment: expired data","written":{"shardId":1,"seriesHash":"qLzgpJ3xfji5AJ+4TEdW8w==","entityValues":[{"str":{"value":"VXNlcg==.0-VXNlcg==-amF5eDo6amF5eC10ZXN0LWZpbmFuY2lhbHNlcnZpY2U=.1-L2hlYWx0aA=="}}],"request":{"metadata":{"group":"measure-minute","name":"endpoint_relation_server_side_minute","modRevision":"55"},"dataPoint":{"timestamp":"2024-09-23T09:38:00Z","tagFamilies":[{"tags":[{"str":{"value":"VXNlcg==.0_VXNlcg=="}},{"str":{"value":"amF5eDo6amF5eC10ZXN0LWZpbmFuY2lhbHNlcnZpY2U=.1_L2hlYWx0aA=="}}]},{"tags":[{"int":{"value":"3001"}},{"str":{"value":"VXNlcg==.0-VXNlcg==-amF5eDo6amF5eC10ZXN0LWZpbmFuY2lhbHNlcnZpY2U=.1-L2hlYWx0aA=="}}]}],"version":"13333525193518869"},"messageId":"13333525193518869"}},"time":"2024-09-23T09:39:07Z","message":"cannot handle write event"}

{"level":"error","module":"METRIC-SERVICE.SCHEDULER.METRICS-COLLECTOR","name":"metrics-collector","panic":"invalid memory address or nil pointer dereference","stack":"goroutine 444 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debug/stack.go:26 +0x5e\ngithub.com/apache/skywalking-banyandb/pkg/timestamp.(*task).run.func1.1()\n\t/mnt/ssd/opt/skywalking-banyandb/pkg/timestamp/scheduler.go:203 +0xd3\npanic({0x1482680?, 0x26f0f00?})\n\t/usr/local/go/src/runtime/panic.go:785 +0x132\ngithub.com/apache/skywalking-banyandb/banyand/measure.(*snapshot).decRef(0xc000da8700?)\n\t/mnt/ssd/opt/skywalking-banyandb/banyand/measure/snapshot.go:73 +0x17\npanic({0x1482680?, 0x26f0f00?})\n\t/usr/local/go/src/runtime/panic.go:785 +0x132\ngithub.com/apache/skywalking-banyandb/banyand/measure.(*tsTable).Collect(0xc000d18fc0, {0x1b43be0?, 0xc000acfa00})\n\t/mnt/ssd/opt/skywalking-banyandb/banyand/measure/metrics.go:300 +0x9a\ngithub.com/apache/skywalking-banyandb/banyand/internal/storage.(*database[...]).collect(0x1b77da0)\n\t/mnt/ssd/opt/skywalking-banyandb/banyand/internal/storage/tsdb.go:163 +0x2ea\ngithub.com/apache/skywalking-banyandb/banyand/observability.(*Collector).collect(0x2?)\n\t/mnt/ssd/opt/skywalking-banyandb/banyand/observability/collector.go:56 +0xd4\ngithub.com/apache/skywalking-banyandb/banyand/observability.(*metricService).Serve.func1({0xc001d1c690?, 0xc00025ea80?, 0xc001d1c690?}, 0xc0067fdf00?)\n\t/mnt/ssd/opt/skywalking-banyandb/banyand/observability/service.go:152 +0x2f\ngithub.com/apache/skywalking-banyandb/pkg/timestamp.(*task).run.func1(0xc000dbd680, 0xc0067fdec8)\n\t/mnt/ssd/opt/skywalking-banyandb/pkg/timestamp/scheduler.go:208 +0xaf\ngithub.com/apache/skywalking-banyandb/pkg/timestamp.(*task).run(0xc000dbd680)\n\t/mnt/ssd/opt/skywalking-banyandb/pkg/timestamp/scheduler.go:209 +0x3f0\ngithub.com/apache/skywalking-banyandb/pkg/timestamp.(*Scheduler).Register.func1()\n\t/mnt/ssd/opt/skywalking-banyandb/pkg/timestamp/scheduler.go:98 +0x47\ncreated by github.com/apache/skywalking-banyandb/pkg/timestamp.(*Scheduler).Register in goroutine 421\n\t/mnt/ssd/opt/skywalking-banyandb/pkg/timestamp/scheduler.go:97 +0x5f1\n","time":"2024-09-23T09:39:43Z","message":"panic"}

and After the error occurred and banyandb restarted, reporting new errors:

{"level":"panic","time":"2024-09-23T09:32:00Z","message":"fails to init the group: File system return error: File is not exist, file name: /tmp/stream/stream-segment/seg-20240923/metadata, error message: open /tmp/stream/stream-segment/seg-20240923/metadata: no such file or directory, error code: 1"}
panic: fails to init the group: File system return error: File is not exist, file name: /tmp/stream/stream-segment/seg-20240923/metadata, error message: open /tmp/stream/stream-segment/seg-20240923/metadata: no such file or directory, error code: 1

goroutine 1 [running]:
github.com/apache/skywalking-banyandb/pkg/logger.Panicf.(*Logger).Panic.func1({0xc0006db600?, 0x0?})
	/home/gao/d/go/pkg/mod/github.com/rs/[email protected]/log.go:405 +0x27
github.com/rs/zerolog.(*Event).msg(0xc002320230, {0xc0006db600, 0xf1})
	/home/gao/d/go/pkg/mod/github.com/rs/[email protected]/event.go:158 +0x2aa
github.com/rs/zerolog.(*Event).Msgf(0xc002320230, {0x16d10d7?, 0x19c7a18?}, {0xc00111f5c0?, 0xc0015bf470?, 0xe?})
	/home/gao/d/go/pkg/mod/github.com/rs/[email protected]/event.go:131 +0x46
github.com/apache/skywalking-banyandb/pkg/logger.Panicf({0x16d10d7, 0x1b}, {0xc00111f5c0, 0x1, 0x1})
	/mnt/ssd/opt/skywalking-banyandb/pkg/logger/suger.go:24 +0x65
github.com/apache/skywalking-banyandb/pkg/schema.(*schemaRepo).processGroup(0xc0017d66e0, {0x1b5af28, 0xc001996b10}, 0xc001a24730, 0x1)
	/mnt/ssd/opt/skywalking-banyandb/pkg/schema/init.go:104 +0x79
github.com/apache/skywalking-banyandb/pkg/schema.(*schemaRepo).Init(0xc0017d66e0, 0x2)
	/mnt/ssd/opt/skywalking-banyandb/pkg/schema/init.go:84 +0x170
github.com/apache/skywalking-banyandb/banyand/stream.(*schemaRepo).OnInit(0xc000fea310?, {0xc0009d7280?, 0x19?, 0x13cab20?})
	/mnt/ssd/opt/skywalking-banyandb/banyand/stream/metadata.go:107 +0x86
github.com/apache/skywalking-banyandb/banyand/metadata/schema.(*etcdSchemaRegistry).RegisterHandler(0xc0011d6120, {0x16b2751, 0x6}, 0x1b, {0x1b54860, 0xc0017da4b0})
	/mnt/ssd/opt/skywalking-banyandb/banyand/metadata/schema/etcd.go:139 +0x216
github.com/apache/skywalking-banyandb/banyand/metadata.(*clientService).RegisterHandler(0xf18765?, {0x16b2751?, 0xf18765?}, 0x1656a20?, {0x1b54860?, 0xc0017da4b0?})
	/mnt/ssd/opt/skywalking-banyandb/banyand/metadata/client.go:161 +0x2d
github.com/apache/skywalking-banyandb/banyand/stream.(*schemaRepo).start(0xc0017da4b0)
	/mnt/ssd/opt/skywalking-banyandb/banyand/stream/metadata.go:90 +0x51
github.com/apache/skywalking-banyandb/banyand/stream.newSchemaRepo({0xc000e56190, 0xb}, 0xc000592000)
	/mnt/ssd/opt/skywalking-banyandb/banyand/stream/metadata.go:67 +0x230
github.com/apache/skywalking-banyandb/banyand/stream.(*service).PreRun(0xc000592000, {0xc000273230?, 0x1482e00?})
	/mnt/ssd/opt/skywalking-banyandb/banyand/stream/service.go:108 +0x1d1
github.com/apache/skywalking-banyandb/pkg/run.(*Group).Run(0xc0005920b0, {0x1b5af28, 0xc000273230})
	/mnt/ssd/opt/skywalking-banyandb/pkg/run/run.go:397 +0x57e
github.com/apache/skywalking-banyandb/pkg/cmdsetup.newStandaloneCmd.func1(0xc00026bb08?, {0x16b3d61?, 0x7?, 0x16b0c9e?})
	/mnt/ssd/opt/skywalking-banyandb/pkg/cmdsetup/standalone.go:97 +0x147
github.com/spf13/cobra.(*Command).execute(0xc00026bb08, {0x2755d80, 0x0, 0x0})
	/home/gao/d/go/pkg/mod/github.com/spf13/[email protected]/command.go:985 +0xaaa
github.com/spf13/cobra.(*Command).ExecuteC(0xc00026b808)
	/home/gao/d/go/pkg/mod/github.com/spf13/[email protected]/command.go:1117 +0x3ff
github.com/spf13/cobra.(*Command).Execute(0xc000465f30?)
	/home/gao/d/go/pkg/mod/github.com/spf13/[email protected]/command.go:1041 +0x13
main.main()
	/mnt/ssd/opt/skywalking-banyandb/banyand/cmd/server/main.go:32 +0x59

Answered by wu-sheng

Sep 23, 2024

Pick helm 4.6, oap 10.0.1 and banyandb 0.6.1. Those are working together.

View full answer

wu-sheng · 2024-09-23T09:48:23Z

wu-sheng
Sep 23, 2024
Collaborator

This is not working. Dev version 0.7 requires the oap 10.1-dev

3 replies

Spinestars Sep 23, 2024
Author

This is not working. Dev version 0.7 requires the oap 10.1-dev

It says 0.7.0 in the docs, but there's no 0.7.0 on dockerhub. Is the latest stable version 0.6.1?

Should skywalking 10.0.1 use version 0.6.1

wu-sheng Sep 23, 2024
Collaborator

Don't use the latest docs, there is no 0.7 and 10.1 releases, only dev or rc.

wu-sheng Sep 23, 2024
Collaborator

And you are using dev version helm, please don't.

Use the released helm always, unless you are going to develop skywalking.

wu-sheng · 2024-09-23T10:14:26Z

wu-sheng
Sep 23, 2024
Collaborator

Pick helm 4.6, oap 10.0.1 and banyandb 0.6.1. Those are working together.

2 replies

Spinestars Sep 24, 2024
Author

sky helm: 4.6.0
banyandb helm: 0.2.0
I am using OAP 10.0.1 and BanyanDB 0.6.1, and the data was just written to the DB less than 1 minute ago, and the DB is already throwing the following error:

{"level":"info","module":"STREAM-SEGMENT","shard_id":1,"time":"2024-09-24T02:34:00Z","message":"creating a shard"}
{"level":"info","module":"STREAM-SEGMENT.SHARD1","shard_id":1,"path":"/tmp/stream/stream-segment/shard-1","time":"2024-09-24T02:34:00Z","message":"creating a shard"}
{"level":"info","module":"STREAM-SEGMENT","shard_id":0,"time":"2024-09-24T02:34:00Z","message":"creating a shard"}
{"level":"info","module":"STREAM-SEGMENT.SHARD0","shard_id":0,"path":"/tmp/stream/stream-segment/shard-0","time":"2024-09-24T02:34:00Z","message":"creating a shard"}
{"level":"info","module":"STREAM-LOG","shard_id":0,"time":"2024-09-24T02:34:01Z","message":"creating a shard"}
{"level":"info","module":"STREAM-LOG.SHARD0","shard_id":0,"path":"/tmp/stream/stream-log/shard-0","time":"2024-09-24T02:34:01Z","message":"creating a shard"}
{"level":"info","module":"MEASURE-MINUTE","shard_id":1,"time":"2024-09-24T02:34:03Z","message":"creating a shard"}
{"level":"info","module":"MEASURE-MINUTE.SHARD1","shard_id":1,"path":"/tmp/measure/measure-minute/shard-1","time":"2024-09-24T02:34:03Z","message":"creating a shard"}
{"level":"info","module":"MEASURE-MINUTE","shard_id":0,"time":"2024-09-24T02:34:03Z","message":"creating a shard"}
{"level":"info","module":"MEASURE-MINUTE.SHARD0","shard_id":0,"path":"/tmp/measure/measure-minute/shard-0","time":"2024-09-24T02:34:03Z","message":"creating a shard"}
{"level":"info","module":"MEASURE-DEFAULT","shard_id":0,"time":"2024-09-24T02:34:03Z","message":"creating a shard"}
{"level":"info","module":"MEASURE-DEFAULT.SHARD0","shard_id":0,"path":"/tmp/measure/measure-default/shard-0","time":"2024-09-24T02:34:03Z","message":"creating a shard"}
{"level":"info","module":"STREAM-SEGMENT.SHARD1","segment":"SegID-20240924","time":"2024-09-24T02:34:15Z","message":"removed a segment"}
panic: runtime error: index out of range [0] with length 0

goroutine 1716 [running]:
github.com/apache/skywalking-banyandb/banyand/internal/storage.(*segmentController[...]).removeSeg(0xedec8aa26, 0x25e31c0?)
	/src/banyand/internal/storage/segment.go:321 +0x1c5
github.com/apache/skywalking-banyandb/banyand/internal/storage.(*segmentController[...]).remove(0x1aa6f00, {0x25e31c0, 0xffff14424c130000?, 0x25e31c0?})
	/src/banyand/internal/storage/segment.go:308 +0x1ba
github.com/apache/skywalking-banyandb/banyand/internal/storage.(*retentionTask[...]).run(0x1a8ab00, {0xc0004fc0d8, 0xc000c0be30?, 0x25e31c0?}, 0xc00106c2d0?)
	/src/banyand/internal/storage/rotation.go:125 +0x1dc
github.com/apache/skywalking-banyandb/banyand/internal/storage.(*database[...]).startRotationTask.func1.1(0x1a95d40?, 0x1aaa7e0?, 0x1808ef608f5b9b40)
	/src/banyand/internal/storage/rotation.go:53 +0x118
github.com/apache/skywalking-banyandb/banyand/internal/storage.(*database[...]).startRotationTask.func1()
	/src/banyand/internal/storage/rotation.go:83 +0x52
created by github.com/apache/skywalking-banyandb/banyand/internal/storage.(*database[...]).startRotationTask in goroutine 320
	/src/banyand/internal/storage/rotation.go:47 +0x18b

Here are my Helm deployment commands:

export SKYWALKING_RELEASE_VERSION=4.6.0  # change the release version according to your need
export SKYWALKING_RELEASE_NAME=skywalking  # change the release name according to your scenario
export SKYWALKING_RELEASE_NAMESPACE=skywalking  # change the namespace to where you want to install SkyWalking

helm upgrade --install "${SKYWALKING_RELEASE_NAME}" \
  -n "${SKYWALKING_RELEASE_NAMESPACE}" \
  --create-namespace \
  -f 4.6.0_10.0.1/myvalue.yaml \
  skywalking-4.6.0 \
  --set oap.image.tag=10.0.1 \
  --set oap.storageType=banyandb \
  --set ui.image.tag=10.0.1 \
  --set elasticsearch.enabled=false \
  --set postgresql.enabled=false \
  --set banyandb.enabled=true \
  --set banyandb.image.tag=0.6.1 \
  --set banyandb.image.repository=myharbor.com/public/apache/skywalking-banyandb \
  --set banyandb.standalone.enabled=true \
  --set banyandb.cluster.enabled=false \
  --set banyandb.standalone.httpSvc.type=NodePort \
  --set banyandb.storage.enabled=true \
  --set "banyandb.storage.persistentVolumeClaims[0].storageClass=cnfs-nas-001" \
  --set "banyandb.storage.persistentVolumeClaims[0].size=100Gi" \
  --set "banyandb.storage.persistentVolumeClaims[1].storageClass=cnfs-nas-001" \
  --set "banyandb.storage.persistentVolumeClaims[1].size=5Gi"

I switched the OAP to version 10.0.0 and BanyanDB to version 0.6.0, and I still got similar errors.
Does this error depend on the client version?

wu-sheng Sep 24, 2024
Collaborator

We are releasing 0.7 soon. If you want to try banyandb, please wait for it.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

skywalking=10.0.1 and banyandb=0.7.0-rc0, errors has occurred #12634

{{title}}

Replies: 2 comments 5 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

skywalking=10.0.1 and banyandb=0.7.0-rc0, errors has occurred #12634

Spinestars Sep 23, 2024

Replies: 2 comments · 5 replies

wu-sheng Sep 23, 2024 Collaborator

Spinestars Sep 23, 2024 Author

wu-sheng Sep 23, 2024 Collaborator

wu-sheng Sep 23, 2024 Collaborator

wu-sheng Sep 23, 2024 Collaborator

Spinestars Sep 24, 2024 Author

wu-sheng Sep 24, 2024 Collaborator

Spinestars
Sep 23, 2024

Replies: 2 comments 5 replies

wu-sheng
Sep 23, 2024
Collaborator

Spinestars Sep 23, 2024
Author

wu-sheng Sep 23, 2024
Collaborator

wu-sheng Sep 23, 2024
Collaborator

wu-sheng
Sep 23, 2024
Collaborator

Spinestars Sep 24, 2024
Author

wu-sheng Sep 24, 2024
Collaborator