Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sql: TestTenantGlobalAggregatedLivebytes failed #132962

Open
cockroach-teamcity opened this issue Oct 18, 2024 · 1 comment
Open

sql: TestTenantGlobalAggregatedLivebytes failed #132962

cockroach-teamcity opened this issue Oct 18, 2024 · 1 comment
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-sql-queries SQL Queries Team

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Oct 18, 2024 •
edited by cockroach-jira-scripts
Loading

sql.TestTenantGlobalAggregatedLivebytes failed with artifacts on master @ d767b9d92dbb8bb547927696847b704cc71906a4:

github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0x40123b5948?, {0x82d3e60, 0x401178c630})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:226 +0x68
panic({0x5dff7c0?, 0xc1e3390?})
	GOROOT/src/runtime/panic.go:770 +0x124
github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotQueue).Admit(0x0, {0x82d3e60, 0x401178cc60}, 0x565)
	github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:165 +0x4c
github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotPacer).Pace(0x401330c740, {0x82d3e60?, 0x401178cc60?}, 0x4005b89a62?, 0x1a?)
	github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:292 +0x70
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*kvBatchSnapshotStrategy).Receive(_, {_, _}, _, {_, _}, {{0xe, 0x2, 0x400fbf2930, 0x400fbf29a0, ...}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:918 +0x16f4
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot(0x400efe1408, {0x82d3e60, 0x401178c630}, 0x40073adb08, {0xffff5ed92cd8, 0x40076b8b70})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:1693 +0x538
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1({0x82d3e60, 0x401178c630})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:209 +0xb0
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0x400cc47b90, {0x82d3e60, 0x401178c630}, {0x69c30d6, 0x1e}, 0x40117eb268)
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 +0x100
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot(0x400efe1408, {0x82d3e28?, 0x400d7917c0?}, 0x40073adb08, {0xffff5ed92cb0, 0x40076b8b70})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:206 +0xc4
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot(0x400d567180, {0x8322570, 0x40076b8b70})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go:638 +0x294
github.com/cockroachdb/cockroach/pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler({0x67f2900?, 0x400d567180}, {0x830b9a8, 0x4004658b00})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/bazel-out/aarch64-fastbuild/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:267 +0xdc
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13({0x67f2900, 0x400d567180}, {0x830cc80, 0x40105132c0}, 0x400d878f00, 0x6dec080)
	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:160 +0x244
google.golang.org/grpc.getChainStreamHandler.func1({0x67f2900, 0x400d567180}, {0x830cc80, 0x40105132c0})
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xa0
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4({0x67f2900, 0x400d567180}, {0x830cc80, 0x40105132c0}, 0x400d878f00?, 0x4010e8a740)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:174 +0x78
google.golang.org/grpc.getChainStreamHandler.func1({0x67f2900, 0x400d567180}, {0x830cc80, 0x40105132c0})
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xa0
github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor({0x4007af2000?, {{0x64594a0?}, {0x832dbc0?, 0x400b3813e0?}}}, {0x67f2900, 0x400d567180}, {0x830cc80, 0x40105132c0}, 0x400d878f00, 0x4010e8a700)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:152 +0x2e4
google.golang.org/grpc.getChainStreamHandler.func1({0x67f2900, 0x400d567180}, {0x830cc80, 0x40105132c0})
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xa0
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1({0x401381c9c8?, 0x82d3e60?})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:141 +0x34
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0x400cc47b90, {0x82d3e60, 0x401178c4b0}, {0x400691b140, 0x29}, 0x401381ca88)
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 +0x100
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2({0x67f2900, 0x400d567180}, {0x830cc80, 0x40105132c0}, 0x400d878f00, 0x4010e8a6c0)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:140 +0xc0
google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2({0x67f2900, 0x400d567180}, {0x830cc80, 0x40105132c0}, 0x400d878f00, 0x40076b8b60?)
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1472 +0x88
google.golang.org/grpc.(*Server).processStreamingRPC(0x40097ba5a0, {0x832e520, 0x400ae549c0}, 0x4006d7c900, 0x400fbc8120, 0xc2e05a0, 0x0)
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1636 +0xf50
google.golang.org/grpc.(*Server).handleStream(0x40097ba5a0, {0x832e520, 0x400ae549c0}, 0x4006d7c900, 0x0)
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1716 +0x780
google.golang.org/grpc.(*Server).serveStreams.func1.1()
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:947 +0xb4
created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 68373
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:958 +0x14c
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

Jira issue: CRDB-43363
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-sql-queries SQL Queries Team labels Oct 18, 2024
@cockroach-teamcity cockroach-teamcity mentioned this issue Oct 19, 2024
Open
@cockroach-teamcity
Copy link
Member Author

sql.TestTenantGlobalAggregatedLivebytes failed on master @ 472ea07a5232c98536293d13bb46cca59f9f2cd0:

Fatal error:

panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x185ad11]

Stack:

goroutine 49846 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0xc00bcc1820?, {0x8bb4548, 0xc001b98b10})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:226 +0x65
panic({0x66d4500?, 0xce28b40?})
	GOROOT/src/runtime/panic.go:770 +0x132
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0xc00bcc1940?, {0x8bb4548, 0xc001b98c90})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:226 +0x65
panic({0x66d4500?, 0xce28b40?})
	GOROOT/src/runtime/panic.go:770 +0x132
github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotQueue).Admit(0x0, {0x8bb4548, 0xc001b99050}, 0x1140)
	github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:165 +0x71
github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotPacer).Pace(0xc00bcc2720, {0x8bb4548?, 0xc001b99050?}, 0xc006a3fce0?, 0x48?)
	github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:292 +0x6d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*kvBatchSnapshotStrategy).Receive(_, {_, _}, _, {_, _}, {{0x11, 0x5, 0xc0094e21c0, 0xc0094e22a0, ...}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:918 +0x1f9f
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot(0xc0030cac08, {0x8bb4548, 0xc001b98c90}, 0xc007cf5688, {0x7f0a00d0fd00, 0xc002384360})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:1693 +0x797
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1({0x8bb4548, 0xc001b98c90})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:209 +0x85
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc00a52d0e0, {0x8bb4548, 0xc001b98c90}, {0x729925b, 0x1e}, 0xc00f14f268)
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 +0x15d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot(0xc0030cac08, {0x8bb4510?, 0xc0121cfd10?}, 0xc007cf5688, {0x7f0a00d0fcd8, 0xc002384360})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:206 +0xf6
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot(0xc006cf8380, {0x8c027f0, 0xc002384360})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go:638 +0x30d
github.com/cockroachdb/cockroach/pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler({0x70c9720?, 0xc006cf8380}, {0x8bebd00, 0xc007ac40c0})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:267 +0xd8
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13({0x70c9720, 0xc006cf8380}, {0x8becf38, 0xc0079073b0}, 0xc00ac983c0, 0x76c2ec0)
	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:160 +0x35b
google.golang.org/grpc.getChainStreamHandler.func1({0x70c9720, 0xc006cf8380}, {0x8becf38, 0xc0079073b0})
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xb2
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4({0x70c9720, 0xc006cf8380}, {0x8becf38, 0xc0079073b0}, 0xc00ac983c0?, 0xc008850d80)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:174 +0x6c
google.golang.org/grpc.getChainStreamHandler.func1({0x70c9720, 0xc006cf8380}, {0x8becf38, 0xc0079073b0})
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xb2
github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor({0xc0029a4000?, {{0x470445?}, {0x8c0df00?, 0xc00596e8a0?}}}, {0x70c9720, 0xc006cf8380}, {0x8becf38, 0xc0079073b0}, 0xc00ac983c0, 0xc008850d40)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:152 +0x384
google.golang.org/grpc.getChainStreamHandler.func1({0x70c9720, 0xc006cf8380}, {0x8becf38, 0xc0079073b0})
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xb2
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1({0x7f0a01f361a0?, 0x8bb4548?})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:141 +0x27
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc00a52d0e0, {0x8bb4548, 0xc001b98b10}, {0xc0040fb920, 0x29}, 0xc003579aa0)
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 +0x15d
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2({0x70c9720, 0xc006cf8380}, {0x8becf38, 0xc0079073b0}, 0xc00ac983c0, 0xc008850d00)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:140 +0xdf
google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2({0x70c9720, 0xc006cf8380}, {0x8becf38, 0xc0079073b0}, 0xc00ac983c0, 0xc002384350?)
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1472 +0x85
google.golang.org/grpc.(*Server).processStreamingRPC(0xc003242d20, {0x8c0e860, 0xc007d07ba0}, 0xc0075538c0, 0xc009eea600, 0xcf25d60, 0x0)
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1636 +0x1247
google.golang.org/grpc.(*Server).handleStream(0xc003242d20, {0x8c0e860, 0xc007d07ba0}, 0xc0075538c0, 0x0)
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1716 +0x999
google.golang.org/grpc.(*Server).serveStreams.func1.1()
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:947 +0xbb
created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 49834
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:958 +0x136
Log preceding fatal error 

*   | 	GOROOT/src/runtime/signal_unix.go:881
*   | github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotQueue).Admit
*   | 	github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:165
*   | github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotPacer).Pace
*   | 	github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:292
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*kvBatchSnapshotStrategy).Receive
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:918
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:1693
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:209
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
*   | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:206
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go:638
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:267
*   | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13
*   | 	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:160
*   | google.golang.org/grpc.getChainStreamHandler.func1
*   | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481
*   | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4
*   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:174
*   | google.golang.org/grpc.getChainStreamHandler.func1
*   | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481
*   | github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor
*   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:152
*   | google.golang.org/grpc.getChainStreamHandler.func1
*   | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481
*   | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1
*   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:141
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
*   | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349
*   | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2
*   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:140
*   | google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2
*   | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1472
*   | google.golang.org/grpc.(*Server).processStreamingRPC
*   | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1636
*   | google.golang.org/grpc.(*Server).handleStream
*   | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1716
*   | google.golang.org/grpc.(*Server).serveStreams.func1.1
*   | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:947
*   | runtime.goexit
*   | 	src/runtime/asm_amd64.s:1695
* Wraps: (2) runtime error: invalid memory address or nil pointer dereference
* Error types: (1) *withstack.withStack (2) runtime.errorString
*

Parameters:

  • attempt=1
  • run=2
  • shard=11
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-sql-queries SQL Queries Team
Projects
SQL Queries
Status: Triage
Milestone
No milestone
Development

No branches or pull requests
1 participant
@cockroach-teamcity