Sentry: cmd_recover_txn.go:136: failed indeterminate commit recovery: programming error: timestamp change by implicitly committed transaction: 1729366007.695096496,2->1729366004.162022264,2 (1) attach...

[cockroach-sentry]

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/6005431687/?referrer=webhooks_plugin

Panic Message:

cmd_recover_txn.go:136: failed indeterminate commit recovery: programming error: timestamp change by implicitly committed transaction: 1729366007.695096496,2->1729366004.162022264,2
(1) attached stack trace
  -- stack trace:
  | github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc.(*Inbox).Next.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc/inbox.go:324
  | runtime.gopanic
  | 	GOROOT/src/runtime/panic.go:914
  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.ExpectedError
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:203
  | github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc.(*Inbox).Next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc/inbox.go:385
  | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*ParallelUnorderedSynchronizer).Init.(*ParallelUnorderedSynchronizer).Init.func1.func2
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/parallel_unordered_synchronizer.go:198
  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:92
  | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*ParallelUnorderedSynchronizer).init.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/parallel_unordered_synchronizer.go:259
  | runtime.goexit
  | 	src/runtime/asm_amd64.s:1650
Wraps: (2)
Wraps: (3) tags: [n6,client=127.0.0.1:50086,hostssl,user=root,f×,distsql.stmt=×,distsql.gateway=6,distsql.appname=×,distsql.txn=×,streamID=×,received-error]
Wraps: (4) tags: [n×,f×,distsql.stmt=×,distsql.gateway=×,distsql.appname=×,distsql.txn=×,streamID=×,sent-error=]
Wraps: (5)
Wraps: (6)
  | (opaque error wrapper)
  | type name: github.com/cockroachdb/cockroach/pkg/sql/colexecerror/*colexecerror.StorageError
Wraps: (7)
  | (opaque error wrapper)
  | type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
  | reportable 0:
  |
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/txnrecovery.(*manager).ResolveIndeterminateCommit
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/txnrecovery/manager.go:109
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleIndeterminateCommitError
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:795
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:567
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:189
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:193
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:202
  | github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal
  | 	github.com/cockroachdb/cockroach/pkg/server/node.go:1328
  | github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch
  | 	github.com/cockroachdb/cockroach/pkg/server/node.go:1462
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:704
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12
  | 	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:97
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:169
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815
  | github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:105
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:136
  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
  | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:336
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:134
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func2
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:714
  | github.com/cockroachdb/cockroach/pkg/rpc.NewContext.ClientInterceptor.func8
  | 	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:228
  | github.com/cockroachdb/cockroach/pkg/rpc.getChainUnaryInvoker.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:899
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func3
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:784
  | github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:907
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:211
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:189
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:2419
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1920
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1488
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1104
  | github.com/cockroachdb/cockroach/pkg/kv.(*CrossRangeTxnWrapperSender).Send
  | 	github.com/cockroachdb/cockroach/pkg/kv/db.go:224
  | github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender
  | 	github.com/cockroachdb/cockroach/pkg/kv/db.go:1114
Wraps: (8) failed indeterminate commit recovery
Wraps: (9) assertion failure
Wraps: (10)
  | (opaque error wrapper)
  | type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
  | reportable 0:
  |
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval.RecoverTxn
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval/cmd_recover_txn.go:136
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.evaluateCommand
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_evaluate.go:487
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.evaluateBatch
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_evaluate.go:304
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateWriteBatchWrapper
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go:720
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateWriteBatchWithServersideRefreshes
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_write...

Stacktrace (expand for inline code snippets):

src/runtime/asm_amd64.s#L1649-L1651

cockroach/pkg/sql/colexec/parallel_unordered_synchronizer.go

Lines 258 to 260 in 4634a65

	case parallelUnorderedSynchronizerStateRunning:
	if err := colexecerror.CatchVectorizedRuntimeError(s.nextBatch[inputIdx]); err != nil {
	if s.getState() == parallelUnorderedSynchronizerStateDraining && s.Ctx.Err() == nil && s.cancelLocalInput[inputIdx] != nil {

cockroach/pkg/sql/colexecerror/error.go

Lines 91 to 93 in 4634a65

	}()
	operation()
	return retErr

cockroach/pkg/sql/colexec/parallel_unordered_synchronizer.go

Lines 197 to 199 in 4634a65

	return func() {
	s.batches[inputIdx] = inputOp.Next()
	}

cockroach/pkg/sql/colflow/colrpc/inbox.go

Lines 384 to 386 in 4634a65

	if receivedErr != nil {
	colexecerror.ExpectedError(receivedErr)
	}

cockroach/pkg/sql/colexecerror/error.go

Lines 202 to 204 in 4634a65

	func ExpectedError(err error) {
	panic(newNotInternalError(err))
	}

GOROOT/src/runtime/panic.go#L913-L915

cockroach/pkg/sql/colflow/colrpc/inbox.go

Lines 323 to 325 in 4634a65

	}
	err := logcrash.PanicAsError(0, panicObj)
	log.VEventf(i.Ctx, 1, "Inbox encountered an error in Next: %v", err)

cockroach/pkg/kv/db.go

Lines 1113 to 1115 in 4634a65

	br, pErr := sender.Send(ctx, ba)
	if pErr != nil {

cockroach/pkg/kv/db.go

Lines 223 to 225 in 4634a65

	br, pErr := s.wrapped.Send(ctx, ba)
	if _, ok := pErr.GetDetail().(*kvpb.OpRequiresTxnError); !ok {

cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go

Lines 1103 to 1105 in 4634a65

	} else {
	rpl, pErr = ds.divideAndSendBatchToRanges(ctx, ba, rs, isReverse, withCommit, 0 /* batchIdx */)
	}

cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go

Lines 1487 to 1489 in 4634a65

	if !ri.NeedAnother(rs) {
	resp := ds.sendPartialBatch(
	ctx, ba, rs, isReverse, withCommit, batchIdx, ri.Token(),

cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go

Lines 1919 to 1921 in 4634a65

	prevTok = routingTok
	reply, err = ds.sendToReplicas(ctx, ba, routingTok, withCommit)

cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go

Lines 2418 to 2420 in 4634a65

	br, err = transport.SendNext(ctx, ba)
	ds.metrics.updateCrossLocalityMetricsOnReplicaAddressedBatchResponse(comparisonResult, int64(br.Size()))

cockroach/pkg/kv/kvclient/kvcoord/transport.go

Lines 188 to 190 in 4634a65

	}
	return gt.sendBatch(ctx, r.NodeID, iface, ba)
	}

cockroach/pkg/kv/kvclient/kvcoord/transport.go

Lines 210 to 212 in 4634a65

	log.VEvent(ctx, 2, "sending batch request")
	reply, err := iface.Batch(ctx, ba)
	log.VEvent(ctx, 2, "received batch response")

https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/rpc/pkg/rpc/context.go#L906-L908
https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/rpc/pkg/rpc/context.go#L783-L785
https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/rpc/pkg/rpc/context.go#L898-L900

cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go

Lines 227 to 229 in 4634a65

	if localRequest {
	return invoker(ctx, method, req, resp, cc, opts...)
	}

https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/rpc/pkg/rpc/context.go#L713-L715
https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/rpc/pkg/rpc/context.go#L814-L816
https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/rpc/pkg/rpc/context.go#L133-L135

cockroach/pkg/util/stop/stopper.go

Lines 335 to 337 in 4634a65

	return f(ctx)
	}

https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/rpc/pkg/rpc/context.go#L135-L137
https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/rpc/pkg/rpc/context.go#L814-L816
https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/rpc/pkg/rpc/auth.go#L104-L106
https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/rpc/pkg/rpc/context.go#L814-L816
https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/rpc/pkg/rpc/context.go#L168-L170
https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/rpc/pkg/rpc/context.go#L814-L816

cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go

Lines 96 to 98 in 4634a65

	if methodExcludedFromTracing(info.FullMethod) {
	return handler(ctx, req)
	}

https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/rpc/pkg/rpc/context.go#L703-L705

cockroach/pkg/server/node.go

Lines 1461 to 1463 in 4634a65

	br, err := n.batchInternal(ctx, tenantID, args)

cockroach/pkg/server/node.go

Lines 1327 to 1329 in 4634a65

	var pErr *kvpb.Error
	br, writeBytes, pErr = n.stores.SendWithWriteBytes(ctx, args)
	if pErr != nil {

https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/kv/kvserver/pkg/kv/kvserver/stores.go#L201-L203
https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go#L192-L194
https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L188-L190
https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L566-L568
https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L794-L796

cockroach/pkg/kv/kvserver/txnrecovery/manager.go

Lines 108 to 110 in 4634a65

	log.VEventf(ctx, 2, "recovery error: %v", res.Err)
	return nil, errors.Wrap(res.Err, "failed indeterminate commit recovery")
	}

https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/rpc/pkg/rpc/context.go#L906-L908
https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/rpc/pkg/rpc/context.go#L783-L785
https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/rpc/pkg/rpc/context.go#L898-L900

cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go

Lines 227 to 229 in 4634a65

	if localRequest {
	return invoker(ctx, method, req, resp, cc, opts...)
	}

https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/rpc/pkg/rpc/context.go#L713-L715
https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/rpc/pkg/rpc/context.go#L814-L816
https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/rpc/pkg/rpc/context.go#L133-L135

cockroach/pkg/util/stop/stopper.go

Lines 335 to 337 in 4634a65

	return f(ctx)
	}

https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/rpc/pkg/rpc/context.go#L135-L137
https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/rpc/pkg/rpc/context.go#L814-L816
https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/rpc/pkg/rpc/auth.go#L104-L106
https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/rpc/pkg/rpc/context.go#L814-L816
https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/rpc/pkg/rpc/context.go#L168-L170
https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/rpc/pkg/rpc/context.go#L814-L816

cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go

Lines 96 to 98 in 4634a65

	if methodExcludedFromTracing(info.FullMethod) {
	return handler(ctx, req)
	}

https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/rpc/pkg/rpc/context.go#L703-L705

cockroach/pkg/server/node.go

Lines 1461 to 1463 in 4634a65

	br, err := n.batchInternal(ctx, tenantID, args)

cockroach/pkg/server/node.go

Lines 1327 to 1329 in 4634a65

	var pErr *kvpb.Error
	br, writeBytes, pErr = n.stores.SendWithWriteBytes(ctx, args)
	if pErr != nil {

https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/kv/kvserver/pkg/kv/kvserver/stores.go#L201-L203
https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go#L192-L194
https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L188-L190
https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L510-L512
https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go#L178-L180
https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L123-L125
https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go#L1001-L1003
https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go#L913-L915
https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go#L456-L458
https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go#L686-L688
https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go#L719-L721
https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/kv/kvserver/pkg/kv/kvserver/replica_evaluate.go#L303-L305
https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/kv/kvserver/pkg/kv/kvserver/replica_evaluate.go#L486-L488

cockroach/pkg/kv/kvserver/batcheval/cmd_recover_txn.go

Lines 135 to 137 in 4634a65

	if was, is := args.Txn.WriteTimestamp, reply.RecoveredTxn.WriteTimestamp; was != is {
	return result.Result{}, errors.AssertionFailedf(
	"programming error: timestamp change by implicitly committed transaction: %v->%v", was, is,

src/runtime/asm_amd64.s in runtime.goexit at line 1650
pkg/sql/colexec/parallel_unordered_synchronizer.go in pkg/sql/colexec.(*ParallelUnorderedSynchronizer).init.func1 at line 259
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 92
pkg/sql/colexec/parallel_unordered_synchronizer.go in pkg/sql/colexec.(*ParallelUnorderedSynchronizer).Init.(*ParallelUnorderedSynchronizer).Init.func1.func2 at line 198
pkg/sql/colflow/colrpc/inbox.go in pkg/sql/colflow/colrpc.(*Inbox).Next at line 385
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.ExpectedError at line 203
GOROOT/src/runtime/panic.go in runtime.gopanic at line 914
pkg/sql/colflow/colrpc/inbox.go in pkg/sql/colflow/colrpc.(*Inbox).Next.func1 at line 324
pkg/kv/db.go in pkg/kv.(*DB).sendUsingSender at line 1114
pkg/kv/db.go in pkg/kv.(*CrossRangeTxnWrapperSender).Send at line 224
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).Send at line 1104
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges at line 1488
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch at line 1920
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas at line 2419
pkg/kv/kvclient/kvcoord/transport.go in pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext at line 189
pkg/kv/kvclient/kvcoord/transport.go in pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch at line 211
pkg/rpc/pkg/rpc/context.go in pkg/rpc.internalClientAdapter.Batch at line 907
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func3 at line 784
pkg/rpc/pkg/rpc/context.go in pkg/rpc.getChainUnaryInvoker.func1 at line 899
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/rpc.NewContext.ClientInterceptor.func8 at line 228
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func2 at line 714
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 815
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func1 at line 134
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 336
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func1.1 at line 136
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 815
pkg/rpc/pkg/rpc/auth.go in pkg/rpc.kvAuth.unaryInterceptor at line 105
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 815
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func3 at line 169
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 815
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/rpc.NewServerEx.ServerInterceptor.func12 at line 97
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func1 at line 704
pkg/server/node.go in pkg/server.(*Node).Batch at line 1462
pkg/server/node.go in pkg/server.(*Node).batchInternal at line 1328
pkg/kv/kvserver/pkg/kv/kvserver/stores.go in pkg/kv/kvserver.(*Stores).SendWithWriteBytes at line 202
pkg/kv/kvserver/pkg/kv/kvserver/store_send.go in pkg/kv/kvserver.(*Store).SendWithWriteBytes at line 193
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).SendWithWriteBytes at line 189
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries at line 567
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).handleIndeterminateCommitError at line 795
pkg/kv/kvserver/txnrecovery/manager.go in pkg/kv/kvserver/txnrecovery.(*manager).ResolveIndeterminateCommit at line 109
pkg/rpc/pkg/rpc/context.go in pkg/rpc.internalClientAdapter.Batch at line 907
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func3 at line 784
pkg/rpc/pkg/rpc/context.go in pkg/rpc.getChainUnaryInvoker.func1 at line 899
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/rpc.NewContext.ClientInterceptor.func8 at line 228
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func2 at line 714
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 815
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func1 at line 134
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 336
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func1.1 at line 136
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 815
pkg/rpc/pkg/rpc/auth.go in pkg/rpc.kvAuth.unaryInterceptor at line 105
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 815
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func3 at line 169
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 815
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/rpc.NewServerEx.ServerInterceptor.func12 at line 97
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func1 at line 704
pkg/server/node.go in pkg/server.(*Node).Batch at line 1462
pkg/server/node.go in pkg/server.(*Node).batchInternal at line 1328
pkg/kv/kvserver/pkg/kv/kvserver/stores.go in pkg/kv/kvserver.(*Stores).SendWithWriteBytes at line 202
pkg/kv/kvserver/pkg/kv/kvserver/store_send.go in pkg/kv/kvserver.(*Store).SendWithWriteBytes at line 193
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).SendWithWriteBytes at line 189
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries at line 511
pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go in pkg/kv/kvserver.(*Replica).executeWriteBatch at line 179
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).evalAndPropose at line 124
pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go in pkg/kv/kvserver.(*Replica).requestToProposal at line 1002
pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go in pkg/kv/kvserver.(*Replica).evaluateProposal at line 914
pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go in pkg/kv/kvserver.(*Replica).evaluateWriteBatch at line 457
pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go in pkg/kv/kvserver.(*Replica).evaluateWriteBatchWithServersideRefreshes at line 687
pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go in pkg/kv/kvserver.(*Replica).evaluateWriteBatchWrapper at line 720
pkg/kv/kvserver/pkg/kv/kvserver/replica_evaluate.go in pkg/kv/kvserver.evaluateBatch at line 304
pkg/kv/kvserver/pkg/kv/kvserver/replica_evaluate.go in pkg/kv/kvserver.evaluateCommand at line 487
pkg/kv/kvserver/batcheval/cmd_recover_txn.go in pkg/kv/kvserver/batcheval.RecoverTxn at line 136

Tags

Tag	Value
Command	server
Environment	v23.2.2
Go Version	go1.21.5 X:nocoverageredesign
Platform	linux amd64
Distribution	CCL
Cockroach Release	v23.2.2
Cockroach SHA	4634a65
# of CPUs	20
# of Goroutines	804

Jira issue: CRDB-43406