Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

panic: send on closed channel when issuing CTRL-C while connectivity tests are running on EKS #2765

Open
christarazi opened this issue Aug 16, 2024 · 1 comment
Labels
kind/bug Something isn't working

Comments

@christarazi
Copy link
Member

General Information

  • Cilium CLI version (run cilium version)
cilium-cli: v0.16.15-3-g1ef635612 compiled with go1.22.1 on linux/amd64
cilium image (default): v1.16.0
cilium image (stable): v1.16.1
cilium image (running): 1.15.8
  • Platform / infrastructure information (e.g. AWS / Azure / GCP, image / kernel versions)
    AWS
[=] [cilium-test-1] Test [client-ingress-from-other-client-icmp-deny] [34/86]
............
^CCancellation request (context canceled) received, cancelling tests...
panic: send on closed channel

goroutine 13879 [running]:
github.com/cilium/cilium-cli/connectivity/check.(*ConcurrentLogger).FinishTest(0xc00092eb40, 0xc001470f20)
        /home/chris/code/cilium/cilium-cli/connectivity/check/logger.go:95 +0x18c
github.com/cilium/cilium-cli/connectivity/check.(*ConnectivityTest).Run.func1.1()
        /home/chris/code/cilium/cilium-cli/connectivity/check/context.go:394 +0x2b
github.com/cilium/cilium-cli/connectivity/check.(*ConnectivityTest).Run.func1()
        /home/chris/code/cilium/cilium-cli/connectivity/check/context.go:406 +0x2b3
created by github.com/cilium/cilium-cli/connectivity/check.(*ConnectivityTest).Run in goroutine 762
        /home/chris/code/cilium/cilium-cli/connectivity/check/context.go:392 +0x91
@christarazi christarazi added the kind/bug Something isn't working label Aug 16, 2024
@ldlb9527
Copy link

func (c *ConcurrentLogger) FinishTest(test *Test) {
	//...
	c.messageCh <- message{
		namespace: test.Context().Params().TestNamespace,
		testName:  test.Name(),
		data:      buf.String(),
		finish:    true,
	}
}

Panic occurs in the FinishTest function,the reason is due to the following code:

		ctx, _ := signal.NotifyContext(cmd.Context(), os.Interrupt, syscall.SIGTERM)

		if params.Timeout > 0 {
			timeoutCtx, cancelFunc := context.WithTimeoutCause(ctx, params.Timeout, fmt.Errorf("connectivity test suite timeout (%s) reached", params.Timeout))
			defer cancelFunc()
			ctx = timeoutCtx
		}

		go func() {
			<-ctx.Done()
			connTests[0].Logf("Cancellation request (%s) received, cancelling tests...", context.Cause(ctx))
		}()

		logger.Start(ctx)
		defer logger.Stop()
		return connectivity.Run(ctx, connTests, hooks)

Receiving an interrupt signal will result in logger.Start(ctx) execute close(messageCh),and then a panic occurs

The code might still have an issue where the logger.Stop() function could attempt to execute close(messageCh) a second time, leading to a panic.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants