[Bug]: Goroutine leaks

[miparnisari]

Testcontainers version

v0.32.0

Using the latest Testcontainers version?

Yes

Host OS

MacOS

Host arch

ARM

Go version

1.22.5

Docker version

Client:
 Version:           27.0.3
 API version:       1.46
 Go version:        go1.21.11
 Git commit:        7d4bcd8
 Built:             Fri Jun 28 23:59:41 2024
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.32.0 (157355)
 Engine:
  Version:          27.0.3
  API version:      1.46 (minimum version 1.24)
  Go version:       go1.21.11
  Git commit:       662f78c
  Built:            Sat Jun 29 00:02:44 2024
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.7.18
  GitCommit:        ae71819c4f5e67bb4d5ae76a6b735f29cc25774e
 runc:
  Version:          1.7.18
  GitCommit:        v1.1.13-0-g58aa920
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Docker info

Client:
 Version:    27.0.3
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.15.1-desktop.1
    Path:     /Users/maria.inesparnisari/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.28.1-desktop.1
    Path:     /Users/maria.inesparnisari/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.32
    Path:     /Users/maria.inesparnisari/.docker/cli-plugins/docker-debug
  desktop: Docker Desktop commands (Alpha) (Docker Inc.)
    Version:  v0.0.14
    Path:     /Users/maria.inesparnisari/.docker/cli-plugins/docker-desktop
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     /Users/maria.inesparnisari/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.25
    Path:     /Users/maria.inesparnisari/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.5
    Path:     /Users/maria.inesparnisari/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.3.0
    Path:     /Users/maria.inesparnisari/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/maria.inesparnisari/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.10.0
    Path:     /Users/maria.inesparnisari/.docker/cli-plugins/docker-scout

Server:
 Containers: 28
  Running: 28
  Paused: 0
  Stopped: 0
 Images: 56
 Server Version: 27.0.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae71819c4f5e67bb4d5ae76a6b735f29cc25774e
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.6.32-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 12
 Total Memory: 7.657GiB
 Name: docker-desktop
 ID: 64d29255-0721-4aaa-a15c-caf7a351e3d7
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///Users/maria.inesparnisari/Library/Containers/com.docker.docker/Data/docker-cli.sock
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

What happened?

I attempted an upgrade in our codebase from v0.30.0 to v0.32.0. My codebase uses

• github.com/testcontainers/testcontainers-go v0.30.0
• github.com/testcontainers/testcontainers-go/modules/mysql v0.30.0
• github.com/testcontainers/testcontainers-go/modules/postgres v0.30.0

And after the upgrade, one of my tests is failing:

--- FAIL: TestServerMetricsReporting (13.86s)
    --- FAIL: TestServerMetricsReporting/mysql (10.76s)
            run_test.go:801: found unexpected goroutines:
            [Goroutine 792 in state IO wait, with internal/poll.runtime_pollWait on top of the stack:
            internal/poll.runtime_pollWait(0x104bc3300, 0x72)
                /opt/homebrew/Cellar/go/1.22.5/libexec/src/runtime/netpoll.go:345 +0xa0
            internal/poll.(*pollDesc).wait(0xc0008a5f20, 0x72, 0x0)
                /opt/homebrew/Cellar/go/1.22.5/libexec/src/internal/poll/fd_poll_runtime.go:84 +0xb8
            internal/poll.(*pollDesc).waitRead(...)
                /opt/homebrew/Cellar/go/1.22.5/libexec/src/internal/poll/fd_poll_runtime.go:89
            internal/poll.(*FD).Read(0xc0008a5f00, {0xc0009aa000, 0x1000, 0x1000})
                /opt/homebrew/Cellar/go/1.22.5/libexec/src/internal/poll/fd_unix.go:164 +0x320
            net.(*netFD).Read(0xc0008a5f00, {0xc0009aa000, 0x1000, 0x1000})
                /opt/homebrew/Cellar/go/1.22.5/libexec/src/net/fd_posix.go:55 +0x48
            net.(*conn).Read(0xc0003869b8, {0xc0009aa000, 0x1000, 0x1000})
                /opt/homebrew/Cellar/go/1.22.5/libexec/src/net/net.go:185 +0x88
            net/http.(*persistConn).Read(0xc000d20240, {0xc0009aa000, 0x1000, 0x1000})
                /opt/homebrew/Cellar/go/1.22.5/libexec/src/net/http/transport.go:1977 +0xc4
            bufio.(*Reader).fill(0xc000e55f20)
                /opt/homebrew/Cellar/go/1.22.5/libexec/src/bufio/bufio.go:110 +0x21c
            bufio.(*Reader).Peek(0xc000e55f20, 0x1)
                /opt/homebrew/Cellar/go/1.22.5/libexec/src/bufio/bufio.go:148 +0xb4
            net/http.(*persistConn).readLoop(0xc000d20240)
                /opt/homebrew/Cellar/go/1.22.5/libexec/src/net/http/transport.go:2141 +0x264
            created by net/http.(*Transport).dialConn in goroutine 791
                /opt/homebrew/Cellar/go/1.22.5/libexec/src/net/http/transport.go:1799 +0x1c64
             Goroutine 793 in state select, with net/http.(*persistConn).writeLoop on top of the stack:
            net/http.(*persistConn).writeLoop(0xc000d20240)
                /opt/homebrew/Cellar/go/1.22.5/libexec/src/net/http/transport.go:2458 +0x120
            created by net/http.(*Transport).dialConn in goroutine 791
                /opt/homebrew/Cellar/go/1.22.5/libexec/src/net/http/transport.go:1800 +0x1cd0
            ]

This test spins up a container with either Postgres or MySQL, runs some tests with a server that connects to it, and then terminates the container. Nothing crazy. And at the end of the test I have this:

t.Cleanup(func() {
		goleak.VerifyNone(t)
	})

to detect any goroutines that were created but not terminated.

So there is something new in testcontainers that is not cleaning properly. The leak happens in both MySQL and Postgres tests. Also it doesn't always occur.

Relevant log output

No response

Additional information

If I may make a recommendation: please add

t.Cleanup(func() {
		goleak.VerifyNone(t)
	})

to every test that creates goroutines or HTTP connections. It's proven to be invaluable in our codebase to detect leaks.

[stevenh]

Can you verify this on the branch for #2664?

[mdelapenya]

@miparnisari we released v0.33.0 Could you please check with that version?

If fixed, I think we can close this one, thanks!