Bug Report: vttestserver-latest Connection refused on grpc port

[danjwilks]

Overview of the Issue

The gRPC port on for the latest version of the vttestserver (vttestserver:mysql80) container is not accessible, while it was accessible in the previous versions vttestserver:v21.0.0-mysql80 & vttestserver:v20.0.3-mysql80

Reproduction Steps

In our repro below, we set PORT=33574 and try to establish a TCP connection to the container on port 33575 which should be the grpc port as per documentation. For vttestserver-latest, we see connect to 172.17.0.4 port 33575 (tcp) failed: Connection refused whereas for vttestserver-21.0.0 and vttestserver-20.0.3 we see Connection to 172.17.0.3 33575 port [tcp/*] succeeded!. Below you can find the repro script followed by its output in the logs fragment section.

#!/usr/bin/env bash

set -eo pipefail

docker_run_args=(
    '-e' 'PORT=33574'
    '-e' 'KEYSPACES=test,unsharded'
    '-e' 'NUM_SHARDS=2,1'
    '-e' 'MYSQL_MAX_CONNECTIONS=70000'
    '-e' 'MYSQL_BIND_HOST=0.0.0.0'
    '--health-cmd="mysqladmin ping -h127.0.0.1 -P33577"'
    '--health-interval=5s'
    '--health-timeout=2s'
    '--health-retries=5'
    '--rm'
)

mysql_port=33577
grpc_port=33575

version_20_container_name=vttestserver-20.0.3
version_21_container_name=vttestserver-21.0.0
latest_container_name=vttestserver-latest

trap "docker kill ${version_20_container_name} >/dev/null ; docker kill ${version_21_container_name} >/dev/null ; docker kill ${latest_container_name} >/dev/null" EXIT

# Version 20.0.3
echo '|------ Version 20.0.3 ---------------------------------------------------------------------------------------------------|'

echo 'Pre-pulling image to keep logs clean'
docker pull vitess/vttestserver:v20.0.3-mysql80
echo ' '

echo "Starting ${version_20_container_name}"
container_id=$(docker run --detach --name="${version_20_container_name}" "${docker_run_args[@]}" vitess/vttestserver:v20.0.3-mysql80)
echo 'Waiting for container to initialize...'
sleep 20s

ip_addr=$(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "${version_20_container_name}")
echo "${version_20_container_name} IP: ${ip_addr}"

echo "Running netcat for MySQL port"
nc -zv "${ip_addr}" "${mysql_port}" || true
echo "Running netcat for gRPC port"
nc -zv "${ip_addr}" "${grpc_port}" || true

echo -e "|-------------------------------------------------------------------------------------------------------------------------|\n\n"

# Version 21.0.0
echo '|------ Version 21.0.0 ---------------------------------------------------------------------------------------------------|'

echo 'Pre-pulling image to keep logs clean'
docker pull vitess/vttestserver:v21.0.0-mysql80
echo ' '

echo "Starting ${version_21_container_name}"
container_id=$(docker run --detach --name="${version_21_container_name}" "${docker_run_args[@]}" vitess/vttestserver:v21.0.0-mysql80)
echo 'Waiting for container to initialize...'
sleep 20s

ip_addr=$(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "${version_21_container_name}")
echo "${version_21_container_name} IP: ${ip_addr}"

echo "Running netcat for MySQL port"
nc -zv "${ip_addr}" "${mysql_port}" || true
echo "Running netcat for gRPC port"
nc -zv "${ip_addr}" "${grpc_port}" || true

echo -e "|-------------------------------------------------------------------------------------------------------------------------|\n\n"

# Latest
echo '|---------- Latest -------------------------------------------------------------------------------------------------------|'

echo 'Pre-pulling image to keep logs clean'
docker pull vitess/vttestserver:mysql80
echo ' '

echo "Starting ${latest_container_name}"
container_id=$(docker run --detach --name="${latest_container_name}" "${docker_run_args[@]}" vitess/vttestserver:mysql80)
echo 'Waiting for container to initialize...'
sleep 20s

ip_addr=$(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "${latest_container_name}")
echo "${latest_container_name} IP: ${ip_addr}"

echo "Running netcat for MySQL port"
nc -zv "${ip_addr}" "${mysql_port}" || true
echo "Running netcat for gRPC port"
nc -zv "${ip_addr}" "${grpc_port}" || true

echo '|-------------------------------------------------------------------------------------------------------------------------|'

Binary Version

vttestserver-latest

Operating System and Environment details

n/a

Log Fragments

|------ Version 20.0.3 ---------------------------------------------------------------------------------------------------|
Pre-pulling image to keep logs clean
v20.0.3-mysql80: Pulling from vitess/vttestserver
6dce3b49cfe6: Pull complete 
c5980ff74b97: Pull complete 
6439b78e07bb: Pull complete 
634d6d66fbba: Pull complete 
09124c96e3c3: Pull complete 
7b6686cf96ac: Pull complete 
decd25bf86f6: Pull complete 
20936e025e2d: Pull complete 
Digest: sha256:683a99d2a3bbe7751acf7ac268055426306f1261c084152eade175319eff730e
Status: Downloaded newer image for vitess/vttestserver:v20.0.3-mysql80
docker.io/vitess/vttestserver:v20.0.3-mysql80
 
Starting vttestserver-20.0.3
Waiting for container to initialize...
vttestserver-20.0.3 IP: 172.17.0.2
Running netcat for MySQL port
Connection to 172.17.0.2 33577 port [tcp/*] succeeded!
Running netcat for gRPC port
Connection to 172.17.0.2 33575 port [tcp/*] succeeded!
|-------------------------------------------------------------------------------------------------------------------------|


|------ Version 21.0.0 ---------------------------------------------------------------------------------------------------|
Pre-pulling image to keep logs clean
v21.0.0-mysql80: Pulling from vitess/vttestserver
6dce3b49cfe6: Already exists 
1ea862120675: Pull complete 
7752a6b7b984: Pull complete 
0d1dcb0a093e: Pull complete 
c41aa64d82d6: Pull complete 
6484229778f0: Pull complete 
c60fbe59811e: Pull complete 
f7690de602ed: Pull complete 
Digest: sha256:ce595dba0a37695a0ead60110d91937c01fa064e04df95be6acf449d9aa01e45
Status: Downloaded newer image for vitess/vttestserver:v21.0.0-mysql80
docker.io/vitess/vttestserver:v21.0.0-mysql80
 
Starting vttestserver-21.0.0
Waiting for container to initialize...
vttestserver-21.0.0 IP: 172.17.0.3
Running netcat for MySQL port
Connection to 172.17.0.3 33577 port [tcp/*] succeeded!
Running netcat for gRPC port
Connection to 172.17.0.3 33575 port [tcp/*] succeeded!
|-------------------------------------------------------------------------------------------------------------------------|


|---------- Latest -------------------------------------------------------------------------------------------------------|
Pre-pulling image to keep logs clean
mysql80: Pulling from vitess/vttestserver
69fb10dc82f9: Pull complete 
9530dbc786c9: Pull complete 
4723cf4cfe1f: Pull complete 
d1c733e95d1a: Pull complete 
e47c9887790a: Pull complete 
6a29be3204ac: Pull complete 
da6352e8c108: Pull complete 
f20a407ca0dd: Pull complete 
Digest: sha256:60dbca30eb144afbe83fe8a97bf290dc966e830f3d109402b4a271c3ab90ae63
Status: Downloaded newer image for vitess/vttestserver:mysql80
docker.io/vitess/vttestserver:mysql80
 
Starting vttestserver-latest
Waiting for container to initialize...
vttestserver-latest IP: 172.17.0.4
Running netcat for MySQL port
Connection to 172.17.0.4 33577 port [tcp/*] succeeded!
Running netcat for gRPC port
nc: connect to 172.17.0.4 port 33575 (tcp) failed: Connection refused
|-------------------------------------------------------------------------------------------------------------------------|