Multiple containers on a single machine.

[kczpl]

Hi!
I switched a couple of my projects to Kamal. In some of them, especially in staging environments, I use multiple containers on one machine.
Usually, we work with separate repositories for the frontend (FE) and backend (BE). On both, I use Kamal to deploy containers. Most of these applications are behind a load balancer, which also handles SSL termination.

               -> App server 1 (BE+FE containers)
Load Balancer  -> App server 2 (BE+FE containers)
               -> App server 3 (BE+FE containers)

The issue is that I experienced weird behavior during deployments. I postponed creating this issue because I couldn't find a common reason for it. Generally, sometimes deployments stop working, showing that the health check doesn't pass. When I stop the container and rerun the deployment in my CI/CD pipeline, it works again.

The error looks like this:

 ERROR (SSHKit::Command::Failed): Exception while executing on host <domain of random App Server>: docker exit status: 1
docker stdout: Nothing written
docker stderr: Error: target failed to become healthy

Additionally, I ran the deployment in verbose mode, and none of the containers returned a status of "unhealthy."

In my humble opinion, deploying multiple containers on one machine is a common use case. As I've investigated this issue for a while, I can say that the load balancer layer works just fine, and the containers are healthy. I assume the issue lies somewhere in Kamal's proxy and the way Kamal handles health checks.

I would love some hints or advice, or maybe there's something I’m doing wrong when defining health checks. Perhaps someone has successfully run such an architecture and can share the solution.

---

Those are my configs:

Backend app:

service: app_backend_app
image: app_backend

builder:
  arch: amd64
  dockerfile: Dockerfile

servers:
  web:
    hosts:
      - api1.internal.domain.com
      - api2.internal.domain.com
  sidekiq:
    cmd: bundle exec sidekiq
    hosts:
      - api1.internal.domain.com
      - api2.internal.domain.com

registry:
  server: my-selfhosted.registry.com
  username:
    - CI_REGISTRY_USER
  password:
    - CI_REGISTRY_PASSWORD

env:
  secret:
    - RAILS_MASTER_KEY
    - RAILS_ENV
    - SOME_ENVS

ssh:
  user: myuser

proxy:
  healthcheck:
    path: /health # this endpoints checks Redis and Postgres connections and returns 200 ("OK" message) if everything is ok
    interval: 2 
    timeout: 30
  host: api.domain.com,api1.internal.domain.com,api2.internal.domain.com
  app_port: 3000
  ssl: false
  forward_headers: true
  response_timeout: 30

Frontend app:

service: ap_frontend_app
image: app_frontend_backend

builder:
  arch: amd64
  dockerfile: Dockerfile

servers:
  web:
    hosts:
      - api1.internal.domain.com
      - api2.internal.domain.com

registry:
  server: my-selfhosted.registry.com
  username:
    - CI_REGISTRY_USER
  password:
    - CI_REGISTRY_PASSWORD

ssh:
  user: myuser

proxy:
  healthcheck:
    path: /health # this endpoints returns 200 OK from NGINX container
    interval: 2 
    timeout: 30
  host: app.domain.com,app1.internal.domain.com,app2.internal.domain.com
  app_port: 3000
  ssl: false
  forward_headers: true
  response_timeout: 30

I deploy containers from CI/CD pipeline using:

    - kamal deploy --skip-push --version=$CI_COMMIT_REF_SLUG -d staging -v

Thats how I build containers in my Gitlab CI, maybe it is important in that case:

build:
  stage: build
  image: docker:25-dind
  script:
    - |
      if [ "$CI_COMMIT_BRANCH" == "master" ]; then
          DOCKERFILE=Dockerfile.production
          LABEL=$PRODUCTION_LABEL # for kamal
      elif [ "$CI_COMMIT_BRANCH" == "staging" ]; then
          DOCKERFILE=Dockerfile.staging
          LABEL=$STAGING_LABEL # for kamal 
      else
          echo "Unknown branch"
          exit 1
      fi
    - echo "$CI_JOB_TOKEN" | docker login -u $CI_REGISTRY_USER --password-stdin $CI_REGISTRY
    - docker pull "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG" || true
    - |
      docker build --cache-from "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG" \
        -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG" \
        --label "service=$LABEL" \ # Kamal label
        -f $DOCKERFILE .
    - docker push "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG"