docker-compose service deployer - Provide a volume to share data with elastic-agent

[andrewkroh]

On Linux, bind mounting the host-based SERVICE_LOGS_DIR into a container can cause permissions problems for elastic-package when the service writes files as root. You can get failures like:

Error: error running package system tests: could not complete test run: could not setup service: removing service logs failed: readdir failed (path: /opt/buildkite-agent/.elastic-package/tmp/service_logs): open /opt/buildkite-agent/.elastic-package/tmp/service_logs: permission denied

Instead of using a host bind-mount, elastic-package could create volume and mount it inside the elastic-agent container. Then when a service want to share logs with the elastic-agent, it can mount the volume instead of SERVICE_LOGS_DIR. This would look like

# ./profiles/default/stack/docker-compose.yml
services:
  elastic-agent:
    volumes:
      - elastic_agent_volume:/elastic_agent_volume

volumes:
  elastic_agent_volume:
    # Prevent docker-compose from prefixing the volume name.
    name: elastic_agent_volume

Then in the integration deployment it refers to an external volume.

# _dev/deploy/docker/docker-compose.yml
version: '3.0'
services:
  gitlab:
    image: gitlab/gitlab-ce:${SERVICE_VERSION}
    volumes:
      - elastic_agent_volume:/var/log/gitlab/gitlab-rails

volumes:
  elastic_agent_volume:
    # To indicate that docker-compose should not create this volume.
    external: true

Related

• Add check about permissions folder for tmp/service_logs #1341

[jsoriano]

For the transparent migration of the docker-compose.yml files that use SERVICE_LOGS_DIR we could append the new volume definition, and set SERVICE_LOGS_DIR to elastic_agent_volume.