WIP: Hrfilter

[zealot128]

a lot of things....

Test addons:

• made the "site" as a build matrix variable.
• added a check.sh for each build "site", e.g. checking if web server is reachable, for hrfilter's case, if sidekiq is running, too.

Role fixes to support hrfilter role natively:

• made nginx/server + nginx/passenger compatible (in our case we ran a fork of the passenger role from Nginx passenger updates #68, that is not longer necessary)
  • nginx modules-enabled stuff came after 17.04 or so, that include was missing in the nginx/server config.
  • option, to disable nginx ppa, as that might collide with the passenger module (see also Nginx passenger updates #68)
• sidekiq systemctl version did not actually work, as a config was missing. I've had fixes in a fork that I've integrated here.

Added HRfilter site test:

• used "newish" import_role vs. task/pre_task/roles build up
• passenger + sidekiq is tested here
• Ubuntu 18

Let's see what the test results brings back

[zealot128]

Uff.... @arBmind I think I give up for now. Already burned a couple of hours. Maybe someone with more init/docker/systemd knowledge can create a container that works with started network services.

Having to run systemd service inside Docker seems to be not ...easy....

What I've tried:

• various default (ubuntu) images with systemd -> Systemd complains that it is not run as init PID1 every single time with different permutations of Dockerfile/Docker Run (privileged, mounting of cgroups etc)
• Docker replacement script seems to be a canonical way https://github.com/gdraheim/docker-systemctl-images ... which ...kind of works, but in the end does not start any daemons cause the port binding seems to be failing??

e.g. current playbook just apt: redis start redis, but then:
Creating Server TCP listening socket ::1:6379: bind: Cannot assign requested address

If google looks like that, that means I should take a break :D

[arBmind]

@zealot128 did you take a look @strfry changes in pr #77 ? - I guess he ran into the same issues and solved it.

[zealot128]

Thx for the tip; Just tried that with exact same run_opts / image, but Ubuntu 18.04:

docker: Error response from daemon: oci runtime error: container_linux.go:247: 
  starting container process caused "exec: \"/sbin/init\": stat /sbin/init: no such file or directory".

Using the given Ubuntu 16.04 Dockerfile and the run_opts, I get this failure from the first ansible task that uses Service Module, or if I attach via docker exec and using service/systemctl:

fatal: [localhost]: FAILED! => {"changed": false, "cmd": "/bin/systemctl", "msg": "Failed to connect to bus: No such file or directory", "rc": 1, 
"stderr": "Failed to connect to bus: No such file or directory\n", "stderr_lines": ["Failed to connect to bus: No such file or directory"], "stdout": "", "stdout_lines": []}

root@48a7a5c17f88:/# service redis-server status
Failed to connect to bus: No such file or directory
Failed to connect to bus: No such file or directory

---

Full test script for reference (Unchanged Ubuntu 16.04 Dockerfile)

distribution='ubuntu'
version='16.04'  
init="/sbin/init"           
site="hrfilter"                             
run_opts="--privileged --volume=/sys/fs/cgroup:/sys/fs/cgroup:ro --volume=/sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd:rw"

docker build --rm=true --file=tests/Dockerfile.${distribution}-${version} \
  --tag=${distribution}-${version}:ansible tests
container_id=$(mktemp)
echo "Starting container..." 

docker run --detach --ulimit nofile=1024 \
  --volume=${PWD}:/etc/ansible/roles/dresden-weekly.Rails:ro ${run_opts} \
  ${distribution}-${version}:ansible ${init} > ${container_id}

echo "Running tests
 you can enter the container like this:

docker exec -it $(cat ${container_id}) bash
"

docker exec -e ANSIBLE_FORCE_COLOR=1 "$(cat ${container_id})" env TERM=xterm \
   ansible-playbook /etc/ansible/roles/dresden-weekly.Rails/tests/${site}/playbook.yml

Playbook will e.g. crash on PostgreSQL or any other thing with native ansible "service" (like apt: redis-server; service: name=redis-server state=started)

[arBmind]

I got it somewhat working:

• travisci/ubuntu-systemd:18.04
• add init and some stuff required for Ansible (iproute2 for ip setup)
• PostgreSql has bug that prevents it to start with the regular utf8 locales (added something to work around the issue)

See: https://travis-ci.org/arBmind/ansible-rails/builds/528273312
Right now it gets stuck by the general slowness of docker on Travis.

• The rvm keys may fail to be downloaded (already tried to fix this with ipv4)
• The ruby download or compilation might get stuck
• The bundle installation might get stuck

I already added a script to prevent the Travis 10 minute timeout. But this wont help to debug where Ansible got stuck and requires a manual cancel if we don't want to be banned by Travis :)
So I never reached the end of the installation, but the major hurdles seem to work.

[zealot128]

Great work!

that brings me further... I've noticed that I've also maybe ran in circles. For brevity, I only tried to install redis and run it with the default service module. After trying your docker file & run_opts, redis still failed, but the reason seems to be ipv6 related redis/redis#5055

Now, after removing ipv6 binding the script will run further. Let's check... :)

[zealot128]

regarding compile time:

# from within container
$ rvm install 2.6.0
Searching for binary rubies, this might take some time.
Found remote file https://rvm_io.global.ssl.fastly.net/binaries/ubuntu/18.04/x86_64/ruby-2.6.0.tar.bz2

Libraries missing for ruby-2.6.0: /home/deploy/.rvm/rubies/ruby-2.6.0/lib/ruby/2.6.0/x86_64-linux/openssl.so:. Refer to your system manual for installing libraries
Mounting remote ruby failed with status 10, trying to compile.

Thus, will compile from source. With openssl problems, maybe uses rvm pkg get openssl and compiles openssl from source, too... no wonder \o/

Tried various variations of different ubuntu libs, but none worked. seems to be some issues in RVM, too rvm/rvm#4616