Problem: SSH's StrictHostKeyChecking is blocking SSH connections and task runs

[northwestnodes-eric]

Issue

ISSUE
On Ubuntu 24.04, when starting a new Semaphore installation (binary install) it seems that the host's StrictHostKeyChecking setting (which is enabled by default) is preventing any new host keys from getting added to the known_hosts repository. The Strict Host Key Checking feature forces the user to confirm the addition of a host key to the known_hosts repository.

However, obviously Semaphore runs SSH in a non-interactive manner and thus a confirmation will never come, resulting in a stuck task run:

<10.32.0.4> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="USERNAME"' -o ConnectTimeout=10 -o 'ControlPath="/tmp/semaphore/.ansible/cp/86ba7f68f3"' HOSTNAME '/bin/sh -c '"'"'echo ~USERNAME && sleep 0'"'"''

... will halt the task execution because SSH is awaiting for confirmation to add the host key to its repository due to strict host key checking.

On the target host side, stopping the task template run will result in a [preauth] disconnect message, confirming this behavior.

WORK-AROUND
A work-around for this show stopper is to force SSH to accept fingerprints by editing /etc/ssh/ssh_config and setting StrictHostKeyChecking no to disable strict host key checking system-wide. When I changed this on my system, all went well and new host keys would be accepted without a problem.

PROPOSED SOLUTION
However, a better solution is for Semaphore to run a proper SSH command and add the -o StrictHostKeyChecking=accept-new option. This would accept a new host key without having to change the underlying system settings.

Impact

Service (scheduled tasks, alerts), Ansible (task execution)

Installation method

Binary

Database

Postgres

Browser

Chrome

Semaphore Version

2.10.35-a71a3c5-1730103563

Ansible Version

ansible [core 2.16.3]
  config file = None
  configured module search path = ['/home/azureuser/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3/dist-packages/ansible
  ansible collection location = /home/azureuser/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.12.3 (main, Sep 11 2024, 14:17:37) [GCC 13.2.0] (/usr/bin/python3)
  jinja version = 3.1.2
  libyaml = True

### Logs & errors

_No response_

### Manual installation - system information

Linux HOSTNAME 6.8.0-1017-azure #20-Ubuntu SMP Tue Oct 22 03:43:13 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

### Configuration

_No response_

### Additional information

_No response_