Logs state that 'sudo: no password was provided' when previous entries states that it did

[umamibeef]

Hi everyone, need some help with an issue I'm seeing with a particular host. This host is remotely power cycled and I am using ParallelSSH to probe its status and execute commands on it. This host is running Ubuntu 22.04.4 LTS (GNU/Linux 5.17.11spm x86_64) and its openssl version is OpenSSL 3.0.2 15 Mar 2022 (Library: OpenSSL 3.0.2 15 Mar 2022). My Python script is running in a Docker container from the "ubuntu:22.04" base image with the same version of openssl (OpenSSL 3.0.2 15 Mar 2022 (Library: OpenSSL 3.0.2 15 Mar 2022)).

Note: the reason I'm using the close_channel() function below is because through trial and error (specifically with this host), it has allowed me to get deterministic behaviour out of it, especially during power cycles where connection is disrupted and strange behaviours are observed. As a side question, I would like to know what the proper way of "cleaning up" the connection in cases like these, where the socket is expected to get forcibly disconnected.

In the current implementation, a host is power cycled and it is polled to determine if it's ready to accept SSH commands. Here is that code:

client = SSHClient(self.ssh_ip, user=self.ssh_username, password=self.ssh_password, port=self.ssh_port, timeout=3, num_retries=1)
output = client.run_command("whoami")
if output.channel:
    client.close_channel(output.channel)

This is looped until we receive a response and works as intended:

...
DEBUG:pssh.clients.base.single:Connecting to 10.10.8.191:22                                                                                                                 
ERROR:pssh.clients.base.single:Error connecting to host '10.10.8.191:22' - retry 1/1                                                                                        
[DEBUG][HostManager] No response from host                                                                                             
[DEBUG][HostManager] Will try again in 30 seconds...                                                                                   
[INFO ][HostManager] Elapsed time: 3m 54s                                                                                              
DEBUG:pssh.clients.base.single:Connecting to 10.10.8.191:22                                                                                                                 
ERROR:pssh.clients.base.single:Error connecting to host '10.10.8.191:22' - retry 1/1                                                                                        
[DEBUG][HostManager] No response from host                                                                                             
[DEBUG][HostManager] Will try again in 30 seconds...                                                                                   
[INFO ][HostManager] Elapsed time: 4m 27s                                                                                              
DEBUG:pssh.clients.base.single:Connecting to 10.10.8.191:22                                                                                                                 
DEBUG:pssh.clients.base.single:Agent auth failed with AgentConnectionError('Unable to connect to agent') continuing with other authentication methods                       
DEBUG:pssh.clients.base.single:Private key auth failed, trying password                                                                                                     
DEBUG:pssh.clients.base.single:Authentication completed successfully - setting session to non-blocking mode                                                                 
DEBUG:pssh.clients.native.single:Executing command 'b'whoami''                                                                                                              
DEBUG:pssh.clients.native.single:Closing channel                                                                                                                            
[INFO ][HostManager] Was able to successfully ssh into 10.10.8.191:22 in 4m 58s
...

However, subsequent commands are failing. And logs are revealing that no password has been provided.

Here's the next chunk of code dispatching commands after the server poll:

@dataclass
class HostManagerResult:
    """
    Datatype that encapsulates the command and result (stdout, stderr, return code) of a command executed by HostManager.
    """
    command: str
    stdout: list[str]
    stderr: list[str]
    return_code: int

client = SSHClient(self.ssh_ip, user=self.ssh_username, password=self.ssh_password, port=self.ssh_port, timeout=60, num_retries=1)
output = client.run_command(command=command, sudo=True, read_timeout=60)
if output.channel:
    client.close_channel(output.channel)

...

results.append(HostManagerResult(command, [*output.stdout], [*output.stderr], output.exit_code))

Here's the resulting log:

...
[DEBUG][HostManager] [ssh @ 10.10.8.191:22]                                                                                            
[DEBUG][HostManager]   +-[stdin  >>]                                                                                                   
[DEBUG][HostManager]   |                                                                                                               
[DEBUG][HostManager]   |  sudo -S $SHELL -c 'docker images -q payload'                                                          
[DEBUG][HostManager]   |                                                                                                               
DEBUG:pssh.clients.base.single:Connecting to 10.10.8.191:22                                                                                                                 
DEBUG:pssh.clients.base.single:Agent auth failed with AgentConnectionError('Unable to connect to agent') continuing with other authentication methods                       
DEBUG:pssh.clients.base.single:Private key auth failed, trying password                                                                                                     
DEBUG:pssh.clients.base.single:Authentication completed successfully - setting session to non-blocking mode                                                                 
DEBUG:pssh.clients.native.single:Executing command 'b"sudo -S $SHELL -c 'docker images -q payload'"'
DEBUG:pssh.clients.native.single:Closing channel
DEBUG:pssh.clients.base.single:Reading from stdout buffer, timeout=60
DEBUG:pssh.clients.base.single:Reading from stderr buffer, timeout=60
INFO:pssh.host_logger:[10.10.8.191]     [err]   [sudo] password for user: 
INFO:pssh.host_logger:[10.10.8.191]     [err]   sudo: no password was provided
INFO:pssh.host_logger:[10.10.8.191]     [err]   sudo: a password is required
DEBUG:pssh.clients.base.single:Reading from stdout buffer, timeout=60
DEBUG:pssh.clients.base.single:Reading from stderr buffer, timeout=60
...

I have confirmed that I am passing it the same credentials. Is there something I'm doing that is preventing the password from being provided?

[umamibeef]

Adding client.disconnect() after closing the channel appears to have resolved my issues.

It would be nice to have this documented somewhere (setup and teardown of the single native client).