Add support for remote execution with ssh, similar to ansible

[xmnlab]

Feature Request: Add SSH Support with Paramiko Backend

Overview

Makim aims to enhance local automation workflows by simplifying script execution, environment management, and task dependencies. Building upon its reference to Ansible, Makim seeks to introduce robust SSH support to facilitate remote task execution. This feature leverages Paramiko as the backend for SSH connections, providing a flexible and secure method for interacting with remote hosts.

Proposed Configuration Structure

To integrate SSH support, the .makim.yaml configuration will be extended with a hosts section defined in the global scope. Tasks or groups can then specify a remote attribute to target the desired host. Below is the proposed structure:

version: 1.0

hosts:
  myserver1:
    host: "{{ env.STAGING_HOST }}"
    user: "{{ env.SSH_USER }}"
    file: "{{ env.SSH_PRIVATE_KEY }}"
    protocol: ssh
    port: 22
    allow_agent: true
    look_for_keys: true
    timeout: 30
    keepalive: 10
    password: "{{ env.SSH_PASSWORD }}"  # Optional, if password-based authentication is required

groups:
  deploy:
    tasks:
      git-pull:
        remote: myserver1
        run: |
          git pull

Configuration Details

• Global hosts Section:

  • host: The hostname or IP address of the remote server. Supports Jinja2 templating for dynamic configuration via environment variables.
  • user: The SSH username for authentication.
  • file: Path to the SSH private key file for key-based authentication.
  • protocol: Connection protocol, currently set to ssh.
  • port (optional): SSH port number (default is 22).
  • allow_agent (optional): Boolean to allow SSH agent forwarding (true or false).
  • look_for_keys (optional): Boolean to enable searching for SSH keys in standard locations (true or false).
  • timeout (optional): Connection timeout in seconds.
  • keepalive (optional): Interval in seconds for sending keepalive messages.
  • password (optional): Password for SSH authentication if key-based authentication is not used.

• Group or Task remote Attribute:

  • remote: References a host defined in the global hosts section. This attribute directs the task to execute on the specified remote host.

Additional SSH Configuration Options

To provide comprehensive SSH connectivity, the following additional parameters are recommended:

• password: Allows password-based SSH authentication.
• allow_agent: Enables or disables SSH agent forwarding.
• look_for_keys: Determines whether to search for SSH keys in standard locations.
• timeout: Sets the SSH connection timeout duration.
• keepalive: Configures the interval for sending keepalive messages to maintain the connection.
• compress (optional): Enables compression for SSH data transfer.
• proxy_command (optional): Defines a proxy command for SSH connections requiring a proxy.
• ssh_config (optional): Path to a custom SSH configuration file for advanced SSH settings.

These options align with Paramiko's SSH client capabilities, ensuring compatibility and flexibility for various SSH authentication and connection scenarios.

Rationale

• Flexibility: By defining SSH connections in the global scope and referencing them within tasks or groups, users can easily manage multiple remote hosts with varying configurations.
• Security: Leveraging Paramiko ensures secure SSH connections with support for key-based and password-based authentication, agent forwarding, and other security features.
• Scalability: This configuration structure supports scalability, allowing users to define numerous hosts and manage complex deployment workflows efficiently.
• Configurability: The inclusion of additional SSH parameters provides users with the necessary tools to tailor SSH connections to their specific environments and requirements.

Implementation Considerations

• Paramiko Integration: Ensure that Makim integrates seamlessly with Paramiko, handling SSH connections, authentication, and command execution reliably.
• Error Handling: Implement robust error handling for SSH-related issues, such as connection failures, authentication errors, and command execution problems.
• Documentation: Update Makim's documentation to include detailed instructions and examples for configuring SSH connections, covering all supported parameters.
• Testing: Develop comprehensive tests to validate SSH connectivity, authentication methods, and task execution across different SSH configurations.

Example Usage

.makim.yaml Configuration:

version: 1.0

hosts:
  staging_server:
    host: "{{ env.STAGING_HOST }}"
    user: "{{ env.SSH_USER }}"
    file: "{{ env.SSH_PRIVATE_KEY }}"
    protocol: ssh
    port: 2222
    allow_agent: false
    look_for_keys: true
    timeout: 30
    keepalive: 10
    password: "{{ env.SSH_PASSWORD }}"  # Optional

groups:
  deploy:
    tasks:
      git-pull:
        remote: staging_server
        run: |
          git pull

Executing the Task:

makim deploy.git-pull

This command connects to the staging_server using the specified SSH configurations and executes the git pull command remotely.

Conclusion

Integrating SSH support with Paramiko backend into Makim will significantly enhance its automation capabilities, enabling secure and flexible remote task execution. The proposed configuration structure and additional SSH options aim to provide users with a robust framework for managing remote operations efficiently.

[xmnlab]

it seems that ansible uses cryptography for the ssh connections.
maybe we could use https://paramiko.org/ which relies on cryptography
paramiko is the base foundation used by Fabric, which is a popular high-level SSH library/tool

[xmnlab]

https://docs.ansible.com/ansible/latest/inventory_guide/connection_details.html#controlpersist-and-paramiko

[xmnlab]

for now we will just support ssh remote host, if another protocol is given, it should fail.

[xmnlab]

@abhijeetSaroha , I just updated the first comment here with all the details.
let me know if you have any question

[xmnlab]

from gpt, suggestion for the implementation:

Integrating remote execution capabilities into Makim using Paramiko is a valuable enhancement that can significantly broaden the tool's applicability. Your initial approach to segregate local and remote execution by introducing separate functions (_call_shell_app_local and _call_shell_remote) is a sound strategy. This separation adheres to the Single Responsibility Principle, promoting cleaner and more maintainable code. Below are detailed recommendations and considerations to guide your implementation:

1. Structuring Remote and Local Execution

a. Separate Execution Functions

Pros:

• Clarity: Distinguishes clearly between local and remote execution paths.
• Maintainability: Easier to manage and update each execution type independently.
• Extensibility: Facilitates adding more execution types in the future (e.g., containerized execution).

Implementation:

def _call_shell_app_local(self, cmd: str) -> None:
    """Execute command locally using the specified shell application."""
    # Existing implementation of local execution
    self._load_shell_app()
    # ... rest of the local execution code ...

def _call_shell_remote(self, cmd: str, host_config: dict) -> None:
    """Execute command remotely using Paramiko."""
    import paramiko

    try:
        # Initialize SSH client
        ssh = paramiko.SSHClient()
        ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
        
        # Establish SSH connection
        ssh.connect(
            hostname=host_config['host'],
            port=host_config.get('port', 22),
            username=host_config['user'],
            key_filename=host_config['file'],
            timeout=host_config.get('timeout', 30),
            allow_agent=host_config.get('allow_agent', True),
            look_for_keys=host_config.get('look_for_keys', True),
        )
        
        # Execute the command
        stdin, stdout, stderr = ssh.exec_command(cmd)
        
        # Optionally, handle output
        output = stdout.read().decode('utf-8')
        error = stderr.read().decode('utf-8')
        
        if self.verbose:
            MakimLogs.print_info(output)
            if error:
                MakimLogs.print_error(error)
        
        # Close the connection
        ssh.close()
        
    except paramiko.AuthenticationException:
        MakimLogs.raise_error(
            "Authentication failed when connecting to {}".format(host_config['host']),
            MakimError.SSH_AUTHENTICATION_FAILED,
        )
    except paramiko.SSHException as sshException:
        MakimLogs.raise_error(
            "Unable to establish SSH connection: {}".format(sshException),
            MakimError.SSH_CONNECTION_ERROR,
        )
    except Exception as e:
        MakimLogs.raise_error(
            "Error occurred during SSH execution: {}".format(e),
            MakimError.SSH_EXECUTION_ERROR,
        )

b. Dispatcher Function

Modify _call_shell_app to act as a dispatcher that determines whether to execute locally or remotely based on the task configuration.

def _call_shell_app(self, cmd: str) -> None:
    """Dispatch command execution to local or remote."""
    remote_name = self.task_data.get('remote')
    if remote_name:
        host_config = self.global_data.get('hosts', {}).get(remote_name)
        if not host_config:
            MakimLogs.raise_error(
                f"Remote host '{remote_name}' configuration not found.",
                MakimError.REMOTE_HOST_NOT_FOUND,
            )
        self._call_shell_remote(cmd, host_config)
    else:
        self._call_shell_app_local(cmd)

2. Enhancing SSH Configuration

To make SSH configuration more robust and user-friendly, consider including the following additional parameters that are supported by Paramiko:

• port: SSH port number (default is 22).
• password: Password for SSH authentication (optional, if key-based auth is not used).
• allow_agent: Boolean to allow SSH agent forwarding (true or false).
• look_for_keys: Boolean to enable searching for SSH keys in standard locations (true or false).
• timeout: Connection timeout in seconds.
• keepalive: Interval in seconds for sending keepalive messages.
• compress: Boolean to enable compression.
• proxy_command: Command string for proxying SSH connections.
• ssh_config: Path to a custom SSH configuration file.

Updated Configuration Example

version: 1.0

hosts:
  staging_server:
    host: "{{ env.STAGING_HOST }}"
    user: "{{ env.SSH_USER }}"
    file: "{{ env.SSH_PRIVATE_KEY }}"
    protocol: ssh
    port: 22
    allow_agent: true
    look_for_keys: true
    timeout: 30
    keepalive: 10
    password: "{{ env.SSH_PASSWORD }}"  # Optional
    compress: true
    proxy_command: "ssh -W %h:%p gateway.example.com"
    ssh_config: "/path/to/custom_ssh_config"

groups:
  deploy:
    tasks:
      git-pull:
        remote: staging_server
        run: |
          git pull

3. Environment Variable and Security Considerations

• Sensitive Data Management: Avoid hardcoding sensitive information like passwords. Use environment variables or secure storage solutions to inject them at runtime.

• SSH Agent Forwarding: Ensure that allow_agent is set judiciously to prevent potential security risks.

• Host Key Verification: While AutoAddPolicy is convenient, it can pose security risks. Consider implementing a stricter host key verification strategy in production environments.

4. Error Handling and Logging

Enhance error handling to provide more granular feedback and facilitate easier debugging:

• Connection Errors: Differentiate between various SSH errors (e.g., authentication failure, connection timeout) and provide descriptive error messages.

• Command Execution Errors: Capture and log standard error output to aid in diagnosing issues with remote commands.

Enhanced Error Handling Example

except paramiko.AuthenticationException:
    MakimLogs.raise_error(
        f"Authentication failed when connecting to {host_config['host']}.",
        MakimError.SSH_AUTHENTICATION_FAILED,
    )
except paramiko.SSHException as sshException:
    MakimLogs.raise_error(
        f"SSH connection error with {host_config['host']}: {sshException}",
        MakimError.SSH_CONNECTION_ERROR,
    )
except Exception as e:
    MakimLogs.raise_error(
        f"Unexpected error during SSH execution on {host_config['host']}: {e}",
        MakimError.SSH_EXECUTION_ERROR,
    )

5. Reusability and Connection Pooling

Establishing an SSH connection for every task can be inefficient, especially when executing multiple tasks on the same host. Implementing connection pooling or reusing existing connections can optimize performance.

Implementing Connection Reuse

Consider maintaining a persistent SSH connection within the Makim class:

class Makim:
    def __init__(self):
        # Existing initialization
        self.ssh_clients = {}

    def _get_ssh_client(self, host_config: dict) -> paramiko.SSHClient:
        host_identifier = host_config['host']
        if host_identifier not in self.ssh_clients:
            ssh = paramiko.SSHClient()
            ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
            ssh.connect(
                hostname=host_config['host'],
                port=host_config.get('port', 22),
                username=host_config['user'],
                key_filename=host_config['file'],
                password=host_config.get('password'),
                timeout=host_config.get('timeout', 30),
                allow_agent=host_config.get('allow_agent', True),
                look_for_keys=host_config.get('look_for_keys', True),
            )
            self.ssh_clients[host_identifier] = ssh
        return self.ssh_clients[host_identifier]

    def _call_shell_remote(self, cmd: str, host_config: dict) -> None:
        try:
            ssh = self._get_ssh_client(host_config)
            stdin, stdout, stderr = ssh.exec_command(cmd)
            output = stdout.read().decode('utf-8')
            error = stderr.read().decode('utf-8')
            
            if self.verbose:
                MakimLogs.print_info(output)
                if error:
                    MakimLogs.print_error(error)
        except paramiko.AuthenticationException:
            # Handle exceptions as before
            pass
        # ... other exception handling ...

Pros:

• Performance: Reduces the overhead of establishing new SSH connections for each task.
• Efficiency: Useful when multiple tasks target the same host consecutively.

Cons:

• Resource Management: Requires careful handling to close connections appropriately, especially when the program terminates or encounters errors.

6. Task Configuration and Flexibility

Ensure that the task configuration remains flexible, allowing users to specify whether a task should be executed locally or remotely. This can be achieved by making the remote attribute optional.

Example Configuration with Optional Remote Execution

version: 1.0

hosts:
  staging_server:
    host: "{{ env.STAGING_HOST }}"
    user: "{{ env.SSH_USER }}"
    file: "{{ env.SSH_PRIVATE_KEY }}"
    protocol: ssh
    port: 22
    allow_agent: true
    look_for_keys: true
    timeout: 30
    keepalive: 10
    password: "{{ env.SSH_PASSWORD }}"  # Optional

groups:
  deploy:
    tasks:
      git-pull:
        remote: staging_server  # Optional: if omitted, executes locally
        run: |
          git pull

      local-task:
        run: |
          echo "This runs locally."

Dispatcher Adjustment

Update the dispatcher to handle tasks without the remote attribute gracefully:

def _call_shell_app(self, cmd: str) -> None:
    """Dispatch command execution to local or remote."""
    remote_name = self.task_data.get('remote')
    if remote_name:
        host_config = self.global_data.get('hosts', {}).get(remote_name)
        if not host_config:
            MakimLogs.raise_error(
                f"Remote host '{remote_name}' configuration not found.",
                MakimError.REMOTE_HOST_NOT_FOUND,
            )
        self._call_shell_remote(cmd, host_config)
    else:
        self._call_shell_app_local(cmd)

7. Documentation and User Guidance

Provide comprehensive documentation to assist users in configuring and utilizing the new SSH features effectively:

• Configuration Examples: Include various configuration snippets demonstrating different SSH setups (key-based, password-based, proxy commands).

• Usage Instructions: Clearly explain how to define remote tasks and the implications of different SSH settings.

• Troubleshooting Guide: Offer solutions for common SSH-related issues, such as authentication failures or connection timeouts.

8. Testing and Validation

Implement thorough testing to ensure the reliability and security of the SSH integration:

• Unit Tests: Mock SSH connections to test the execution flow without requiring actual SSH servers.

• Integration Tests: Set up test environments with SSH servers to validate real-world command execution.

• Security Audits: Review the implementation for potential security vulnerabilities, ensuring that sensitive information is handled securely.

9. Additional Recommendations

a. Asynchronous Execution

If Makim is expected to handle numerous remote tasks concurrently, consider implementing asynchronous execution to improve performance. Libraries like asyncssh could be explored for non-blocking SSH operations.

b. Connection Configuration Validation

Implement validation logic to ensure that the SSH configurations provided by the user are correct and complete before attempting to execute commands. This preemptively catches configuration errors, enhancing user experience.

c. Extensibility for Other Protocols

Design the execution dispatcher and configuration structure to accommodate additional connection protocols in the future (e.g., WinRM, Docker). This forward-thinking approach ensures that Makim remains adaptable to evolving automation needs.

d. Logging Enhancements

Enhance logging to differentiate between local and remote executions, including details about the remote host, executed commands, and their outputs. This aids in monitoring and debugging automation workflows.

Conclusion

Your proposed approach to integrating remote execution via Paramiko into Makim is well-conceived. By separating local and remote execution paths, expanding SSH configuration options, and ensuring robust error handling and security practices, you can create a flexible and powerful automation tool. Implementing the recommendations above will further enhance Makim's functionality, making it a versatile choice for both local and remote task automation.