ABI SNO On KVM (#263)

Addes support for Agent Based Installer SNO (Single Node) on KVM --------- Signed-off-by: Klaus Smolin <[email protected]> Signed-off-by: Sumit Kumar Solanki <[email protected]> Signed-off-by: root <[email protected]> Signed-off-by: veera-damisetti <[email protected]> Signed-off-by: Mohammed Zeeshan Ahmed <[email protected]> Signed-off-by: Amadeuds Podvratnik <[email protected]> Signed-off-by: K Shiva Sai <[email protected]> Co-authored-by: Klaus Smolin <[email protected]> Co-authored-by: Veerabhadrarao Damisetti <[email protected]> Co-authored-by: root <[email protected]> Co-authored-by: Amadeuds Podvratnik <[email protected]> Co-authored-by: Mohammed Ahmed <[email protected]> Co-authored-by: Sumit Kumar Solanki <[email protected]> Co-authored-by: k-shiva-sai <[email protected]> Co-authored-by: K Shiva Sai <[email protected]>
IBM · Apr 23, 2024 · 9dbe8f0 · 9dbe8f0
1 parent 91fd841
commit 9dbe8f0
Show file tree

Hide file tree

Showing 15 changed files with 481 additions and 1 deletion.
diff --git a/ansible.cfg b/ansible.cfg
@@ -6,6 +6,7 @@ roles_path=roles
 interpreter_python=auto
 host_key_checking=False
 deprecation_warnings=False
+# enable_task_debugger = True
 pipelining=True
 gathering=smart
 fact_caching=jsonfile

diff --git a/docs/run-the-playbooks-for-abi.md b/docs/run-the-playbooks-for-abi.md
@@ -0,0 +1,110 @@
+# Run the Playbooks
+## Prerequisites 
+* KVM host with root user access or user with sudo privileges.
+
+## Note: 
+* This playbook only support for single node cluster (SNO) on KVM using ABI.
+* As of now we are supporting only macvtap for Agent based installation (ABI) on KVM
+
+### Steps: 
+
+## Step-1: Initial Setup for ABI
+* Navigate to the [root folder of the cloned Git repository](https://github.com/IBM/Ansible-OpenShift-Provisioning) in your terminal (`ls` should show [ansible.cfg](https://github.com/IBM/Ansible-OpenShift-Provisioning/blob/main/ansible.cfg)).
+* Update variables in Section (1 - 9) and Section 12 - OpenShift Settings 
+* Update variables in Section - 19 ( Agent Based Installer ) in [all.yaml](https://github.com/veera-damisetti/Ansible-OpenShift-Provisioning/blob/main/inventories/default/group_vars/all.yaml.template) before running the playbooks.
+* In case of SNO Section 9 ( Compute Nodes ) need to be comment or remove 
+* First playbook to be run is 0_setup.yaml which will create inventory file for ABI and will add ssh key to the kvm host.
+
+* Run this shell command:
+```
+ansible-playbook playbooks/0_setup.yaml
+```
+
+* Run each part step-by-step by running one playbook at a time, or all at once using [playbooks/master_playbook_for_abi.yaml](https://github.com/IBM/Ansible-OpenShift-Provisioning/blob/main/playbooks/master_playbook_for_abi.yaml).
+* Here's the full list of playbooks to be run in order, full descriptions of each can be found further down the page:
+    * 0_setup.yaml ([code](https://github.com/IBM/Ansible-OpenShift-Provisioning/blob/main/playbooks/0_setup.yaml))
+    * 3_setup_kvm_host.yaml ([code](https://github.com/IBM/Ansible-OpenShift-Provisioning/blob/main/playbooks/3_setup_kvm_host.yaml))
+    * 4_create_bastion.yaml ([code](https://github.com/IBM/Ansible-OpenShift-Provisioning/blob/main/playbooks/4_create_bastion.yaml))
+    * 5_setup_bastion.yaml ([code](https://github.com/IBM/Ansible-OpenShift-Provisioning/blob/main/playbooks/5_setup_bastion.yaml))
+    * create_abi_cluster.yaml ([code](https://github.com/IBM/Ansible-OpenShift-Provisioning/blob/main/playbooks/create_abi_cluster.yaml))
+
+* Watch Ansible as it completes the installation, correcting errors if they arise.
+* To look at what tasks are running in detail, open the playbook or roles/role-name/tasks/main.yaml
+* Alternatively, to run all the playbooks at once, start the master playbook by running this shell command:
+
+```shell
+ansible-playbook playbooks/master_playbook_for_abi.yaml
+```
+
+* If the process fails in error, go through the steps in the [troubleshooting](troubleshooting.md) page.
+
+## Step-2: Setup Playbook (0_setup.yaml)
+### Overview
+First-time setup of the Ansible Controller, the machine running Ansible.
+### Outcomes
+* Packages and Ansible Galaxy collections are confirmed to be installed properly.
+* host_vars files are confirmed to match KVM host(s) hostnames.
+* Ansible inventory is templated out and working properly.
+* SSH key generated for Ansible passwordless authentication.
+* SSH agent is setup on the Ansible Controller.
+* Ansible SSH key is copied to the file server.
+### Notes
+* You can use an existing SSH key as your Ansible key, or have Ansible create one for you. It is highly recommended to use one without a passphrase.
+
+## Step-3: Setup KVM Host Playbook (3_setup_kvm_host.yaml)
+### Overview
+Configures the RHEL server(s) installed natively on the LPAR(s) to act as virtualization hypervisor(s) to host the virtual machines that make up the eventual cluster.
+### Outcomes
+* Ansible SSH key is copied to all KVM hosts for passwordless authentication.
+* RHEL subscription is auto-attached to all KVM hosts.
+* Software packages specified in group_vars/all.yaml have been installed.
+* Cockpit console enabled for Graphical User Interface via web browser. Go to http://kvm-ip-here:9090 to view it.
+* Libvirt is started and enabled.
+* Logical volume group that was created during kickstart is extended to fill all available space.
+* A macvtap bridge has been created on the host's networking interface.
+### Notes
+* If you're using a pre-existing LPAR, take a look at roles/configure_storage/tasks/main.yaml to make sure that the commands that will be run to extend the logical volume will work. Storage configurations can vary widely. The values there are the defaults from using autopart during kickstart. Also be aware that if lpar.storage_group_2.auto_config is True, the role roles/configure_storage/tasks/main.yaml will be non-idempotent. Meaning, it will fail if you run it twice.
+
+## Step-4: Create Bastion Playbook (4_create_bastion.yaml)
+### Overview
+Creates the bastion KVM guest on the first KVM host. The bastion hosts essential services for the cluster. If you already have a bastion server, that can be used instead of running this playbook.
+### Outcomes
+* Bastion configs are templated out to the file server.
+* Bastion is booted using virt-install.
+* Bastion is kickstarted for fully automated setup of the operating system.
+### Notes
+* This can be a particularly sticky part of the process.
+* If any of the variables used in the virt-install or kickstart are off, the bastion won't be able to boot.
+* Recommend watching it come up from the first KVM host's cockpit. Go to http://kvm-ip-here:9090 via web-browser to view it. You'll have to sign in, enable administrative access (top right), and then click on the virtual machines tab on the left-hand toolbar.
+## Step-5: Setup Bastion Playbook (5_setup_bastion.yaml)
+### Overview
+Configuration of the bastion to host essential infrastructure services for the cluster. Can be first-time setup or use an existing server.
+### Outcomes
+* Ansible SSH key copied to bastion for passwordless authentication.
+* Software packages specified in group_vars/all.yaml have been installed.
+* An OCP-specific SSH key is generated for passing into the install-config (then passed to the nodes).
+* Firewall is configured to permit traffic through the necessary ports.
+* Domain Name Server (DNS) configured to resolve cluster's IP addresses and APIs. Only done if env.bastion.options.dns is true.
+* DNS is checked to make sure all the necessary Fully Qualified Domain Names, including APIs resolve properly. Also ensures outside access is working.
+* High Availability Proxy (HAProxy) load balancer is configured. Only done if env.bastion.options.loadbalancer.on_bastion is true.
+* If the the cluster is to be highly available (meaning spread across more than one LPAR), an OpenVPN server is setup on the bastion to allow for the KVM hosts to communicate between eachother. OpenVPN clients are configured on the KVM hosts.
+* CoreOS roofts is pulled to the bastion if not already there.
+* OCP client and installer are pulled down if not there already.
+* oc, kubectl and openshift-install binaries are installed.
+* OCP install-config is templated and backed up. In disconnected mode, if platform is mirrored (currently only legacy), image content source policy and additionalTrustBundle is also patched.
+* Manfifests are created.
+* OCP install directory found at /root/ocpinst/ is created and populated with necessary files.
+* Ignition files for the bootstrap, control, and compute nodes are transferred to HTTP-accessible directory for booting nodes.
+### Notes
+* The stickiest part is DNS setup and get_ocp role at the end.
+
+## Step-6: Master Playbook (master_playbook_for_abi)
+### Overview
+* Use this playbook to run all required 5 playbooks (0_setup, 3_setup_kvm_host,4_create_bastion, 5_setup_bastion, create_abi_cluster) at once.
+### Outcomes
+* Same as all the above outcomes for all required playbooks.
+* At the end you will have an OpenShift cluster deployed and first-time login credentials.
+
+## Test Playbook (test.yaml)
+### Overview
+* Use this playbook for your testing purposes, if needed.
diff --git a/docs/set-variables-group-vars.md b/docs/set-variables-group-vars.md
@@ -318,3 +318,12 @@
 **day2_compute_node.vm_vm_interface** | The network interface used for given IP addresses of the compute node. | enc1
 **day2_compute_node.hostname** | The hostname of the KVM host | kvm-host-01
 **day2_compute_node.host_arch** | KVM host architecture.  | s390x
+
+## 19 - (Optional) Agent Based Installer
+
+**Variable Name** | **Description** | **Example**
+:--- | :--- | :---
+**abi.flag** | This is the flag, Will be used to identify during execution. Few checks in the playbook will be depend on this (default value will be False)  | True
+**abi.ansible_workdir** | This will be work directory name, it will keep required data that need to be present during or after execution | ansible_workdir
+**abi.ocp_installer_version** | Version will contain value of openshift-installer binary version user desired to be used | '4.15.0-rc.8'
+**abi.ocp_installer_url** | This is the base url of openshift installer binary it will remain same as static value, User Do not need to give value until user wants to change the mirror | 'https://mirror.openshift.com/pub/openshift-v4/s390x/clients/ocp/'
diff --git a/inventories/default/group_vars/all.yaml.template b/inventories/default/group_vars/all.yaml.template
@@ -477,3 +477,11 @@ day2_compute_node:
   vm_interface:
   hostname:
   host_arch:
+
+
+# Section 19 - Agent Based Installer ( Optional )
+abi: 
+  flag: True
+  ansible_workdir: 'ansible_workdir'
+  ocp_installer_version: '4.15.0-rc.8' 
+  ocp_installer_url: 'https://mirror.openshift.com/pub/openshift-v4/s390x/clients/ocp/'
diff --git a/playbooks/5_setup_bastion.yaml b/playbooks/5_setup_bastion.yaml
@@ -62,6 +62,7 @@
   become: true
   vars:
     packages: "{{ env.pkgs.bastion }}"
+    control_node_count: "{{ env.cluster.nodes.control.ip | length }}"
   vars_files:
     - "{{ inventory_dir }}/group_vars/all.yaml"
   pre_tasks:
@@ -78,6 +79,7 @@
     - { role: dns, when: env.bastion.options.dns }
     - { role: check_dns, when: env.bastion.options.dns is defined and env.bastion.options.dns }
     - { role: haproxy, when: env.bastion.options.loadbalancer.on_bastion }
+    - { role: sno_haproxy, when: env.bastion.options.loadbalancer.on_bastion and control_node_count | int == 1 }
     - httpd
 
 - hosts: bastion
@@ -176,4 +178,4 @@
     - "{{ inventory_dir }}/group_vars/all.yaml"
   roles:
     - common
-    - get_ocp
+    - { role: get_ocp, when: abi.flag is defined and abi.flag != True }
diff --git a/playbooks/create_abi_cluster.yaml b/playbooks/create_abi_cluster.yaml
@@ -0,0 +1,24 @@
+---
+
+# ABI Installation Proccess.
+- name: ABI Installation Proccess
+  hosts: bastion
+  become: false
+  vars_files:
+    - "{{ inventory_dir }}/group_vars/all.yaml"
+    - "{{ inventory_dir }}/host_vars/{{ env.z.lpar1.hostname }}.yaml"
+  roles:
+    - common # Common Variable the will be used by all the inwalked roles.
+    - download_ocp_installer # Download Openshift Installer.
+    - prepare_configs # Prepare AgentConfig & InstallConfig.
+    - create_agent # Create Agents || Build initrd.img, rootfs.img & kernelfs.img.
+
+# Boot ABI Agents. 
+- name: Boot ABI Agents
+  hosts: kvm_host[0]
+  become: false
+  vars_files:
+    - "{{ inventory_dir }}/group_vars/all.yaml"
+  roles:
+    - common
+    - boot_abi_agents 
diff --git a/playbooks/master_playbook_for_abi.yaml b/playbooks/master_playbook_for_abi.yaml
@@ -0,0 +1,8 @@
+# Master playbook For ABI . If you want to do everything all in one, use this.
+---
+
+- import_playbook: 0_setup.yaml # Import Playbook To Generate Inventory & Adding SSH-Keys.
+- import_playbook: 3_setup_kvm_host.yaml # Import Playbook To Install Prerequisites On KVM_HOST.
+- import_playbook: 4_create_bastion.yaml # Import Playbook To Create Bastion.
+- import_playbook: 5_setup_bastion.yaml # Import Playbook To Configure Bastion.
+- import_playbook: create_abi_cluster.yaml # Import Playbook To Create ABI Cluster.
diff --git a/roles/boot_abi_agents/tasks/main.yml b/roles/boot_abi_agents/tasks/main.yml
@@ -0,0 +1,47 @@
+---
+- name: Add Route To Bastion From KVM Host.
+  command: "ip route add {{ env.bastion.networking.ip }} via {{ env.bastion.networking.gateway }}"
+  ignore_errors: yes
+
+- name: Get and print virsh list
+  block:
+    - name: Get virsh list
+      community.libvirt.virt:
+        command: list_vms
+      register: cmd_virsh_list
+    - name: Print virsh list
+      ansible.builtin.debug:
+        var: cmd_virsh_list
+
+- name: Create CoreOS Control Agent Nodes On The KVM host.
+  ansible.builtin.shell: |
+    # Delete Agent node VM, if already exists
+    virsh destroy {{ env.cluster.nodes.control.vm_name[i] }} || true
+    virsh undefine {{ env.cluster.nodes.control.vm_name[i] }} --remove-all-storage --nvram || true
+    virt-install \
+    --name {{ env.cluster.nodes.control.vm_name[i] }} \
+    --autostart \
+    --disk pool={{ env.cluster.networking.metadata_name }}-vdisk,size={{ env.cluster.nodes.control.disk_size }},cache=none,io=native \
+    --ram {{ env.cluster.nodes.control.ram }} \
+    {{ env.cluster.nodes.control.vcpu_model_option }} \
+    --vcpus {{ env.cluster.nodes.control.vcpu }} \
+    --network network={{ env.bridge_name }}{{ (',mac=' + env.cluster.nodes.control.mac_address[i]) }} \
+    --location "http://{{ env.bastion.networking.ip }}:8080/,kernel={{ env.cluster.networking.metadata_name }}.{{ ansible_architecture }}-kernel.img,initrd={{ env.cluster.networking.metadata_name }}.{{ ansible_architecture }}-initrd.img" \
+    --extra-args "coreos.live.rootfs_url=http://{{ env.bastion.networking.ip }}:8080/{{ env.cluster.networking.metadata_name }}.{{ ansible_architecture }}-rootfs.img" \
+    {% if (env.cluster.nodes.control.mac_address[i] is defined and env.use_dhcp) %}
+    --extra-args "ip=dhcp" \
+    {% else %}
+    --extra-args "ip={{ env.cluster.nodes.control.ip[i] }}::{{ env.cluster.networking.gateway }}:{{ env.cluster.networking.subnetmask }}:{{ env.cluster.nodes.control.hostname[i] }}:{{ env.cluster.networking.interface }}:none" \
+    {% endif %}
+    --extra-args "rd.neednet=1 nameserver={{ env.cluster.networking.nameserver1 }}" \
+    --extra-args "random.trust_cpu=on rd.luks.options=discard ignition.firstboot ignition.platform.id=metal" \
+    --memballoon none \
+    --graphics none \
+    --wait=-1 \
+    --extra-args "{{ _vm_console }}" \
+    --noautoconsole 
+  timeout: 300
+  with_sequence: start=0 end={{ (env.cluster.nodes.control.hostname | length) - 1 }} stride=1
+  loop_control:
+    extended: yes
+    index_var: i
diff --git a/roles/create_agent/tasks/main.yml b/roles/create_agent/tasks/main.yml
@@ -0,0 +1,10 @@
+---
+
+- name: Create Agent Using PXE Boot
+  command: openshift-install agent create pxe-files --log-level=debug chdir=~/{{abi.ansible_workdir}}
+
+- name: Copy initrd.img	kernel.img rootfs.img 
+  ansible.builtin.copy:
+    src: "~/{{ abi.ansible_workdir }}/boot-artifacts/"
+    dest: /var/www/html/
+    remote_src: yes
diff --git a/roles/download_ocp_installer/tasks/main.yaml b/roles/download_ocp_installer/tasks/main.yaml
@@ -0,0 +1,30 @@
+---
+- name: Download OpenShift Installer.
+  get_url:
+    url: "{{ abi.ocp_installer_url }}{{abi.ocp_installer_version}}/openshift-install-linux.tar.gz"
+    dest: /tmp
+    validate_certs: false
+
+- name: Extract & Unzip Downloaded OpenShift Installer tar file on Remote
+  ansible.builtin.unarchive:
+    src: /tmp/openshift-install-linux.tar.gz
+    dest: /usr/local/bin
+    remote_src: yes
+
+- name: Download OpenShift Client.
+  get_url:
+    url: "{{ ocp_download_url }}{{ ocp_client_tgz }}"
+    dest: "/tmp/"
+    mode: "0755"
+
+- name: Extract & Unzip Downloaded OpenShift Client tar file on Remote
+  ansible.builtin.unarchive:
+    src: "{{ ocp_download_url }}{{ ocp_client_tgz }}"
+    dest: /usr/local/bin
+    remote_src: yes
+
+- name: Install NMState package
+  ansible.builtin.yum:
+    name: nmstate
+    state: latest
+    skip_broken: yes
diff --git a/roles/prepare_configs/tasks/main.yaml b/roles/prepare_configs/tasks/main.yaml
@@ -0,0 +1,38 @@
+---
+- name: Create Work Directory
+  file:
+    path: ~/ansible_workdir
+    state: directory
+
+- name: Prepare Agent Config
+  ansible.builtin.template:
+    src: agent-config.yaml.j2
+    dest: ~/ansible_workdir/agent-config.yaml
+    owner: root
+    group: root
+    mode: "0755"
+    backup: yes
+
+- name: Check if SSH key exists
+  stat:
+    path: "~/.ssh/{{ env.ansible_key_name }}.pub"
+  register: ssh_key
+
+- name: Generate SSH key
+  command: ssh-keygen -t rsa -b 4096 -N "" -f "~/.ssh/{{ env.ansible_key_name }}"
+  when: ssh_key.stat.exists == false
+
+- name: Read SSH Key
+  command: cat ~/.ssh/{{ env.ansible_key_name }}.pub
+  register: ssh_key
+
+- name: Prepare Install Config
+  vars:
+    ssh_key: ssh_key
+  ansible.builtin.template:
+    src: install-config.yaml.j2
+    dest: ~/ansible_workdir/install-config.yaml
+    owner: root
+    group: root
+    mode: "0755"
+    backup: yes
diff --git a/roles/prepare_configs/templates/agent-config.yaml.j2 b/roles/prepare_configs/templates/agent-config.yaml.j2
@@ -0,0 +1,67 @@
+apiVersion: v1alpha1
+kind: AgentConfig
+metadata:
+  name: {{ env.cluster.networking.metadata_name }}
+rendezvousIP: {{ env.cluster.nodes.control.ip[0] }}
+
+hosts:
+{% for item in range( env.cluster.nodes.control.ip | length ) %}
+  - hostname: "{{ env.cluster.nodes.control.hostname[item] }}"
+    role: master
+    interfaces:
+      - name: eth0
+        macAddress: "{{ env.cluster.nodes.control.mac_address[item] }}"
+    networkConfig:
+      interfaces:
+        - name: eth0
+          type: ethernet
+          state: up
+          mac-address: "{{ env.cluster.nodes.control.mac_address[item] }}"
+          ipv4:
+            enabled: true
+            address:
+              - ip: "{{ env.cluster.nodes.control.ip[item] }}"
+                prefix-length: 24
+            dhcp: false
+      dns-resolver:
+        config:
+          server:
+            - "{{ env.bastion.networking.ip }}"
+      routes:
+        config:
+          - destination: 0.0.0.0/0
+            next-hop-address: "{{ env.bastion.networking.gateway }}"
+            next-hop-interface: eth0
+            table-id: 254           
+{% endfor %}
+{% if env.cluster.nodes.compute.ip is defined %}
+{% for item in range( env.cluster.nodes.compute.ip | length ) %}
+  - hostname: "{{ env.cluster.nodes.compute.hostname[item] }}"
+    role: worker
+    interfaces:
+      - name: eth0
+        macAddress: "{{ env.cluster.nodes.compute.mac_address[item] }}"
+    networkConfig:
+      interfaces:
+        - name: eth0
+          type: ethernet
+          state: up
+          mac-address: "{{ env.cluster.nodes.compute.mac_address[item] }}"
+          ipv4:
+            enabled: true
+            address:
+              - ip: "{{ env.cluster.nodes.compute.ip[item] }}"
+                prefix-length: 24
+            dhcp: false
+      dns-resolver:
+        config:
+          server:
+            - "{{ env.bastion.networking.ip }}"
+      routes:
+        config:
+          - destination: 0.0.0.0/0
+            next-hop-address: "{{ env.bastion.networking.gateway }}"
+            next-hop-interface: eth0
+            table-id: 254 
+{% endfor %}
+{% endif %}