Merge pull request #52 from cdot65/50-enhance-ha-handling-in-batch-up…

…grades-for-pan-os-upgrade-script

Enhancements to HA Workflow, Multi-Threading Support, and Documentation Updates

docker/Dockerfile

@@ -11,8 +11,8 @@ RUN apk add --no-cache gcc musl-dev libffi-dev make
 WORKDIR /app
 
 # Install any needed packages specified in requirements.txt
-# Note: The requirements.txt should contain pan-os-upgrade==0.25
-RUN pip install --no-cache-dir pan-os-upgrade==0.25
+# Note: The requirements.txt should contain pan-os-upgrade==0.3.0
+RUN pip install --no-cache-dir pan-os-upgrade==0.3.0
 
 # Set the locale to avoid issues with emoji rendering
 ENV LANG C.UTF-8

docs/about/release-notes.md

@@ -2,10 +2,21 @@
 
 Welcome to the release notes for the `pan-os-upgrade` tool. This document provides a detailed record of changes, enhancements, and fixes in each version of the tool.
 
+## Version 0.3.0
+
+**Release Date:** *<20240125>*
+
+### What's New
+
+- Multi-threading added for concurrent upgrades (max limit of threads is 10).
+- Gracefully handle HA upgrades for HA active/passive peers.
+- Added hostname to log entries to differentiate threaded upgrades.
+
 ## Version 0.2.5
 
 **Release Date:** *<20240123>*
 
+<!-- trunk-ignore(markdownlint/MD024) -->
 ### What's New
 
 - Supports the ability to connect to Panorama as a proxy for firewall connections

docs/user-guide/docker/execution.md

@@ -44,6 +44,108 @@ docker run -v %CD%/assurance:/app/assurance -v %CD%/logs:/app/logs -it ghcr.io/c
 
 The container runs interactively, prompting you for details like IP address, username, password, and target PAN-OS version. If connecting to firewalls through Panorama as a proxy, you will also be prompted to provide a `--filter` option to specify the criteria for selecting the managed firewalls to upgrade.
 
+<div class="termy">
+
+```console
+$ docker run -v $(pwd)/assurance:/app/assurance -v $(pwd)/logs:/app/logs -it ghcr.io/cdot65/pan-os-upgrade:latest
+Hostname or IP: panorama.cdot.io
+Username: cdot
+Password:
+Target PAN-OS version: 10.2.2-h2
+Filter string (only applicable for Panorama) []: hostname=Woodlands*
+✅ panorama.cdot.io: Connection to Panorama established. Firewall connections will be proxied!
+📝 Woodlands-fw1: 007954000123451 192.168.255.43
+📝 Woodlands-fw2: 007954000123452 192.168.255.44
+📝 Woodlands-fw1: HA mode: passive
+📝 Woodlands-fw2: HA mode: active
+🔍 Woodlands-fw2: Detected active firewall in HA pair running the same version as its peer. Added firewall to revisit list.
+📝 Woodlands-fw1: Current PAN-OS version: 10.2.2
+📝 Woodlands-fw1: Target PAN-OS version: 10.2.2-h2
+✅ Woodlands-fw1: Upgrade required from 10.2.2 to 10.2.2-h2
+✅ Woodlands-fw1: PAN-OS version 10.2.2-h2 is available for download
+✅ Woodlands-fw1: Base image for 10.2.2-h2 is already downloaded
+🚀 Woodlands-fw1: Performing test to see if 10.2.2-h2 is already downloaded...
+🔍 Woodlands-fw1: PAN-OS version 10.2.2-h2 is not on the firewall
+🚀 Woodlands-fw1: PAN-OS version 10.2.2-h2 is beginning download
+Device 007954000123451 downloading version: 10.2.2-h2
+🔧 Woodlands-fw1: Downloading PAN-OS version 10.2.2-h2 - HA will sync image - Elapsed time: 5 seconds
+🔧 Woodlands-fw1: Downloading PAN-OS version 10.2.2-h2 - HA will sync image - Elapsed time: 37 seconds
+🔧 Woodlands-fw1: Downloading PAN-OS version 10.2.2-h2 - HA will sync image - Elapsed time: 68 seconds
+🔧 Woodlands-fw1: Downloading PAN-OS version 10.2.2-h2 - HA will sync image - Elapsed time: 100 seconds
+🔧 Woodlands-fw1: Downloading PAN-OS version 10.2.2-h2 - HA will sync image - Elapsed time: 133 seconds
+🔧 Woodlands-fw1: Downloading PAN-OS version 10.2.2-h2 - HA will sync image - Elapsed time: 167 seconds
+✅ Woodlands-fw1: 10.2.2-h2 downloaded in 199 seconds
+✅ Woodlands-fw1: 10.2.2-h2 has been downloaded and sync'd to HA peer.
+🚀 Woodlands-fw1: Performing snapshot of network state information...
+✅ Woodlands-fw1: Network snapshot created successfully
+🚀 Woodlands-fw1: Performing readiness checks to determine if firewall is ready for upgrade...
+✅ Woodlands-fw1: Passed Readiness Check: Check if there are pending changes on device
+✅ Woodlands-fw1: Passed Readiness Check: No Expired Licenses
+✅ Woodlands-fw1: Passed Readiness Check: Checks HA pair status from the perspective of the current device
+✅ Woodlands-fw1: Passed Readiness Check: Check if NTP is synchronized
+✅ Woodlands-fw1: Passed Readiness Check: Check connectivity with the Panorama appliance
+✅ Woodlands-fw1: Readiness Checks completed
+🚀 Woodlands-fw1: Checking if HA peer is in sync...
+✅ Woodlands-fw1: HA peer sync test has been completed.
+🚀 Woodlands-fw1: Performing backup of configuration to local filesystem...
+🚀 Woodlands-fw1: Not a dry run, continue with upgrade...
+🚀 Woodlands-fw1: Performing upgrade to version 10.2.2-h2...
+🚀 Woodlands-fw1: Attempting upgrade to version 10.2.2-h2 (Attempt 1 of 3)...
+Device 007954000123451 installing version: 10.2.2-h2
+✅ Woodlands-fw1: Upgrade completed successfully
+🚀 Woodlands-fw1: Rebooting the passive HA firewall...
+📝 Woodlands-fw1: Command succeeded with no output
+🔧 Woodlands-fw1: Firewall is rebooting...
+🔧 Woodlands-fw1: Firewall is rebooting...
+🔧 Woodlands-fw1: Firewall is rebooting...
+🔧 Woodlands-fw1: Firewall is rebooting...
+🔧 Woodlands-fw1: Firewall is rebooting...
+🔧 Woodlands-fw1: Firewall is rebooting...
+🔧 Woodlands-fw1: Firewall is rebooting...
+✅ Woodlands-fw1: HA passive firewall rebooted and synchronized with its peer in 499 seconds
+🚀 panorama.cdot.io: Revisiting firewalls that were active in an HA pair and had the same version as their peers.
+📝 Woodlands-fw2: 007954000123452 192.168.255.44
+📝 Woodlands-fw2: HA mode: active
+❌ Woodlands-fw2: Error suspending active firewall HA state: argument of type 'NoneType' is not iterable
+📝 Woodlands-fw2: Current PAN-OS version: 10.2.2
+📝 Woodlands-fw2: Target PAN-OS version: 10.2.2-h2
+✅ Woodlands-fw2: Upgrade required from 10.2.2 to 10.2.2-h2
+✅ Woodlands-fw2: PAN-OS version 10.2.2-h2 is available for download
+✅ Woodlands-fw2: Base image for 10.2.2-h2 is already downloaded
+🚀 Woodlands-fw2: Performing test to see if 10.2.2-h2 is already downloaded...
+✅ Woodlands-fw2: PAN-OS version 10.2.2-h2 already on firewall.
+✅ Woodlands-fw2: 10.2.2-h2 has been downloaded and sync'd to HA peer.
+🚀 Woodlands-fw2: Performing snapshot of network state information...
+✅ Woodlands-fw2: Network snapshot created successfully
+🚀 Woodlands-fw2: Performing readiness checks to determine if firewall is ready for upgrade...
+✅ Woodlands-fw2: Passed Readiness Check: Check if there are pending changes on device
+✅ Woodlands-fw2: Passed Readiness Check: No Expired Licenses
+✅ Woodlands-fw2: Passed Readiness Check: Check if NTP is synchronized
+✅ Woodlands-fw2: Passed Readiness Check: Check connectivity with the Panorama appliance
+✅ Woodlands-fw2: Readiness Checks completed
+🚀 Woodlands-fw2: Checking if HA peer is in sync...
+✅ Woodlands-fw2: HA peer sync test has been completed.
+🚀 Woodlands-fw2: Performing backup of configuration to local filesystem...
+🚀 Woodlands-fw2: Not a dry run, continue with upgrade...
+🚀 Woodlands-fw2: Performing upgrade to version 10.2.2-h2...
+🚀 Woodlands-fw2: Attempting upgrade to version 10.2.2-h2 (Attempt 1 of 3)...
+Device 007954000123452 installing version: 10.2.2-h2
+✅ Woodlands-fw2: Upgrade completed successfully
+🚀 Woodlands-fw2: Rebooting the passive HA firewall...
+📝 Woodlands-fw2: Command succeeded with no output
+🔧 Woodlands-fw2: Firewall is rebooting...
+🔧 Woodlands-fw2: Firewall is rebooting...
+🔧 Woodlands-fw2: Firewall is rebooting...
+🔧 Woodlands-fw2: Firewall is rebooting...
+🔧 Woodlands-fw2: Firewall is rebooting...
+🔧 Woodlands-fw2: Firewall is rebooting...
+🔧 Woodlands-fw2: Firewall is rebooting...
+✅ Woodlands-fw2: HA passive firewall rebooted and synchronized with its peer in 483 seconds
+✅ panorama.cdot.io: Completed revisiting firewalls
+```
+
+</div>
+
 ## Troubleshooting Panorama Proxy Connections
 
 When using Panorama as a connection proxy:

docs/user-guide/introduction.md

@@ -32,9 +32,12 @@ The Docker workflow simplifies the setup by encapsulating the tool and its depen
 
 `pan-os-upgrade` is equipped with several features for efficient and reliable upgrades:
 
-- **Leveraging `panos-upgrade-assurance`**: It utilizes the `panos-upgrade-assurance` library to manage complex aspects of the upgrade process.
-- **Data Validation with Pydantic**: Ensures robust data structure validation, minimizing bugs and streamlining workflow execution.
-- **Flexible Connection Methods**: Connect to firewalls directly or by targeting a Panorama appliance with a `--filter` CLI option.
+- **Automation of Routine Tasks**: Reduces manual errors and saves time by automating upgrades, configurations, and system checks.
+- **Support for Direct and Proxy Connections**: Connect directly to firewalls or through a Panorama appliance, with support for targeting specific devices using filters.
+- **Active/Passive High Availability (HA) Workflow**: Fully supports upgrading devices in active/passive HA configurations, ensuring both members are properly upgraded and synchronized.
+- **Multi-threading for Efficiency**: Utilizes multi-threading to parallelize upgrades, especially beneficial when upgrading multiple devices through Panorama, enhancing performance and reducing overall upgrade time.
+- **Customizable and Extensible**: Scripts can be tailored to fit diverse network environments and requirements, offering flexibility for various deployment scenarios.
+- **Comprehensive PAN-OS Interactions**: Facilitates extensive interactions with Palo Alto Networks appliances for operations like readiness checks, state snapshots, and report generation.
 
 ## Next Steps