From 2402cc6fc2959b3c6265d68da9c1ff9bbb702a1c Mon Sep 17 00:00:00 2001 From: Calvin Remsburg Date: Sun, 4 Feb 2024 12:05:40 -0600 Subject: [PATCH] Add release notes for version 1.1.0 and troubleshooting steps for ARP table comparison failures --- README.md | 252 ++++++++---- docs/about/release-notes.md | 15 + docs/index.md | 298 ++++++++------- docs/user-guide/docker/execution.md | 403 ++++++++++++++------ docs/user-guide/docker/troubleshooting.md | 15 + docs/user-guide/python/execution.md | 445 ++++++++++++++-------- docs/user-guide/python/troubleshooting.md | 15 + 7 files changed, 960 insertions(+), 483 deletions(-) diff --git a/README.md b/README.md index 9b563ed..c200569 100644 --- a/README.md +++ b/README.md @@ -53,6 +53,7 @@ Documentation: [https://cdot65.github.io/pan-os-upgrade/](https://cdot65.github. - The script will support up to ten simultaneous upgrades - **Automation of Routine Tasks**: Reduces manual errors and saves time by automating upgrades, configurations, and system checks. - **Support for Direct and Proxy Connections**: Connect directly to firewalls or through a Panorama appliance, with support for targeting specific devices using filters. +- **Pre/Post Diff**: Network snapshots are taken before and after the upgrade process, providing a PDF report of changes within the network environment after the upgrade completes. - **Active/Passive High Availability (HA) Workflow**: Fully supports upgrading devices in active/passive HA configurations, ensuring both members are properly upgraded and synchronized. - **Multi-threading for Efficiency**: Utilizes multi-threading to parallelize upgrades, especially beneficial when upgrading multiple devices through Panorama, enhancing performance and reducing overall upgrade time. - **Customizable and Extensible**: Execution of the script can be tailored to fit diverse network environments and requirements, offering flexibility for various deployment scenarios. @@ -69,7 +70,7 @@ pan-os-upgrade batch Panorama hostname or IP: panorama.cdot.io Panorama username: cdot Panorama password: -Firewall target version (ex: 10.1.2): 10.2.3 +Firewall target version (ex: 10.1.2): 10.2.7-h3 Filter string (ex: hostname=Woodlands*) []: hostname=Woodlands* Dry Run? [y/N]: =========================================================================== @@ -81,93 +82,174 @@ No settings.yaml file was found. Default values will be used. Create a settings.yaml file with 'pan-os-upgrade settings' command. =========================================================================== ✅ panorama.cdot.io: Connection to Panorama established. Firewall connections will be proxied! -📝 Woodlands-fw2: 007954000123452 192.168.255.44 -📝 Woodlands-fw1: 007954000123451 192.168.255.43 -📝 Woodlands-fw2: HA mode: passive -📝 Woodlands-fw1: HA mode: active -🔍 Woodlands-fw1: Detected active target device in HA pair running the same version as its peer. Added target device to revisit list. -📝 Woodlands-fw2: Current version: 10.2.2-h2 -📝 Woodlands-fw2: Target version: 10.2.3 -✅ Woodlands-fw2: Upgrade required from 10.2.2-h2 to 10.2.3 -✅ Woodlands-fw2: version 10.2.3 is available for download -✅ Woodlands-fw2: Base image for 10.2.3 is already downloaded -🚀 Woodlands-fw2: Performing test to see if 10.2.3 is already downloaded... -✅ Woodlands-fw2: version 10.2.3 already on target device. -✅ Woodlands-fw2: 10.2.3 has been downloaded and sync'd to HA peer. -🚀 Woodlands-fw2: Performing snapshot of network state information... -✅ Woodlands-fw2: Network snapshot created successfully -🚀 Woodlands-fw2: Performing readiness checks to determine if firewall is ready for upgrade... +📝 Woodlands-fw2: 007954000987652 192.168.255.44 +📝 Woodlands-fw1: 007954000987651 192.168.255.43 +📝 Woodlands-fw2: HA mode: active +📝 Woodlands-fw1: HA mode: passive +📝 Woodlands-fw2: Local state: active, Local version: 10.1.3, Peer version: 10.1.3 +📝 Woodlands-fw2: Version comparison: equal +🔍 Woodlands-fw2: Detected active target device in HA pair running the same version as its peer. Added target device to revisit list. +📝 Woodlands-fw1: Local state: passive, Local version: 10.1.3, Peer version: 10.1.3 +📝 Woodlands-fw1: Version comparison: equal +📝 Woodlands-fw1: Target device is passive +📝 Woodlands-fw1: Current version: 10.1.3 +📝 Woodlands-fw1: Target version: 10.2.7-h3 +✅ Woodlands-fw1: Upgrade required from 10.1.3 to 10.2.7-h3 +🔧 Woodlands-fw1: Refreshing list of available software versions +✅ Woodlands-fw1: version 10.2.7-h3 is available for download +❌ Woodlands-fw1: Base image for 10.2.7-h3 is not downloaded. Attempting download. +🔍 Woodlands-fw1: version 10.2.0 is not on the target device +🚀 Woodlands-fw1: version 10.2.0 is beginning download +Device 007954000987651 downloading version: 10.2.0 +🔧 Woodlands-fw1: Downloading version 10.2.0 - HA will sync image - Elapsed time: 3 seconds +🔧 Woodlands-fw1: Downloading version 10.2.0 - HA will sync image - Elapsed time: 35 seconds +🔧 Woodlands-fw1: Downloading version 10.2.0 - HA will sync image - Elapsed time: 66 seconds +🔧 Woodlands-fw1: Downloading version 10.2.0 - HA will sync image - Elapsed time: 98 seconds +🔧 Woodlands-fw1: Downloading version 10.2.0 - HA will sync image - Elapsed time: 129 seconds +🔧 Woodlands-fw1: Downloading version 10.2.0 - HA will sync image - Elapsed time: 160 seconds +🔧 Woodlands-fw1: Downloading version 10.2.0 - HA will sync image - Elapsed time: 192 seconds +🔧 Woodlands-fw1: Downloading version 10.2.0 - HA will sync image - Elapsed time: 223 seconds +🔧 Woodlands-fw1: Downloading version 10.2.0 - HA will sync image - Elapsed time: 257 seconds +🔧 Woodlands-fw1: Downloading version 10.2.0 - HA will sync image - Elapsed time: 289 seconds +✅ Woodlands-fw1: 10.2.0 downloaded in 321 seconds +✅ Woodlands-fw1: Base image 10.2.0 downloaded successfully +✅ Woodlands-fw1: Pausing for 60 seconds to let 10.2.0 image load into the software manager before downloading 10.2.7-h3 +📝 Woodlands-fw1: Current version: 10.1.3 +📝 Woodlands-fw1: Target version: 10.2.7-h3 +✅ Woodlands-fw1: Upgrade required from 10.1.3 to 10.2.7-h3 +🔧 Woodlands-fw1: Refreshing list of available software versions +✅ Woodlands-fw1: version 10.2.7-h3 is available for download +✅ Woodlands-fw1: Base image for 10.2.7-h3 is already downloaded +🚀 Woodlands-fw1: Performing test to see if 10.2.7-h3 is already downloaded. +🔍 Woodlands-fw1: version 10.2.7-h3 is not on the target device +🚀 Woodlands-fw1: version 10.2.7-h3 is beginning download +Device 007954000987651 downloading version: 10.2.7-h3 +🔧 Woodlands-fw1: Downloading version 10.2.7-h3 - HA will sync image - Elapsed time: 3 seconds +🔧 Woodlands-fw1: Downloading version 10.2.7-h3 - HA will sync image - Elapsed time: 35 seconds +🔧 Woodlands-fw1: Downloading version 10.2.7-h3 - HA will sync image - Elapsed time: 67 seconds +🔧 Woodlands-fw1: Downloading version 10.2.7-h3 - HA will sync image - Elapsed time: 103 seconds +🔧 Woodlands-fw1: Downloading version 10.2.7-h3 - HA will sync image - Elapsed time: 135 seconds +🔧 Woodlands-fw1: Downloading version 10.2.7-h3 - HA will sync image - Elapsed time: 168 seconds +🔧 Woodlands-fw1: Downloading version 10.2.7-h3 - HA will sync image - Elapsed time: 201 seconds +✅ Woodlands-fw1: 10.2.7-h3 downloaded in 233 seconds +✅ Woodlands-fw1: 10.2.7-h3 has been downloaded and sync'd to HA peer. +🚀 Woodlands-fw1: Performing snapshot of network state information. +🚀 Woodlands-fw1: Attempting to capture network state snapshot (Attempt 1 of 3). +✅ Woodlands-fw1: Network snapshot created successfully on attempt 1. +💾 Woodlands-fw1: Network state snapshot collected and saved to assurance/snapshots/Woodlands-fw1/pre/2024-02-04_09-15-40.json +🚀 Woodlands-fw1: Performing readiness checks to determine if firewall is ready for upgrade. +✅ Woodlands-fw1: Passed Readiness Check: Check if active support is available +🟨 Woodlands-fw1: Skipped Readiness Check: Check if a given ARP entry is available in the ARP table +✅ Woodlands-fw1: Passed Readiness Check: Check if there are pending changes on device +🟨 Woodlands-fw1: Skipped Readiness Check: Check if the certificates' keys meet minimum size requirements +🟨 Woodlands-fw1: Skipped Readiness Check: Running Latest Content Version +✅ Woodlands-fw1: Passed Readiness Check: Check if any Dynamic Update job is scheduled to run within the specified time window +✅ Woodlands-fw1: Passed Readiness Check: No Expired Licenses +🟨 Woodlands-fw1: Skipped Readiness Check: Check if a there is enough space on the `/opt/panrepo` volume for downloading an PanOS image. +✅ Woodlands-fw1: Passed Readiness Check: Checks HA pair status from the perspective of the current device +🟨 Woodlands-fw1: Skipped Readiness Check: Check if a given IPsec tunnel is in active state +🟨 Woodlands-fw1: Skipped Readiness Check: Check for any job with status different than FIN +🟨 Woodlands-fw1: Skipped Readiness Check: Check if NTP is synchronized +🟨 Woodlands-fw1: Skipped Readiness Check: Check if the clock is synchronized between dataplane and management plane +✅ Woodlands-fw1: Passed Readiness Check: Check connectivity with the Panorama appliance +🟨 Woodlands-fw1: Skipped Readiness Check: Check if a critical session is present in the sessions table +✅ Woodlands-fw1: Readiness Checks completed +🚀 Woodlands-fw1: Checking if HA peer is in sync. +✅ Woodlands-fw1: HA peer sync test has been completed. +🚀 Woodlands-fw1: Performing backup of configuration to local filesystem. +📝 Woodlands-fw1: Not a dry run, continue with upgrade. +🚀 Woodlands-fw1: Performing upgrade to version 10.2.7-h3. +📝 Woodlands-fw1: The install will take several minutes, check for status details within the GUI. +🚀 Woodlands-fw1: Attempting upgrade to version 10.2.7-h3 (Attempt 1 of 3). +Device 007954000987651 installing version: 10.2.7-h3 +✅ Woodlands-fw1: Upgrade completed successfully +🚀 Woodlands-fw1: Rebooting the target device. +📝 Woodlands-fw1: Command succeeded with no output +🟧 Woodlands-fw1: Retry attempt 1 due to error: 007954000987651 not connected +🟧 Woodlands-fw1: Retry attempt 2 due to error: 007954000987651 not connected +🟧 Woodlands-fw1: Retry attempt 3 due to error: 007954000987651 not connected +🟧 Woodlands-fw1: Retry attempt 4 due to error: 007954000987651 not connected +🟧 Woodlands-fw1: Retry attempt 5 due to error: 007954000987651 not connected +🟧 Woodlands-fw1: Retry attempt 6 due to error: 007954000987651 not connected +🟧 Woodlands-fw1: Retry attempt 7 due to error: 007954000987651 not connected +🟧 Woodlands-fw1: Retry attempt 8 due to error: 007954000987651 not connected +🟧 Woodlands-fw1: Retry attempt 9 due to error: 007954000987651 not connected +📝 Woodlands-fw1: Current device version: 10.2.7-h3 +✅ Woodlands-fw1: Device rebooted to the target version successfully. +🚀 Woodlands-fw1: Performing backup of configuration to local filesystem. +🔧 Woodlands-fw1: Waiting for the device to become ready for the post upgrade snapshot. +🚀 Woodlands-fw1: Performing snapshot of network state information. +🚀 Woodlands-fw1: Attempting to capture network state snapshot (Attempt 1 of 3). +✅ Woodlands-fw1: Network snapshot created successfully on attempt 1. +💾 Woodlands-fw1: Network state snapshot collected and saved to assurance/snapshots/Woodlands-fw1/post/2024-02-04_09-35-39.json +💾 Woodlands-fw1: Snapshot comparison PDF report saved to assurance/snapshots/Woodlands-fw1/diff/2024-02-04_09-35-40_report.pdf +🚀 panorama.cdot.io: Revisiting firewalls that were active in an HA pair and had the same version as their peers. +📝 Woodlands-fw2: 007954000987652 192.168.255.44 +📝 Woodlands-fw2: HA mode: non-functional +📝 Woodlands-fw2: Local state: non-functional, Local version: 10.1.3, Peer version: 10.2.7-h3 +Waiting for HA synchronization to complete on Woodlands-fw2. Attempt 1/3 +HA synchronization complete on Woodlands-fw2. Proceeding with upgrade. +📝 Woodlands-fw2: Version comparison: older +📝 Woodlands-fw2: Target device is on an older version +📝 Woodlands-fw2: Current version: 10.1.3 +📝 Woodlands-fw2: Target version: 10.2.7-h3 +✅ Woodlands-fw2: Upgrade required from 10.1.3 to 10.2.7-h3 +🔧 Woodlands-fw2: Refreshing list of available software versions +✅ Woodlands-fw2: version 10.2.7-h3 is available for download +✅ Woodlands-fw2: Base image for 10.2.7-h3 is already downloaded +🚀 Woodlands-fw2: Performing test to see if 10.2.7-h3 is already downloaded. +✅ Woodlands-fw2: version 10.2.7-h3 already on target device. +✅ Woodlands-fw2: version 10.2.7-h3 has been downloaded. +🚀 Woodlands-fw2: Performing snapshot of network state information. +🚀 Woodlands-fw2: Attempting to capture network state snapshot (Attempt 1 of 3). +✅ Woodlands-fw2: Network snapshot created successfully on attempt 1. +💾 Woodlands-fw2: Network state snapshot collected and saved to assurance/snapshots/Woodlands-fw2/pre/2024-02-04_09-36-48.json +🚀 Woodlands-fw2: Performing readiness checks to determine if firewall is ready for upgrade. +✅ Woodlands-fw2: Passed Readiness Check: Check if active support is available +🟨 Woodlands-fw2: Skipped Readiness Check: Check if a given ARP entry is available in the ARP table ✅ Woodlands-fw2: Passed Readiness Check: Check if there are pending changes on device +🟨 Woodlands-fw2: Skipped Readiness Check: Check if the certificates' keys meet minimum size requirements +🟨 Woodlands-fw2: Skipped Readiness Check: Running Latest Content Version +✅ Woodlands-fw2: Passed Readiness Check: Check if any Dynamic Update job is scheduled to run within the specified time window ✅ Woodlands-fw2: Passed Readiness Check: No Expired Licenses -✅ Woodlands-fw2: Passed Readiness Check: Checks HA pair status from the perspective of the current device -✅ Woodlands-fw2: Passed Readiness Check: Check if NTP is synchronized +🟨 Woodlands-fw2: Skipped Readiness Check: Check if a there is enough space on the `/opt/panrepo` volume for downloading an PanOS image. +🟨 Woodlands-fw2: Skipped Readiness Check: Checks HA pair status from the perspective of the current device +🟨 Woodlands-fw2: Skipped Readiness Check: Check if a given IPsec tunnel is in active state +🟨 Woodlands-fw2: Skipped Readiness Check: Check for any job with status different than FIN +🟨 Woodlands-fw2: Skipped Readiness Check: Check if NTP is synchronized ✅ Woodlands-fw2: Passed Readiness Check: Check if the clock is synchronized between dataplane and management plane ✅ Woodlands-fw2: Passed Readiness Check: Check connectivity with the Panorama appliance +🟨 Woodlands-fw2: Skipped Readiness Check: Check if a critical session is present in the sessions table ✅ Woodlands-fw2: Readiness Checks completed -🚀 Woodlands-fw2: Checking if HA peer is in sync... -✅ Woodlands-fw2: HA peer sync test has been completed. -🚀 Woodlands-fw2: Performing backup of configuration to local filesystem... -🚀 Woodlands-fw2: Not a dry run, continue with upgrade... -🚀 Woodlands-fw2: Performing upgrade to version 10.2.3... -🚀 Woodlands-fw2: Attempting upgrade to version 10.2.3 (Attempt 1 of 3)... -Device 007954000123452 installing version: 10.2.3 +🚀 Woodlands-fw2: Checking if HA peer is in sync. +🟧 Woodlands-fw2: HA peer state is not in sync. This will be noted, but the script will continue. +🚀 Woodlands-fw2: Performing backup of configuration to local filesystem. +📝 Woodlands-fw2: Not a dry run, continue with upgrade. +🚀 Woodlands-fw2: Performing upgrade to version 10.2.7-h3. +📝 Woodlands-fw2: The install will take several minutes, check for status details within the GUI. +🚀 Woodlands-fw2: Attempting upgrade to version 10.2.7-h3 (Attempt 1 of 3). +Device 007954000987652 installing version: 10.2.7-h3 ✅ Woodlands-fw2: Upgrade completed successfully -🚀 Woodlands-fw2: Rebooting the passive HA target device... +🚀 Woodlands-fw2: Rebooting the target device. 📝 Woodlands-fw2: Command succeeded with no output -🔧 Woodlands-fw2: Target device is rebooting... -🔧 Woodlands-fw2: Target device is rebooting... -🔧 Woodlands-fw2: Target device is rebooting... -🔧 Woodlands-fw2: Target device is rebooting... -🔧 Woodlands-fw2: Target device is rebooting... -🔧 Woodlands-fw2: Target device is rebooting... -🔧 Woodlands-fw2: Target device is rebooting... -🔧 Woodlands-fw2: Target device is rebooting... -🔧 Woodlands-fw2: Target device is rebooting... -🔧 Woodlands-fw2: HA passive target device rebooted but not yet synchronized with its peer. Will try again in 60 seconds. -🔧 Woodlands-fw2: HA passive target device rebooted but not yet synchronized with its peer. Will try again in 60 seconds. -🔧 Woodlands-fw2: HA passive target device rebooted but not yet synchronized with its peer. Will try again in 60 seconds. -🔧 Woodlands-fw2: HA passive target device rebooted but not yet synchronized with its peer. Will try again in 60 seconds. -🟧 Woodlands-fw2: HA passive target device rebooted but did not complete a configuration sync with the active after 5 attempts. -🚀 panorama.cdot.io: Revisiting firewalls that were active in an HA pair and had the same version as their peers. -📝 Woodlands-fw1: 007954000123451 192.168.255.43 -📝 Woodlands-fw1: HA mode: active -❌ Woodlands-fw1: Error suspending active target device HA state: argument of type 'NoneType' is not iterable -📝 Woodlands-fw1: Current version: 10.2.2-h2 -📝 Woodlands-fw1: Target version: 10.2.3 -✅ Woodlands-fw1: Upgrade required from 10.2.2-h2 to 10.2.3 -✅ Woodlands-fw1: version 10.2.3 is available for download -✅ Woodlands-fw1: Base image for 10.2.3 is already downloaded -🚀 Woodlands-fw1: Performing test to see if 10.2.3 is already downloaded... -✅ Woodlands-fw1: version 10.2.3 already on target device. -✅ Woodlands-fw1: 10.2.3 has been downloaded and sync'd to HA peer. -🚀 Woodlands-fw1: Performing snapshot of network state information... -✅ Woodlands-fw1: Network snapshot created successfully -🚀 Woodlands-fw1: Performing readiness checks to determine if firewall is ready for upgrade... -✅ Woodlands-fw1: Passed Readiness Check: Check if there are pending changes on device -✅ Woodlands-fw1: Passed Readiness Check: No Expired Licenses -✅ Woodlands-fw1: Passed Readiness Check: Check if NTP is synchronized -✅ Woodlands-fw1: Passed Readiness Check: Check connectivity with the Panorama appliance -✅ Woodlands-fw1: Readiness Checks completed -🚀 Woodlands-fw1: Checking if HA peer is in sync... -🟧 Woodlands-fw1: HA peer state is not in sync. This will be noted, but the script will continue. -🚀 Woodlands-fw1: Performing backup of configuration to local filesystem... -🚀 Woodlands-fw1: Not a dry run, continue with upgrade... -🚀 Woodlands-fw1: Performing upgrade to version 10.2.3... -🚀 Woodlands-fw1: Attempting upgrade to version 10.2.3 (Attempt 1 of 3)... -Device 007954000123451 installing version: 10.2.3 -✅ Woodlands-fw1: Upgrade completed successfully -🚀 Woodlands-fw1: Rebooting the passive HA target device... -📝 Woodlands-fw1: Command succeeded with no output -🔧 Woodlands-fw1: Target device is rebooting... -🔧 Woodlands-fw1: Target device is rebooting... -🔧 Woodlands-fw1: Target device is rebooting... -🔧 Woodlands-fw1: Target device is rebooting... -🔧 Woodlands-fw1: Target device is rebooting... -🔧 Woodlands-fw1: Target device is rebooting... -🔧 Woodlands-fw1: Target device is rebooting... -🔧 Woodlands-fw1: Target device is rebooting... -🔧 Woodlands-fw1: Target device is rebooting... -✅ Woodlands-fw1: HA passive target device rebooted and synchronized with its peer in 631 seconds +🟧 Woodlands-fw2: Retry attempt 1 due to error: 007954000987652 not connected +🟧 Woodlands-fw2: Retry attempt 2 due to error: 007954000987652 not connected +🟧 Woodlands-fw2: Retry attempt 3 due to error: 007954000987652 not connected +🟧 Woodlands-fw2: Retry attempt 4 due to error: 007954000987652 not connected +🟧 Woodlands-fw2: Retry attempt 5 due to error: 007954000987652 not connected +🟧 Woodlands-fw2: Retry attempt 6 due to error: 007954000987652 not connected +🟧 Woodlands-fw2: Retry attempt 7 due to error: 007954000987652 not connected +🟧 Woodlands-fw2: Retry attempt 8 due to error: 007954000987652 not connected +🟧 Woodlands-fw2: Retry attempt 9 due to error: 007954000987652 not connected +📝 Woodlands-fw2: Current device version: 10.2.7-h3 +✅ Woodlands-fw2: Device rebooted to the target version successfully. +🚀 Woodlands-fw2: Performing backup of configuration to local filesystem. +🔧 Woodlands-fw2: Waiting for the device to become ready for the post upgrade snapshot. +🚀 Woodlands-fw2: Performing snapshot of network state information. +🚀 Woodlands-fw2: Attempting to capture network state snapshot (Attempt 1 of 3). +✅ Woodlands-fw2: Network snapshot created successfully on attempt 1. +💾 Woodlands-fw2: Network state snapshot collected and saved to assurance/snapshots/Woodlands-fw2/post/2024-02-04_09-57-36.json +💾 Woodlands-fw2: Snapshot comparison PDF report saved to assurance/snapshots/Woodlands-fw2/diff/2024-02-04_09-57-38_report.pdf ✅ panorama.cdot.io: Completed revisiting firewalls ``` @@ -218,8 +300,10 @@ Email Address - cremsburg.dev at gmail.com ## Acknowledgments -* [Palo Alto Networks](https://www.paloaltonetworks.com/) -* [Python.org](https://python.org/) +This project is built upon the shoulders of two powerful Python libraries: `pan-os-python` and `panos-upgrade-assurance`. Both of these libraries are developed and maintained by Palo Alto Networks, providing an incredible amount of capabilities when automating PAN-OS and Panorama with Python. + +- [pan-os-python](https://pan-os-python.readthedocs.io/en/stable/) +- [panos-upgrade-assurance](https://github.com/PaloAltoNetworks/pan-os-upgrade-assurance/)

(back to top)

@@ -232,5 +316,5 @@ Email Address - cremsburg.dev at gmail.com [stars-url]: https://github.com/cdot65/pan-os-upgrade/stargazers [issues-shield]: https://img.shields.io/github/issues/cdot65/pan-os-upgrade.svg?style=for-the-badge [issues-url]: https://github.com/cdot65/pan-os-upgrade/issues -[license-shield]: https://img.shields.io/github/license/cdot65/pan-os-upgrade.svg?style=for-the-badge +[license-shield]: https://img.shields.io/github/**license**/cdot65/pan-os-upgrade.svg?style=for-the-badge [license-url]: https://github.com/cdot65/pan-os-upgrade/blob/main/LICENSE diff --git a/docs/about/release-notes.md b/docs/about/release-notes.md index 7ca0baf..eba62b0 100644 --- a/docs/about/release-notes.md +++ b/docs/about/release-notes.md @@ -2,10 +2,25 @@ Welcome to the release notes for the `pan-os-upgrade` tool. This document provides a detailed record of changes, enhancements, and fixes in each version of the tool. +## Version 1.1.0 + +**Release Date:** *<20240204>* + +### What's New + +- Pre/Post upgrade diff report created in PDF format +- Changed structure of AssuranceOptions +- `enabled_by_default` key added to allow for declaring which tests will execute by default +- Introduced "skipped" emoji to bring awareness to which tests and checks are skipped +- Using custom fork for `panos-upgrade-assurance` to account for integer values for `ttl` in ARP snapshots +- Added a new troubleshooting item to address how to handle when ARP snapshots fail due to a bug in the dependency +- Formatting and docstrings revisited + ## Version 1.0.0 **Release Date:** *<20240131>* + ### What's New - Shipping first release! 🚀 diff --git a/docs/index.md b/docs/index.md index 0a532c2..037cceb 100644 --- a/docs/index.md +++ b/docs/index.md @@ -38,6 +38,21 @@ hide: pan-os-upgrade is a modern Python CLI tool that provides a comprehensive automated workflow for PAN-OS firewalls. It offers two primary methods of execution: through a Python virtual environment or via a Docker container, catering to various operational requirements and preferences. +## Key Features + +- **Three Unique Upgrade Workflows Supported**: + - `firewall`: targets and upgrades an individual firewall + - `panorama`: targets and upgrades an individual Panorama appliance + - `batch`: targets a Panorama appliance and upgrades firewalls in batch + - The script will support up to ten simultaneous upgrades +- **Automation of Routine Tasks**: Reduces manual errors and saves time by automating upgrades, configurations, and system checks. +- **Support for Direct and Proxy Connections**: Connect directly to firewalls or through a Panorama appliance, with support for targeting specific devices using filters. +- **Pre/Post Diff**: Network snapshots are taken before and after the upgrade process, providing a PDF report of changes within the network environment after the upgrade completes. +- **Active/Passive High Availability (HA) Workflow**: Fully supports upgrading devices in active/passive HA configurations, ensuring both members are properly upgraded and synchronized. +- **Multi-threading for Efficiency**: Utilizes multi-threading to parallelize upgrades, especially beneficial when upgrading multiple devices through Panorama, enhancing performance and reducing overall upgrade time. +- **Customizable and Extensible**: Execution of the script can be tailored to fit diverse network environments and requirements, offering flexibility for various deployment scenarios. +- **Comprehensive PAN-OS Interactions**: Facilitates extensive interactions with Palo Alto Networks appliances for operations like readiness checks, state snapshots, and report generation. +**** ## Python Virtual Environment Workflow This approach involves setting up a Python virtual environment and running `pan-os-upgrade` within this isolated environment, ensuring compatibility and preventing any conflicts with system-wide Python installations. @@ -93,155 +108,174 @@ No settings.yaml file was found. Default values will be used. Create a settings.yaml file with 'pan-os-upgrade settings' command. =========================================================================== ✅ panorama.cdot.io: Connection to Panorama established. Firewall connections will be proxied! -📝 Woodlands-fw2: 007954001234562 192.168.255.44 -🚀 Woodlands-fw2: Getting 007954001234562 deployment information... -📝 Woodlands-fw1: 007954001234561 192.168.255.43 -🚀 Woodlands-fw1: Getting 007954001234561 deployment information... -📝 Woodlands-fw2: Target device deployment: passive -📝 Woodlands-fw2: HA mode: passive -🚀 Woodlands-fw2: Getting 007954001234562 deployment information... -📝 Woodlands-fw1: Target device deployment: active -📝 Woodlands-fw1: HA mode: active -🚀 Woodlands-fw1: Getting 007954001234561 deployment information... -📝 Woodlands-fw2: Target device deployment: passive -📝 Woodlands-fw1: Target device deployment: active -📝 Woodlands-fw2: Local state: passive, Local version: 10.1.3, Peer version: 10.1.3 -📝 Woodlands-fw1: Local state: active, Local version: 10.1.3, Peer version: 10.1.3 +📝 Woodlands-fw2: 007954000987652 192.168.255.44 +📝 Woodlands-fw1: 007954000987651 192.168.255.43 +📝 Woodlands-fw2: HA mode: active +📝 Woodlands-fw1: HA mode: passive +📝 Woodlands-fw2: Local state: active, Local version: 10.1.3, Peer version: 10.1.3 📝 Woodlands-fw2: Version comparison: equal +🔍 Woodlands-fw2: Detected active target device in HA pair running the same version as its peer. Added target device to revisit list. +📝 Woodlands-fw1: Local state: passive, Local version: 10.1.3, Peer version: 10.1.3 📝 Woodlands-fw1: Version comparison: equal -📝 Woodlands-fw2: Target device is passive -🔍 Woodlands-fw1: Detected active target device in HA pair running the same version as its peer. Added target device to revisit list. -📝 Woodlands-fw2: Current version: 10.1.3 -📝 Woodlands-fw2: Target version: 10.2.7-h3 -✅ Woodlands-fw2: Upgrade required from 10.1.3 to 10.2.7-h3 -✅ Woodlands-fw2: version 10.2.7-h3 is available for download -❌ Woodlands-fw2: Base image for 10.2.7-h3 is not downloaded. Attempting download... -🔍 Woodlands-fw2: version 10.2.0 is not on the target device -🚀 Woodlands-fw2: version 10.2.0 is beginning download -Device 007954001234562 downloading version: 10.2.0 -🔧 Woodlands-fw2: Downloading version 10.2.0 - HA will sync image - Elapsed time: 3 seconds -🔧 Woodlands-fw2: Downloading version 10.2.0 - HA will sync image - Elapsed time: 34 seconds -🔧 Woodlands-fw2: Downloading version 10.2.0 - HA will sync image - Elapsed time: 67 seconds -🔧 Woodlands-fw2: Downloading version 10.2.0 - HA will sync image - Elapsed time: 99 seconds -🔧 Woodlands-fw2: Downloading version 10.2.0 - HA will sync image - Elapsed time: 131 seconds -🔧 Woodlands-fw2: Downloading version 10.2.0 - HA will sync image - Elapsed time: 164 seconds -🔧 Woodlands-fw2: Downloading version 10.2.0 - HA will sync image - Elapsed time: 196 seconds -🔧 Woodlands-fw2: Downloading version 10.2.0 - HA will sync image - Elapsed time: 227 seconds -🔧 Woodlands-fw2: Downloading version 10.2.0 - HA will sync image - Elapsed time: 258 seconds -🔧 Woodlands-fw2: Downloading version 10.2.0 - HA will sync image - Elapsed time: 290 seconds -🔧 Woodlands-fw2: Downloading version 10.2.0 - HA will sync image - Elapsed time: 322 seconds -🔧 Woodlands-fw2: Downloading version 10.2.0 - HA will sync image - Elapsed time: 353 seconds -🔧 Woodlands-fw2: Downloading version 10.2.0 - HA will sync image - Elapsed time: 386 seconds -✅ Woodlands-fw2: 10.2.0 downloaded in 418 seconds -✅ Woodlands-fw2: Base image 10.2.0 downloaded successfully -✅ Woodlands-fw2: Pausing for 60 seconds to let 10.2.0 image load into the software manager before downloading 10.2.7-h3 -📝 Woodlands-fw2: Current version: 10.1.3 -📝 Woodlands-fw2: Target version: 10.2.7-h3 -✅ Woodlands-fw2: Upgrade required from 10.1.3 to 10.2.7-h3 -✅ Woodlands-fw2: version 10.2.7-h3 is available for download -✅ Woodlands-fw2: Base image for 10.2.7-h3 is already downloaded -🚀 Woodlands-fw2: Performing test to see if 10.2.7-h3 is already downloaded... -🔍 Woodlands-fw2: version 10.2.7-h3 is not on the target device -🚀 Woodlands-fw2: version 10.2.7-h3 is beginning download -Device 007954001234562 downloading version: 10.2.7-h3 -🔧 Woodlands-fw2: Downloading version 10.2.7-h3 - HA will sync image - Elapsed time: 3 seconds -🔧 Woodlands-fw2: Downloading version 10.2.7-h3 - HA will sync image - Elapsed time: 36 seconds -🔧 Woodlands-fw2: Downloading version 10.2.7-h3 - HA will sync image - Elapsed time: 67 seconds -🔧 Woodlands-fw2: Downloading version 10.2.7-h3 - HA will sync image - Elapsed time: 99 seconds -🔧 Woodlands-fw2: Downloading version 10.2.7-h3 - HA will sync image - Elapsed time: 132 seconds -🔧 Woodlands-fw2: Downloading version 10.2.7-h3 - HA will sync image - Elapsed time: 163 seconds -🔧 Woodlands-fw2: Downloading version 10.2.7-h3 - HA will sync image - Elapsed time: 195 seconds -🔧 Woodlands-fw2: Downloading version 10.2.7-h3 - HA will sync image - Elapsed time: 227 seconds -🔧 Woodlands-fw2: Downloading version 10.2.7-h3 - HA will sync image - Elapsed time: 260 seconds -✅ Woodlands-fw2: 10.2.7-h3 downloaded in 291 seconds -✅ Woodlands-fw2: 10.2.7-h3 has been downloaded and sync'd to HA peer. -🚀 Woodlands-fw2: Performing snapshot of network state information... -✅ Woodlands-fw2: Network snapshot created successfully -🚀 Woodlands-fw2: Performing readiness checks to determine if firewall is ready for upgrade... -✅ Woodlands-fw2: Passed Readiness Check: Check if there are pending changes on device -✅ Woodlands-fw2: Passed Readiness Check: No Expired Licenses -✅ Woodlands-fw2: Passed Readiness Check: Checks HA pair status from the perspective of the current device -✅ Woodlands-fw2: Passed Readiness Check: Check if NTP is synchronized -✅ Woodlands-fw2: Passed Readiness Check: Check if the clock is synchronized between dataplane and management plane -✅ Woodlands-fw2: Passed Readiness Check: Check connectivity with the Panorama appliance -✅ Woodlands-fw2: Readiness Checks completed -🚀 Woodlands-fw2: Checking if HA peer is in sync... -✅ Woodlands-fw2: HA peer sync test has been completed. -🚀 Woodlands-fw2: Performing backup of configuration to local filesystem... -🚀 Woodlands-fw2: Not a dry run, continue with upgrade... -🚀 Woodlands-fw2: Performing upgrade to version 10.2.7-h3... -🚀 Woodlands-fw2: Attempting upgrade to version 10.2.7-h3 (Attempt 1 of 3)... -Device 007954001234562 installing version: 10.2.7-h3 -✅ Woodlands-fw2: Upgrade completed successfully -🚀 Woodlands-fw2: Rebooting the target device... -📝 Woodlands-fw2: Command succeeded with no output -🟧 Woodlands-fw2: Retry attempt 1 due to error: 007954001234562 not connected -🟧 Woodlands-fw2: Retry attempt 2 due to error: 007954001234562 not connected -🟧 Woodlands-fw2: Retry attempt 3 due to error: 007954001234562 not connected -🟧 Woodlands-fw2: Retry attempt 4 due to error: 007954001234562 not connected -🟧 Woodlands-fw2: Retry attempt 5 due to error: 007954001234562 not connected -🟧 Woodlands-fw2: Retry attempt 6 due to error: 007954001234562 not connected -🟧 Woodlands-fw2: Retry attempt 7 due to error: 007954001234562 not connected -🟧 Woodlands-fw2: Retry attempt 8 due to error: 007954001234562 not connected -📝 Woodlands-fw2: Current device version: 10.2.7-h3 -✅ Woodlands-fw2: Device rebooted to the target version successfully. -🚀 panorama.cdot.io: Revisiting firewalls that were active in an HA pair and had the same version as their peers. -📝 Woodlands-fw1: 007954001234561 192.168.255.43 -🚀 Woodlands-fw1: Getting 007954001234561 deployment information... -📝 Woodlands-fw1: Target device deployment: active -📝 Woodlands-fw1: HA mode: active -🚀 Woodlands-fw1: Getting 007954001234561 deployment information... -📝 Woodlands-fw1: Target device deployment: active -📝 Woodlands-fw1: Local state: active, Local version: 10.1.3, Peer version: 10.1.3 -Waiting for HA synchronization to complete on Woodlands-fw1. Attempt 1/3 -🚀 Woodlands-fw1: Getting 007954001234561 deployment information... -📝 Woodlands-fw1: Target device deployment: active -HA synchronization still in progress on Woodlands-fw1. Rechecking after wait period. -Waiting for HA synchronization to complete on Woodlands-fw1. Attempt 2/3 -🚀 Woodlands-fw1: Getting 007954001234561 deployment information... -📝 Woodlands-fw1: Target device deployment: non-functional -HA synchronization complete on Woodlands-fw1. Proceeding with upgrade. -📝 Woodlands-fw1: Version comparison: older -📝 Woodlands-fw1: Target device is on an older version -📝 Woodlands-fw1: Suspending HA state of active -❌ Woodlands-fw1: Error suspending active target device HA state: argument of type 'NoneType' is not iterable +📝 Woodlands-fw1: Target device is passive 📝 Woodlands-fw1: Current version: 10.1.3 📝 Woodlands-fw1: Target version: 10.2.7-h3 ✅ Woodlands-fw1: Upgrade required from 10.1.3 to 10.2.7-h3 +🔧 Woodlands-fw1: Refreshing list of available software versions +✅ Woodlands-fw1: version 10.2.7-h3 is available for download +❌ Woodlands-fw1: Base image for 10.2.7-h3 is not downloaded. Attempting download. +🔍 Woodlands-fw1: version 10.2.0 is not on the target device +🚀 Woodlands-fw1: version 10.2.0 is beginning download +Device 007954000987651 downloading version: 10.2.0 +🔧 Woodlands-fw1: Downloading version 10.2.0 - HA will sync image - Elapsed time: 3 seconds +🔧 Woodlands-fw1: Downloading version 10.2.0 - HA will sync image - Elapsed time: 35 seconds +🔧 Woodlands-fw1: Downloading version 10.2.0 - HA will sync image - Elapsed time: 66 seconds +🔧 Woodlands-fw1: Downloading version 10.2.0 - HA will sync image - Elapsed time: 98 seconds +🔧 Woodlands-fw1: Downloading version 10.2.0 - HA will sync image - Elapsed time: 129 seconds +🔧 Woodlands-fw1: Downloading version 10.2.0 - HA will sync image - Elapsed time: 160 seconds +🔧 Woodlands-fw1: Downloading version 10.2.0 - HA will sync image - Elapsed time: 192 seconds +🔧 Woodlands-fw1: Downloading version 10.2.0 - HA will sync image - Elapsed time: 223 seconds +🔧 Woodlands-fw1: Downloading version 10.2.0 - HA will sync image - Elapsed time: 257 seconds +🔧 Woodlands-fw1: Downloading version 10.2.0 - HA will sync image - Elapsed time: 289 seconds +✅ Woodlands-fw1: 10.2.0 downloaded in 321 seconds +✅ Woodlands-fw1: Base image 10.2.0 downloaded successfully +✅ Woodlands-fw1: Pausing for 60 seconds to let 10.2.0 image load into the software manager before downloading 10.2.7-h3 +📝 Woodlands-fw1: Current version: 10.1.3 +📝 Woodlands-fw1: Target version: 10.2.7-h3 +✅ Woodlands-fw1: Upgrade required from 10.1.3 to 10.2.7-h3 +🔧 Woodlands-fw1: Refreshing list of available software versions ✅ Woodlands-fw1: version 10.2.7-h3 is available for download ✅ Woodlands-fw1: Base image for 10.2.7-h3 is already downloaded -🚀 Woodlands-fw1: Performing test to see if 10.2.7-h3 is already downloaded... -✅ Woodlands-fw1: version 10.2.7-h3 already on target device. +🚀 Woodlands-fw1: Performing test to see if 10.2.7-h3 is already downloaded. +🔍 Woodlands-fw1: version 10.2.7-h3 is not on the target device +🚀 Woodlands-fw1: version 10.2.7-h3 is beginning download +Device 007954000987651 downloading version: 10.2.7-h3 +🔧 Woodlands-fw1: Downloading version 10.2.7-h3 - HA will sync image - Elapsed time: 3 seconds +🔧 Woodlands-fw1: Downloading version 10.2.7-h3 - HA will sync image - Elapsed time: 35 seconds +🔧 Woodlands-fw1: Downloading version 10.2.7-h3 - HA will sync image - Elapsed time: 67 seconds +🔧 Woodlands-fw1: Downloading version 10.2.7-h3 - HA will sync image - Elapsed time: 103 seconds +🔧 Woodlands-fw1: Downloading version 10.2.7-h3 - HA will sync image - Elapsed time: 135 seconds +🔧 Woodlands-fw1: Downloading version 10.2.7-h3 - HA will sync image - Elapsed time: 168 seconds +🔧 Woodlands-fw1: Downloading version 10.2.7-h3 - HA will sync image - Elapsed time: 201 seconds +✅ Woodlands-fw1: 10.2.7-h3 downloaded in 233 seconds ✅ Woodlands-fw1: 10.2.7-h3 has been downloaded and sync'd to HA peer. -🚀 Woodlands-fw1: Performing snapshot of network state information... -✅ Woodlands-fw1: Network snapshot created successfully -🚀 Woodlands-fw1: Performing readiness checks to determine if firewall is ready for upgrade... +🚀 Woodlands-fw1: Performing snapshot of network state information. +🚀 Woodlands-fw1: Attempting to capture network state snapshot (Attempt 1 of 3). +✅ Woodlands-fw1: Network snapshot created successfully on attempt 1. +💾 Woodlands-fw1: Network state snapshot collected and saved to assurance/snapshots/Woodlands-fw1/pre/2024-02-04_09-15-40.json +🚀 Woodlands-fw1: Performing readiness checks to determine if firewall is ready for upgrade. +✅ Woodlands-fw1: Passed Readiness Check: Check if active support is available +🟨 Woodlands-fw1: Skipped Readiness Check: Check if a given ARP entry is available in the ARP table ✅ Woodlands-fw1: Passed Readiness Check: Check if there are pending changes on device +🟨 Woodlands-fw1: Skipped Readiness Check: Check if the certificates' keys meet minimum size requirements +🟨 Woodlands-fw1: Skipped Readiness Check: Running Latest Content Version +✅ Woodlands-fw1: Passed Readiness Check: Check if any Dynamic Update job is scheduled to run within the specified time window ✅ Woodlands-fw1: Passed Readiness Check: No Expired Licenses -✅ Woodlands-fw1: Passed Readiness Check: Check if NTP is synchronized -✅ Woodlands-fw1: Passed Readiness Check: Check if the clock is synchronized between dataplane and management plane +🟨 Woodlands-fw1: Skipped Readiness Check: Check if a there is enough space on the `/opt/panrepo` volume for downloading an PanOS image. +✅ Woodlands-fw1: Passed Readiness Check: Checks HA pair status from the perspective of the current device +🟨 Woodlands-fw1: Skipped Readiness Check: Check if a given IPsec tunnel is in active state +🟨 Woodlands-fw1: Skipped Readiness Check: Check for any job with status different than FIN +🟨 Woodlands-fw1: Skipped Readiness Check: Check if NTP is synchronized +🟨 Woodlands-fw1: Skipped Readiness Check: Check if the clock is synchronized between dataplane and management plane ✅ Woodlands-fw1: Passed Readiness Check: Check connectivity with the Panorama appliance +🟨 Woodlands-fw1: Skipped Readiness Check: Check if a critical session is present in the sessions table ✅ Woodlands-fw1: Readiness Checks completed -🚀 Woodlands-fw1: Checking if HA peer is in sync... +🚀 Woodlands-fw1: Checking if HA peer is in sync. ✅ Woodlands-fw1: HA peer sync test has been completed. -🚀 Woodlands-fw1: Performing backup of configuration to local filesystem... -🚀 Woodlands-fw1: Not a dry run, continue with upgrade... -🚀 Woodlands-fw1: Performing upgrade to version 10.2.7-h3... -🚀 Woodlands-fw1: Attempting upgrade to version 10.2.7-h3 (Attempt 1 of 3)... -Device 007954001234561 installing version: 10.2.7-h3 +🚀 Woodlands-fw1: Performing backup of configuration to local filesystem. +📝 Woodlands-fw1: Not a dry run, continue with upgrade. +🚀 Woodlands-fw1: Performing upgrade to version 10.2.7-h3. +📝 Woodlands-fw1: The install will take several minutes, check for status details within the GUI. +🚀 Woodlands-fw1: Attempting upgrade to version 10.2.7-h3 (Attempt 1 of 3). +Device 007954000987651 installing version: 10.2.7-h3 ✅ Woodlands-fw1: Upgrade completed successfully -🚀 Woodlands-fw1: Rebooting the target device... +🚀 Woodlands-fw1: Rebooting the target device. 📝 Woodlands-fw1: Command succeeded with no output -🟧 Woodlands-fw1: Retry attempt 1 due to error: 007954001234561 not connected -🟧 Woodlands-fw1: Retry attempt 2 due to error: 007954001234561 not connected -🟧 Woodlands-fw1: Retry attempt 3 due to error: 007954001234561 not connected -🟧 Woodlands-fw1: Retry attempt 4 due to error: 007954001234561 not connected -🟧 Woodlands-fw1: Retry attempt 5 due to error: 007954001234561 not connected -🟧 Woodlands-fw1: Retry attempt 6 due to error: 007954001234561 not connected -🟧 Woodlands-fw1: Retry attempt 7 due to error: 007954001234561 not connected -🟧 Woodlands-fw1: Retry attempt 8 due to error: 007954001234561 not connected +🟧 Woodlands-fw1: Retry attempt 1 due to error: 007954000987651 not connected +🟧 Woodlands-fw1: Retry attempt 2 due to error: 007954000987651 not connected +🟧 Woodlands-fw1: Retry attempt 3 due to error: 007954000987651 not connected +🟧 Woodlands-fw1: Retry attempt 4 due to error: 007954000987651 not connected +🟧 Woodlands-fw1: Retry attempt 5 due to error: 007954000987651 not connected +🟧 Woodlands-fw1: Retry attempt 6 due to error: 007954000987651 not connected +🟧 Woodlands-fw1: Retry attempt 7 due to error: 007954000987651 not connected +🟧 Woodlands-fw1: Retry attempt 8 due to error: 007954000987651 not connected +🟧 Woodlands-fw1: Retry attempt 9 due to error: 007954000987651 not connected 📝 Woodlands-fw1: Current device version: 10.2.7-h3 ✅ Woodlands-fw1: Device rebooted to the target version successfully. +🚀 Woodlands-fw1: Performing backup of configuration to local filesystem. +🔧 Woodlands-fw1: Waiting for the device to become ready for the post upgrade snapshot. +🚀 Woodlands-fw1: Performing snapshot of network state information. +🚀 Woodlands-fw1: Attempting to capture network state snapshot (Attempt 1 of 3). +✅ Woodlands-fw1: Network snapshot created successfully on attempt 1. +💾 Woodlands-fw1: Network state snapshot collected and saved to assurance/snapshots/Woodlands-fw1/post/2024-02-04_09-35-39.json +💾 Woodlands-fw1: Snapshot comparison PDF report saved to assurance/snapshots/Woodlands-fw1/diff/2024-02-04_09-35-40_report.pdf +🚀 panorama.cdot.io: Revisiting firewalls that were active in an HA pair and had the same version as their peers. +📝 Woodlands-fw2: 007954000987652 192.168.255.44 +📝 Woodlands-fw2: HA mode: non-functional +📝 Woodlands-fw2: Local state: non-functional, Local version: 10.1.3, Peer version: 10.2.7-h3 +Waiting for HA synchronization to complete on Woodlands-fw2. Attempt 1/3 +HA synchronization complete on Woodlands-fw2. Proceeding with upgrade. +📝 Woodlands-fw2: Version comparison: older +📝 Woodlands-fw2: Target device is on an older version +📝 Woodlands-fw2: Current version: 10.1.3 +📝 Woodlands-fw2: Target version: 10.2.7-h3 +✅ Woodlands-fw2: Upgrade required from 10.1.3 to 10.2.7-h3 +🔧 Woodlands-fw2: Refreshing list of available software versions +✅ Woodlands-fw2: version 10.2.7-h3 is available for download +✅ Woodlands-fw2: Base image for 10.2.7-h3 is already downloaded +🚀 Woodlands-fw2: Performing test to see if 10.2.7-h3 is already downloaded. +✅ Woodlands-fw2: version 10.2.7-h3 already on target device. +✅ Woodlands-fw2: version 10.2.7-h3 has been downloaded. +🚀 Woodlands-fw2: Performing snapshot of network state information. +🚀 Woodlands-fw2: Attempting to capture network state snapshot (Attempt 1 of 3). +✅ Woodlands-fw2: Network snapshot created successfully on attempt 1. +💾 Woodlands-fw2: Network state snapshot collected and saved to assurance/snapshots/Woodlands-fw2/pre/2024-02-04_09-36-48.json +🚀 Woodlands-fw2: Performing readiness checks to determine if firewall is ready for upgrade. +✅ Woodlands-fw2: Passed Readiness Check: Check if active support is available +🟨 Woodlands-fw2: Skipped Readiness Check: Check if a given ARP entry is available in the ARP table +✅ Woodlands-fw2: Passed Readiness Check: Check if there are pending changes on device +🟨 Woodlands-fw2: Skipped Readiness Check: Check if the certificates' keys meet minimum size requirements +🟨 Woodlands-fw2: Skipped Readiness Check: Running Latest Content Version +✅ Woodlands-fw2: Passed Readiness Check: Check if any Dynamic Update job is scheduled to run within the specified time window +✅ Woodlands-fw2: Passed Readiness Check: No Expired Licenses +🟨 Woodlands-fw2: Skipped Readiness Check: Check if a there is enough space on the `/opt/panrepo` volume for downloading an PanOS image. +🟨 Woodlands-fw2: Skipped Readiness Check: Checks HA pair status from the perspective of the current device +🟨 Woodlands-fw2: Skipped Readiness Check: Check if a given IPsec tunnel is in active state +🟨 Woodlands-fw2: Skipped Readiness Check: Check for any job with status different than FIN +🟨 Woodlands-fw2: Skipped Readiness Check: Check if NTP is synchronized +✅ Woodlands-fw2: Passed Readiness Check: Check if the clock is synchronized between dataplane and management plane +✅ Woodlands-fw2: Passed Readiness Check: Check connectivity with the Panorama appliance +🟨 Woodlands-fw2: Skipped Readiness Check: Check if a critical session is present in the sessions table +✅ Woodlands-fw2: Readiness Checks completed +🚀 Woodlands-fw2: Checking if HA peer is in sync. +🟧 Woodlands-fw2: HA peer state is not in sync. This will be noted, but the script will continue. +🚀 Woodlands-fw2: Performing backup of configuration to local filesystem. +📝 Woodlands-fw2: Not a dry run, continue with upgrade. +🚀 Woodlands-fw2: Performing upgrade to version 10.2.7-h3. +📝 Woodlands-fw2: The install will take several minutes, check for status details within the GUI. +🚀 Woodlands-fw2: Attempting upgrade to version 10.2.7-h3 (Attempt 1 of 3). +Device 007954000987652 installing version: 10.2.7-h3 +✅ Woodlands-fw2: Upgrade completed successfully +🚀 Woodlands-fw2: Rebooting the target device. +📝 Woodlands-fw2: Command succeeded with no output +🟧 Woodlands-fw2: Retry attempt 1 due to error: 007954000987652 not connected +🟧 Woodlands-fw2: Retry attempt 2 due to error: 007954000987652 not connected +🟧 Woodlands-fw2: Retry attempt 3 due to error: 007954000987652 not connected +🟧 Woodlands-fw2: Retry attempt 4 due to error: 007954000987652 not connected +🟧 Woodlands-fw2: Retry attempt 5 due to error: 007954000987652 not connected +🟧 Woodlands-fw2: Retry attempt 6 due to error: 007954000987652 not connected +🟧 Woodlands-fw2: Retry attempt 7 due to error: 007954000987652 not connected +🟧 Woodlands-fw2: Retry attempt 8 due to error: 007954000987652 not connected +🟧 Woodlands-fw2: Retry attempt 9 due to error: 007954000987652 not connected +📝 Woodlands-fw2: Current device version: 10.2.7-h3 +✅ Woodlands-fw2: Device rebooted to the target version successfully. +🚀 Woodlands-fw2: Performing backup of configuration to local filesystem. +🔧 Woodlands-fw2: Waiting for the device to become ready for the post upgrade snapshot. +🚀 Woodlands-fw2: Performing snapshot of network state information. +🚀 Woodlands-fw2: Attempting to capture network state snapshot (Attempt 1 of 3). +✅ Woodlands-fw2: Network snapshot created successfully on attempt 1. +💾 Woodlands-fw2: Network state snapshot collected and saved to assurance/snapshots/Woodlands-fw2/post/2024-02-04_09-57-36.json +💾 Woodlands-fw2: Snapshot comparison PDF report saved to assurance/snapshots/Woodlands-fw2/diff/2024-02-04_09-57-38_report.pdf ✅ panorama.cdot.io: Completed revisiting firewalls ``` diff --git a/docs/user-guide/docker/execution.md b/docs/user-guide/docker/execution.md index 7b1eb45..7d6f00a 100644 --- a/docs/user-guide/docker/execution.md +++ b/docs/user-guide/docker/execution.md @@ -78,8 +78,9 @@ docker run \ -v $(pwd)/assurance:/app/assurance \ -v $(pwd)/logs:/app/logs \ -it \ -ghcr.io/cdot65/pan-os-upgrade:latest firewall +ghcr.io/cdot65/pan-os-upgrade:latest firewall -v 11.1.1 -u cdot -h houston.cdot.io Firewall password: +Dry Run? [y/N]: =================================================================== Welcome to the PAN-OS upgrade tool @@ -90,32 +91,89 @@ Create a settings.yaml file with 'pan-os-upgrade settings' command. =================================================================== 📝 houston: 007054000242050 192.168.255.211 📝 houston: HA mode: disabled -📝 houston: Current version: 10.2.5 -📝 houston: Target version: 10.2.6 -✅ houston: Upgrade required from 10.2.5 to 10.2.6 -✅ houston: version 10.2.6 is available for download -✅ houston: Base image for 10.2.6 is already downloaded -🚀 houston: Performing test to see if 10.2.6 is already downloaded... -🔍 houston: version 10.2.6 is not on the target device -🚀 houston: version 10.2.6 is beginning download -Device 007054000242050 downloading version: 10.2.6 -🔧 houston: Downloading version 10.2.6 - Elapsed time: 5 seconds -🔧 houston: Downloading version 10.2.6 - Elapsed time: 41 seconds -🔧 houston: Downloading version 10.2.6 - Elapsed time: 76 seconds -✅ houston: 10.2.6 downloaded in 109 seconds -✅ houston: version 10.2.6 has been downloaded. -🚀 houston: Performing snapshot of network state information... -✅ houston: Network snapshot created successfully -🚀 houston: Performing readiness checks to determine if firewall is ready for upgrade... +📝 houston: Current version: 10.1.3 +📝 houston: Target version: 11.1.1 +✅ houston: Upgrade required from 10.1.3 to 11.1.1 +🔧 houston: Refreshing list of available software versions +✅ houston: version 11.1.1 is available for download +❌ houston: Base image for 11.1.1 is not downloaded. Attempting download. +🔍 houston: version 11.1.0 is not on the target device +🚀 houston: version 11.1.0 is beginning download +Device 007054000242050 downloading version: 11.1.0 +🔧 houston: Downloading version 11.1.0 - Elapsed time: 3 seconds +🔧 houston: Downloading version 11.1.0 - Elapsed time: 37 seconds +🔧 houston: Downloading version 11.1.0 - Elapsed time: 69 seconds +🔧 houston: Downloading version 11.1.0 - Elapsed time: 102 seconds +🔧 houston: Downloading version 11.1.0 - Elapsed time: 134 seconds +✅ houston: 11.1.0 downloaded in 167 seconds +✅ houston: Base image 11.1.0 downloaded successfully +✅ houston: Pausing for 60 seconds to let 11.1.0 image load into the software manager before downloading 11.1.1 +📝 houston: Current version: 10.1.3 +📝 houston: Target version: 11.1.1 +✅ houston: Upgrade required from 10.1.3 to 11.1.1 +🔧 houston: Refreshing list of available software versions +✅ houston: version 11.1.1 is available for download +✅ houston: Base image for 11.1.1 is already downloaded +🚀 houston: Performing test to see if 11.1.1 is already downloaded. +🔍 houston: version 11.1.1 is not on the target device +🚀 houston: version 11.1.1 is beginning download +Device 007054000242050 downloading version: 11.1.1 +🔧 houston: Downloading version 11.1.1 - Elapsed time: 6 seconds +🔧 houston: Downloading version 11.1.1 - Elapsed time: 40 seconds +🔧 houston: Downloading version 11.1.1 - Elapsed time: 74 seconds +✅ houston: 11.1.1 downloaded in 110 seconds +✅ houston: version 11.1.1 has been downloaded. +🚀 houston: Performing snapshot of network state information. +🚀 houston: Attempting to capture network state snapshot (Attempt 1 of 3). +✅ houston: Network snapshot created successfully on attempt 1. +💾 houston: Network state snapshot collected and saved to assurance/snapshots/houston/pre/2024-02-04_09-19-25.json +🚀 houston: Performing readiness checks to determine if firewall is ready for upgrade. +✅ houston: Passed Readiness Check: Check if active support is available +🟨 houston: Skipped Readiness Check: Check if a given ARP entry is available in the ARP table ✅ houston: Passed Readiness Check: Check if there are pending changes on device +🟨 houston: Skipped Readiness Check: Check if the certificates' keys meet minimum size requirements +🟨 houston: Skipped Readiness Check: Running Latest Content Version +✅ houston: Passed Readiness Check: Check if any Dynamic Update job is scheduled to run within the specified time window ✅ houston: Passed Readiness Check: No Expired Licenses -✅ houston: Passed Readiness Check: Check if NTP is synchronized -✅ houston: Passed Readiness Check: Check if the clock is synchronized between dataplane and management plane +✅ houston: Passed Readiness Check: Check if a there is enough space on the `/opt/panrepo` volume for downloading an PanOS image. +🟨 houston: Skipped Readiness Check: Checks HA pair status from the perspective of the current device +🟨 houston: Skipped Readiness Check: Check if a given IPsec tunnel is in active state +🟨 houston: Skipped Readiness Check: Check for any job with status different than FIN +🟨 houston: Skipped Readiness Check: Check if NTP is synchronized +🟨 houston: Skipped Readiness Check: Check if the clock is synchronized between dataplane and management plane ✅ houston: Passed Readiness Check: Check connectivity with the Panorama appliance +🟨 houston: Skipped Readiness Check: Check if a critical session is present in the sessions table ✅ houston: Readiness Checks completed -🚀 houston: Performing backup of configuration to local filesystem... -✅ houston: Dry run complete, exiting... -🛑 houston: Halting script. +🚀 houston: Performing backup of configuration to local filesystem. +📝 houston: Not a dry run, continue with upgrade. +🚀 houston: Performing upgrade to version 11.1.1. +📝 houston: The install will take several minutes, check for status details within the GUI. +🚀 houston: Attempting upgrade to version 11.1.1 (Attempt 1 of 3). +Device 007054000242050 installing version: 11.1.1 +❌ houston: Upgrade error: Device 007054000242050 attempt to install version 11.1.1 failed: ['Failed to install 11.1.1 with the following errors.\nSW version is 11.1.1\nThe software manager is currently in use. Please try again later.\nFailed to install version 11.1.1 type panos\n\n'] +🟧 houston: Software manager is busy. Retrying in 60 seconds. +🚀 houston: Attempting upgrade to version 11.1.1 (Attempt 2 of 3). +Device 007054000242050 installing version: 11.1.1 +✅ houston: Upgrade completed successfully +🚀 houston: Rebooting the target device. +📝 houston: Command succeeded with no output +🟧 houston: Retry attempt 1 due to error: URLError: reason: [Errno 60] Operation timed out +🟧 houston: Retry attempt 2 due to error: URLError: reason: [Errno 60] Operation timed out +🟧 houston: Retry attempt 3 due to error: URLError: reason: [Errno 61] Connection refused +🟧 houston: Retry attempt 4 due to error: URLError: reason: [Errno 61] Connection refused +🟧 houston: Retry attempt 5 due to error: URLError: reason: [Errno 61] Connection refused +🟧 houston: Retry attempt 6 due to error: URLError: code: 403 reason: API Error: Invalid Credential +🟧 houston: Retry attempt 7 due to error: URLError: code: 403 reason: API Error: Invalid Credential +🟧 houston: Retry attempt 8 due to error: URLError: code: 403 reason: API Error: Invalid Credential +📝 houston: Current device version: 11.1.1 +✅ houston: Device rebooted to the target version successfully. +🚀 houston: Performing backup of configuration to local filesystem. +🔧 houston: Waiting for the device to become ready for the post upgrade snapshot. +🚀 houston: Performing snapshot of network state information. +🚀 houston: Attempting to capture network state snapshot (Attempt 1 of 3). +✅ houston: Network snapshot created successfully on attempt 1. +💾 houston: Network state snapshot collected and saved to assurance/snapshots/houston/post/2024-02-04_09-44-21.json +💾 houston: Snapshot comparison PDF report saved to assurance/snapshots/houston/diff/2024-02-04_09-44-25_report.pdf ``` @@ -190,7 +248,7 @@ ghcr.io/cdot65/pan-os-upgrade:latest batch Panorama hostname or IP: panorama.cdot.io Panorama username: cdot Panorama password: -Firewall target version (ex: 10.1.2): 10.2.3 +Firewall target version (ex: 10.1.2): 10.2.7-h3 Filter string (ex: hostname=Woodlands*) []: hostname=Woodlands* Dry Run? [y/N]: =========================================================================== @@ -202,93 +260,174 @@ No settings.yaml file was found. Default values will be used. Create a settings.yaml file with 'pan-os-upgrade settings' command. =========================================================================== ✅ panorama.cdot.io: Connection to Panorama established. Firewall connections will be proxied! -📝 Woodlands-fw2: 007954000123452 192.168.255.44 -📝 Woodlands-fw1: 007954000123451 192.168.255.43 -📝 Woodlands-fw2: HA mode: passive -📝 Woodlands-fw1: HA mode: active -🔍 Woodlands-fw1: Detected active target device in HA pair running the same version as its peer. Added target device to revisit list. -📝 Woodlands-fw2: Current version: 10.2.2-h2 -📝 Woodlands-fw2: Target version: 10.2.3 -✅ Woodlands-fw2: Upgrade required from 10.2.2-h2 to 10.2.3 -✅ Woodlands-fw2: version 10.2.3 is available for download -✅ Woodlands-fw2: Base image for 10.2.3 is already downloaded -🚀 Woodlands-fw2: Performing test to see if 10.2.3 is already downloaded... -✅ Woodlands-fw2: version 10.2.3 already on target device. -✅ Woodlands-fw2: 10.2.3 has been downloaded and sync'd to HA peer. -🚀 Woodlands-fw2: Performing snapshot of network state information... -✅ Woodlands-fw2: Network snapshot created successfully -🚀 Woodlands-fw2: Performing readiness checks to determine if firewall is ready for upgrade... +📝 Woodlands-fw2: 007954000987652 192.168.255.44 +📝 Woodlands-fw1: 007954000987651 192.168.255.43 +📝 Woodlands-fw2: HA mode: active +📝 Woodlands-fw1: HA mode: passive +📝 Woodlands-fw2: Local state: active, Local version: 10.1.3, Peer version: 10.1.3 +📝 Woodlands-fw2: Version comparison: equal +🔍 Woodlands-fw2: Detected active target device in HA pair running the same version as its peer. Added target device to revisit list. +📝 Woodlands-fw1: Local state: passive, Local version: 10.1.3, Peer version: 10.1.3 +📝 Woodlands-fw1: Version comparison: equal +📝 Woodlands-fw1: Target device is passive +📝 Woodlands-fw1: Current version: 10.1.3 +📝 Woodlands-fw1: Target version: 10.2.7-h3 +✅ Woodlands-fw1: Upgrade required from 10.1.3 to 10.2.7-h3 +🔧 Woodlands-fw1: Refreshing list of available software versions +✅ Woodlands-fw1: version 10.2.7-h3 is available for download +❌ Woodlands-fw1: Base image for 10.2.7-h3 is not downloaded. Attempting download. +🔍 Woodlands-fw1: version 10.2.0 is not on the target device +🚀 Woodlands-fw1: version 10.2.0 is beginning download +Device 007954000987651 downloading version: 10.2.0 +🔧 Woodlands-fw1: Downloading version 10.2.0 - HA will sync image - Elapsed time: 3 seconds +🔧 Woodlands-fw1: Downloading version 10.2.0 - HA will sync image - Elapsed time: 35 seconds +🔧 Woodlands-fw1: Downloading version 10.2.0 - HA will sync image - Elapsed time: 66 seconds +🔧 Woodlands-fw1: Downloading version 10.2.0 - HA will sync image - Elapsed time: 98 seconds +🔧 Woodlands-fw1: Downloading version 10.2.0 - HA will sync image - Elapsed time: 129 seconds +🔧 Woodlands-fw1: Downloading version 10.2.0 - HA will sync image - Elapsed time: 160 seconds +🔧 Woodlands-fw1: Downloading version 10.2.0 - HA will sync image - Elapsed time: 192 seconds +🔧 Woodlands-fw1: Downloading version 10.2.0 - HA will sync image - Elapsed time: 223 seconds +🔧 Woodlands-fw1: Downloading version 10.2.0 - HA will sync image - Elapsed time: 257 seconds +🔧 Woodlands-fw1: Downloading version 10.2.0 - HA will sync image - Elapsed time: 289 seconds +✅ Woodlands-fw1: 10.2.0 downloaded in 321 seconds +✅ Woodlands-fw1: Base image 10.2.0 downloaded successfully +✅ Woodlands-fw1: Pausing for 60 seconds to let 10.2.0 image load into the software manager before downloading 10.2.7-h3 +📝 Woodlands-fw1: Current version: 10.1.3 +📝 Woodlands-fw1: Target version: 10.2.7-h3 +✅ Woodlands-fw1: Upgrade required from 10.1.3 to 10.2.7-h3 +🔧 Woodlands-fw1: Refreshing list of available software versions +✅ Woodlands-fw1: version 10.2.7-h3 is available for download +✅ Woodlands-fw1: Base image for 10.2.7-h3 is already downloaded +🚀 Woodlands-fw1: Performing test to see if 10.2.7-h3 is already downloaded. +🔍 Woodlands-fw1: version 10.2.7-h3 is not on the target device +🚀 Woodlands-fw1: version 10.2.7-h3 is beginning download +Device 007954000987651 downloading version: 10.2.7-h3 +🔧 Woodlands-fw1: Downloading version 10.2.7-h3 - HA will sync image - Elapsed time: 3 seconds +🔧 Woodlands-fw1: Downloading version 10.2.7-h3 - HA will sync image - Elapsed time: 35 seconds +🔧 Woodlands-fw1: Downloading version 10.2.7-h3 - HA will sync image - Elapsed time: 67 seconds +🔧 Woodlands-fw1: Downloading version 10.2.7-h3 - HA will sync image - Elapsed time: 103 seconds +🔧 Woodlands-fw1: Downloading version 10.2.7-h3 - HA will sync image - Elapsed time: 135 seconds +🔧 Woodlands-fw1: Downloading version 10.2.7-h3 - HA will sync image - Elapsed time: 168 seconds +🔧 Woodlands-fw1: Downloading version 10.2.7-h3 - HA will sync image - Elapsed time: 201 seconds +✅ Woodlands-fw1: 10.2.7-h3 downloaded in 233 seconds +✅ Woodlands-fw1: 10.2.7-h3 has been downloaded and sync'd to HA peer. +🚀 Woodlands-fw1: Performing snapshot of network state information. +🚀 Woodlands-fw1: Attempting to capture network state snapshot (Attempt 1 of 3). +✅ Woodlands-fw1: Network snapshot created successfully on attempt 1. +💾 Woodlands-fw1: Network state snapshot collected and saved to assurance/snapshots/Woodlands-fw1/pre/2024-02-04_09-15-40.json +🚀 Woodlands-fw1: Performing readiness checks to determine if firewall is ready for upgrade. +✅ Woodlands-fw1: Passed Readiness Check: Check if active support is available +🟨 Woodlands-fw1: Skipped Readiness Check: Check if a given ARP entry is available in the ARP table +✅ Woodlands-fw1: Passed Readiness Check: Check if there are pending changes on device +🟨 Woodlands-fw1: Skipped Readiness Check: Check if the certificates' keys meet minimum size requirements +🟨 Woodlands-fw1: Skipped Readiness Check: Running Latest Content Version +✅ Woodlands-fw1: Passed Readiness Check: Check if any Dynamic Update job is scheduled to run within the specified time window +✅ Woodlands-fw1: Passed Readiness Check: No Expired Licenses +🟨 Woodlands-fw1: Skipped Readiness Check: Check if a there is enough space on the `/opt/panrepo` volume for downloading an PanOS image. +✅ Woodlands-fw1: Passed Readiness Check: Checks HA pair status from the perspective of the current device +🟨 Woodlands-fw1: Skipped Readiness Check: Check if a given IPsec tunnel is in active state +🟨 Woodlands-fw1: Skipped Readiness Check: Check for any job with status different than FIN +🟨 Woodlands-fw1: Skipped Readiness Check: Check if NTP is synchronized +🟨 Woodlands-fw1: Skipped Readiness Check: Check if the clock is synchronized between dataplane and management plane +✅ Woodlands-fw1: Passed Readiness Check: Check connectivity with the Panorama appliance +🟨 Woodlands-fw1: Skipped Readiness Check: Check if a critical session is present in the sessions table +✅ Woodlands-fw1: Readiness Checks completed +🚀 Woodlands-fw1: Checking if HA peer is in sync. +✅ Woodlands-fw1: HA peer sync test has been completed. +🚀 Woodlands-fw1: Performing backup of configuration to local filesystem. +📝 Woodlands-fw1: Not a dry run, continue with upgrade. +🚀 Woodlands-fw1: Performing upgrade to version 10.2.7-h3. +📝 Woodlands-fw1: The install will take several minutes, check for status details within the GUI. +🚀 Woodlands-fw1: Attempting upgrade to version 10.2.7-h3 (Attempt 1 of 3). +Device 007954000987651 installing version: 10.2.7-h3 +✅ Woodlands-fw1: Upgrade completed successfully +🚀 Woodlands-fw1: Rebooting the target device. +📝 Woodlands-fw1: Command succeeded with no output +🟧 Woodlands-fw1: Retry attempt 1 due to error: 007954000987651 not connected +🟧 Woodlands-fw1: Retry attempt 2 due to error: 007954000987651 not connected +🟧 Woodlands-fw1: Retry attempt 3 due to error: 007954000987651 not connected +🟧 Woodlands-fw1: Retry attempt 4 due to error: 007954000987651 not connected +🟧 Woodlands-fw1: Retry attempt 5 due to error: 007954000987651 not connected +🟧 Woodlands-fw1: Retry attempt 6 due to error: 007954000987651 not connected +🟧 Woodlands-fw1: Retry attempt 7 due to error: 007954000987651 not connected +🟧 Woodlands-fw1: Retry attempt 8 due to error: 007954000987651 not connected +🟧 Woodlands-fw1: Retry attempt 9 due to error: 007954000987651 not connected +📝 Woodlands-fw1: Current device version: 10.2.7-h3 +✅ Woodlands-fw1: Device rebooted to the target version successfully. +🚀 Woodlands-fw1: Performing backup of configuration to local filesystem. +🔧 Woodlands-fw1: Waiting for the device to become ready for the post upgrade snapshot. +🚀 Woodlands-fw1: Performing snapshot of network state information. +🚀 Woodlands-fw1: Attempting to capture network state snapshot (Attempt 1 of 3). +✅ Woodlands-fw1: Network snapshot created successfully on attempt 1. +💾 Woodlands-fw1: Network state snapshot collected and saved to assurance/snapshots/Woodlands-fw1/post/2024-02-04_09-35-39.json +💾 Woodlands-fw1: Snapshot comparison PDF report saved to assurance/snapshots/Woodlands-fw1/diff/2024-02-04_09-35-40_report.pdf +🚀 panorama.cdot.io: Revisiting firewalls that were active in an HA pair and had the same version as their peers. +📝 Woodlands-fw2: 007954000987652 192.168.255.44 +📝 Woodlands-fw2: HA mode: non-functional +📝 Woodlands-fw2: Local state: non-functional, Local version: 10.1.3, Peer version: 10.2.7-h3 +Waiting for HA synchronization to complete on Woodlands-fw2. Attempt 1/3 +HA synchronization complete on Woodlands-fw2. Proceeding with upgrade. +📝 Woodlands-fw2: Version comparison: older +📝 Woodlands-fw2: Target device is on an older version +📝 Woodlands-fw2: Current version: 10.1.3 +📝 Woodlands-fw2: Target version: 10.2.7-h3 +✅ Woodlands-fw2: Upgrade required from 10.1.3 to 10.2.7-h3 +🔧 Woodlands-fw2: Refreshing list of available software versions +✅ Woodlands-fw2: version 10.2.7-h3 is available for download +✅ Woodlands-fw2: Base image for 10.2.7-h3 is already downloaded +🚀 Woodlands-fw2: Performing test to see if 10.2.7-h3 is already downloaded. +✅ Woodlands-fw2: version 10.2.7-h3 already on target device. +✅ Woodlands-fw2: version 10.2.7-h3 has been downloaded. +🚀 Woodlands-fw2: Performing snapshot of network state information. +🚀 Woodlands-fw2: Attempting to capture network state snapshot (Attempt 1 of 3). +✅ Woodlands-fw2: Network snapshot created successfully on attempt 1. +💾 Woodlands-fw2: Network state snapshot collected and saved to assurance/snapshots/Woodlands-fw2/pre/2024-02-04_09-36-48.json +🚀 Woodlands-fw2: Performing readiness checks to determine if firewall is ready for upgrade. +✅ Woodlands-fw2: Passed Readiness Check: Check if active support is available +🟨 Woodlands-fw2: Skipped Readiness Check: Check if a given ARP entry is available in the ARP table ✅ Woodlands-fw2: Passed Readiness Check: Check if there are pending changes on device +🟨 Woodlands-fw2: Skipped Readiness Check: Check if the certificates' keys meet minimum size requirements +🟨 Woodlands-fw2: Skipped Readiness Check: Running Latest Content Version +✅ Woodlands-fw2: Passed Readiness Check: Check if any Dynamic Update job is scheduled to run within the specified time window ✅ Woodlands-fw2: Passed Readiness Check: No Expired Licenses -✅ Woodlands-fw2: Passed Readiness Check: Checks HA pair status from the perspective of the current device -✅ Woodlands-fw2: Passed Readiness Check: Check if NTP is synchronized +🟨 Woodlands-fw2: Skipped Readiness Check: Check if a there is enough space on the `/opt/panrepo` volume for downloading an PanOS image. +🟨 Woodlands-fw2: Skipped Readiness Check: Checks HA pair status from the perspective of the current device +🟨 Woodlands-fw2: Skipped Readiness Check: Check if a given IPsec tunnel is in active state +🟨 Woodlands-fw2: Skipped Readiness Check: Check for any job with status different than FIN +🟨 Woodlands-fw2: Skipped Readiness Check: Check if NTP is synchronized ✅ Woodlands-fw2: Passed Readiness Check: Check if the clock is synchronized between dataplane and management plane ✅ Woodlands-fw2: Passed Readiness Check: Check connectivity with the Panorama appliance +🟨 Woodlands-fw2: Skipped Readiness Check: Check if a critical session is present in the sessions table ✅ Woodlands-fw2: Readiness Checks completed -🚀 Woodlands-fw2: Checking if HA peer is in sync... -✅ Woodlands-fw2: HA peer sync test has been completed. -🚀 Woodlands-fw2: Performing backup of configuration to local filesystem... -🚀 Woodlands-fw2: Not a dry run, continue with upgrade... -🚀 Woodlands-fw2: Performing upgrade to version 10.2.3... -🚀 Woodlands-fw2: Attempting upgrade to version 10.2.3 (Attempt 1 of 3)... -Device 007954000123452 installing version: 10.2.3 +🚀 Woodlands-fw2: Checking if HA peer is in sync. +🟧 Woodlands-fw2: HA peer state is not in sync. This will be noted, but the script will continue. +🚀 Woodlands-fw2: Performing backup of configuration to local filesystem. +📝 Woodlands-fw2: Not a dry run, continue with upgrade. +🚀 Woodlands-fw2: Performing upgrade to version 10.2.7-h3. +📝 Woodlands-fw2: The install will take several minutes, check for status details within the GUI. +🚀 Woodlands-fw2: Attempting upgrade to version 10.2.7-h3 (Attempt 1 of 3). +Device 007954000987652 installing version: 10.2.7-h3 ✅ Woodlands-fw2: Upgrade completed successfully -🚀 Woodlands-fw2: Rebooting the passive HA target device... +🚀 Woodlands-fw2: Rebooting the target device. 📝 Woodlands-fw2: Command succeeded with no output -🔧 Woodlands-fw2: Target device is rebooting... -🔧 Woodlands-fw2: Target device is rebooting... -🔧 Woodlands-fw2: Target device is rebooting... -🔧 Woodlands-fw2: Target device is rebooting... -🔧 Woodlands-fw2: Target device is rebooting... -🔧 Woodlands-fw2: Target device is rebooting... -🔧 Woodlands-fw2: Target device is rebooting... -🔧 Woodlands-fw2: Target device is rebooting... -🔧 Woodlands-fw2: Target device is rebooting... -🔧 Woodlands-fw2: HA passive target device rebooted but not yet synchronized with its peer. Will try again in 60 seconds. -🔧 Woodlands-fw2: HA passive target device rebooted but not yet synchronized with its peer. Will try again in 60 seconds. -🔧 Woodlands-fw2: HA passive target device rebooted but not yet synchronized with its peer. Will try again in 60 seconds. -🔧 Woodlands-fw2: HA passive target device rebooted but not yet synchronized with its peer. Will try again in 60 seconds. -🟧 Woodlands-fw2: HA passive target device rebooted but did not complete a configuration sync with the active after 5 attempts. -🚀 panorama.cdot.io: Revisiting firewalls that were active in an HA pair and had the same version as their peers. -📝 Woodlands-fw1: 007954000123451 192.168.255.43 -📝 Woodlands-fw1: HA mode: active -❌ Woodlands-fw1: Error suspending active target device HA state: argument of type 'NoneType' is not iterable -📝 Woodlands-fw1: Current version: 10.2.2-h2 -📝 Woodlands-fw1: Target version: 10.2.3 -✅ Woodlands-fw1: Upgrade required from 10.2.2-h2 to 10.2.3 -✅ Woodlands-fw1: version 10.2.3 is available for download -✅ Woodlands-fw1: Base image for 10.2.3 is already downloaded -🚀 Woodlands-fw1: Performing test to see if 10.2.3 is already downloaded... -✅ Woodlands-fw1: version 10.2.3 already on target device. -✅ Woodlands-fw1: 10.2.3 has been downloaded and sync'd to HA peer. -🚀 Woodlands-fw1: Performing snapshot of network state information... -✅ Woodlands-fw1: Network snapshot created successfully -🚀 Woodlands-fw1: Performing readiness checks to determine if firewall is ready for upgrade... -✅ Woodlands-fw1: Passed Readiness Check: Check if there are pending changes on device -✅ Woodlands-fw1: Passed Readiness Check: No Expired Licenses -✅ Woodlands-fw1: Passed Readiness Check: Check if NTP is synchronized -✅ Woodlands-fw1: Passed Readiness Check: Check connectivity with the Panorama appliance -✅ Woodlands-fw1: Readiness Checks completed -🚀 Woodlands-fw1: Checking if HA peer is in sync... -🟧 Woodlands-fw1: HA peer state is not in sync. This will be noted, but the script will continue. -🚀 Woodlands-fw1: Performing backup of configuration to local filesystem... -🚀 Woodlands-fw1: Not a dry run, continue with upgrade... -🚀 Woodlands-fw1: Performing upgrade to version 10.2.3... -🚀 Woodlands-fw1: Attempting upgrade to version 10.2.3 (Attempt 1 of 3)... -Device 007954000123451 installing version: 10.2.3 -✅ Woodlands-fw1: Upgrade completed successfully -🚀 Woodlands-fw1: Rebooting the passive HA target device... -📝 Woodlands-fw1: Command succeeded with no output -🔧 Woodlands-fw1: Target device is rebooting... -🔧 Woodlands-fw1: Target device is rebooting... -🔧 Woodlands-fw1: Target device is rebooting... -🔧 Woodlands-fw1: Target device is rebooting... -🔧 Woodlands-fw1: Target device is rebooting... -🔧 Woodlands-fw1: Target device is rebooting... -🔧 Woodlands-fw1: Target device is rebooting... -🔧 Woodlands-fw1: Target device is rebooting... -🔧 Woodlands-fw1: Target device is rebooting... -✅ Woodlands-fw1: HA passive target device rebooted and synchronized with its peer in 631 seconds +🟧 Woodlands-fw2: Retry attempt 1 due to error: 007954000987652 not connected +🟧 Woodlands-fw2: Retry attempt 2 due to error: 007954000987652 not connected +🟧 Woodlands-fw2: Retry attempt 3 due to error: 007954000987652 not connected +🟧 Woodlands-fw2: Retry attempt 4 due to error: 007954000987652 not connected +🟧 Woodlands-fw2: Retry attempt 5 due to error: 007954000987652 not connected +🟧 Woodlands-fw2: Retry attempt 6 due to error: 007954000987652 not connected +🟧 Woodlands-fw2: Retry attempt 7 due to error: 007954000987652 not connected +🟧 Woodlands-fw2: Retry attempt 8 due to error: 007954000987652 not connected +🟧 Woodlands-fw2: Retry attempt 9 due to error: 007954000987652 not connected +📝 Woodlands-fw2: Current device version: 10.2.7-h3 +✅ Woodlands-fw2: Device rebooted to the target version successfully. +🚀 Woodlands-fw2: Performing backup of configuration to local filesystem. +🔧 Woodlands-fw2: Waiting for the device to become ready for the post upgrade snapshot. +🚀 Woodlands-fw2: Performing snapshot of network state information. +🚀 Woodlands-fw2: Attempting to capture network state snapshot (Attempt 1 of 3). +✅ Woodlands-fw2: Network snapshot created successfully on attempt 1. +💾 Woodlands-fw2: Network state snapshot collected and saved to assurance/snapshots/Woodlands-fw2/post/2024-02-04_09-57-36.json +💾 Woodlands-fw2: Snapshot comparison PDF report saved to assurance/snapshots/Woodlands-fw2/diff/2024-02-04_09-57-38_report.pdf ✅ panorama.cdot.io: Completed revisiting firewalls ``` @@ -300,6 +439,60 @@ If you would like to change the default settings of `pan-os-upgrade` tool, you c Create the empty `settings.yaml` file within your current working directory +### Readiness Checks + +The following table lists the available readiness checks, their descriptions, and whether they are enabled by default. These checks are designed to ensure the device's readiness for an upgrade by validating its operational and configuration status. + +| Readiness Check | Description | Enabled by Default | +| --------------------------- | ----------------------------------------------------------------------------------------- | :----------------: | +| `active_support` | Check if active support is available | Yes | +| `arp_entry_exist` | Check if a given ARP entry is available in the ARP table | No | +| `candidate_config` | Check if there are pending changes on device | Yes | +| `certificates_requirements` | Check if the certificates' keys meet minimum size requirements | No | +| `content_version` | Running Latest Content Version | Yes | +| `dynamic_updates` | Check if any Dynamic Update job is scheduled to run within the specified time window | Yes | +| `expired_licenses` | No Expired Licenses | Yes | +| `free_disk_space` | Check if there is enough space on the `/opt/panrepo` volume for downloading a PanOS image | Yes | +| `ha` | Checks HA pair status from the perspective of the current device | Yes | +| `ip_sec_tunnel_status` | Check if a given IPsec tunnel is in active state | Yes | +| `jobs` | Check for any job with status different than FIN | No | +| `ntp_sync` | Check if NTP is synchronized | No | +| `panorama` | Check connectivity with the Panorama appliance | Yes | +| `planes_clock_sync` | Check if the clock is synchronized between dataplane and management plane | Yes | +| `session_exist` | Check if a critical session is present in the sessions table | No | + +### State Snapshots + +The following table lists the categories of state snapshots that can be captured to document essential data about the device's current state. These snapshots are crucial for diagnostics and verifying the device's operational status before proceeding with the upgrade. + +| Snapshot | Description | Enabled by Default | +| ----------------- | ----------------------------------- | :----------------: | +| `arp_table` | Snapshot of the ARP Table | Yes | +| `content_version` | Snapshot of the Content Version | Yes | +| `ip_sec_tunnels` | Snapshot of the IPsec Tunnels | No | +| `license` | Snapshot of the License Information | Yes | +| `nics` | Snapshot of the Network Interfaces | Yes | +| `routes` | Snapshot of the Routing Table | Yes | +| `session_stats` | Snapshot of the Session Statistics | No | + +### Customizing Default Settings + +The default settings for readiness checks and snapshots can be customized using the `pan-os-upgrade settings` subcommand. This interactive command guides you through a series of prompts to configure various aspects of the script's behavior, including which readiness checks and snapshots are enabled. + +To override the default settings: + +1. Run the `pan-os-upgrade settings` command. +2. Follow the prompts to enable or disable specific readiness checks and snapshots. +3. The resulting configurations are saved to a `settings.yaml` file in the current working directory. + + ```bash + pan-os-upgrade settings + ``` + +#### Note + +The `settings.yaml` file created by this command can be edited manually for further customization. +
```console @@ -404,8 +597,6 @@ Custom configuration loaded from: 📝 houston: Target version: 10.2.5 ✅ houston: Upgrade required from 10.2.4-h4 to 10.2.5 ... shortened for brevity ... -🟧 houston: Retry attempt 4 due to error: URLError: reason: [Errno 111] Connection refused -📝 houston: Current device version: 10.2.5 ✅ houston: Device rebooted to the target version successfully. ``` diff --git a/docs/user-guide/docker/troubleshooting.md b/docs/user-guide/docker/troubleshooting.md index 4d48978..2e9a539 100644 --- a/docs/user-guide/docker/troubleshooting.md +++ b/docs/user-guide/docker/troubleshooting.md @@ -54,6 +54,21 @@ Encountering issues during the Docker execution of `pan-os-upgrade` can happen, **Solution:** Ensure the filter syntax is correctly formatted and the specified criteria accurately reflect your firewall configuration in Panorama. Double-check network connectivity to Panorama and ensure the filters match the attributes of the firewalls you intend to upgrade. +### 9. ARP Table Comparison Failures + +**Problem:** When capturing ARP tables for comparison, the script fails with `WrongDataTypeException: Unknown value format for key ttl`. + +**Solution:** This issue can arise when ARP table entries contain integer values for `ttl`, which the current implementation may not handle properly. To address this, consider installing a custom fork of `panos-upgrade-assurance` that includes a fix for this issue, available at [https://github.com/cdot65/pan-os-upgrade-assurance/tree/main](https://github.com/cdot65/pan-os-upgrade-assurance/tree/main). Alternatively, you can configure the script to omit ARP snapshots from the tests if modifying the script is not feasible. + +**Steps to Install Custom Fork:** + +1. Run this command: `pip install git+https://github.com/cdot65/pan-os-upgrade-assurance.git@main` + +**Steps to Omit ARP Snapshots:** + +1. If using a `settings.yaml` file, ensure ARP snapshots are disabled. +2. If running the script interactively, choose not to capture ARP snapshots when prompted. + ## General Tips - Always verify your Docker setup and configurations before running `pan-os-upgrade`. diff --git a/docs/user-guide/python/execution.md b/docs/user-guide/python/execution.md index 0d111fa..6ef02d6 100644 --- a/docs/user-guide/python/execution.md +++ b/docs/user-guide/python/execution.md @@ -15,8 +15,8 @@ pan-os-upgrade firewall Firewall hostname or IP: houston.cdot.io Firewall username: cdot Firewall password: -Target version: 10.2.4-h4 -Dry Run? [y/N]: N +Target version: 11.1.1 +Dry Run? [y/N]: =================================================================== Welcome to the PAN-OS upgrade tool @@ -27,37 +27,89 @@ Create a settings.yaml file with 'pan-os-upgrade settings' command. =================================================================== 📝 houston: 007054000242050 192.168.255.211 📝 houston: HA mode: disabled -📝 houston: Current version: 10.2.4-h3 -📝 houston: Target version: 10.2.4-h4 -✅ houston: Upgrade required from 10.2.4-h3 to 10.2.4-h4 -✅ houston: version 10.2.4-h4 is available for download -✅ houston: Base image for 10.2.4-h4 is already downloaded -🚀 houston: Performing test to see if 10.2.4-h4 is already downloaded... -✅ houston: version 10.2.4-h4 already on target device. -✅ houston: version 10.2.4-h4 has been downloaded. -🚀 houston: Performing snapshot of network state information... -✅ houston: Network snapshot created successfully -🚀 houston: Performing readiness checks to determine if firewall is ready for upgrade... +📝 houston: Current version: 10.1.3 +📝 houston: Target version: 11.1.1 +✅ houston: Upgrade required from 10.1.3 to 11.1.1 +🔧 houston: Refreshing list of available software versions +✅ houston: version 11.1.1 is available for download +❌ houston: Base image for 11.1.1 is not downloaded. Attempting download. +🔍 houston: version 11.1.0 is not on the target device +🚀 houston: version 11.1.0 is beginning download +Device 007054000242050 downloading version: 11.1.0 +🔧 houston: Downloading version 11.1.0 - Elapsed time: 3 seconds +🔧 houston: Downloading version 11.1.0 - Elapsed time: 37 seconds +🔧 houston: Downloading version 11.1.0 - Elapsed time: 69 seconds +🔧 houston: Downloading version 11.1.0 - Elapsed time: 102 seconds +🔧 houston: Downloading version 11.1.0 - Elapsed time: 134 seconds +✅ houston: 11.1.0 downloaded in 167 seconds +✅ houston: Base image 11.1.0 downloaded successfully +✅ houston: Pausing for 60 seconds to let 11.1.0 image load into the software manager before downloading 11.1.1 +📝 houston: Current version: 10.1.3 +📝 houston: Target version: 11.1.1 +✅ houston: Upgrade required from 10.1.3 to 11.1.1 +🔧 houston: Refreshing list of available software versions +✅ houston: version 11.1.1 is available for download +✅ houston: Base image for 11.1.1 is already downloaded +🚀 houston: Performing test to see if 11.1.1 is already downloaded. +🔍 houston: version 11.1.1 is not on the target device +🚀 houston: version 11.1.1 is beginning download +Device 007054000242050 downloading version: 11.1.1 +🔧 houston: Downloading version 11.1.1 - Elapsed time: 6 seconds +🔧 houston: Downloading version 11.1.1 - Elapsed time: 40 seconds +🔧 houston: Downloading version 11.1.1 - Elapsed time: 74 seconds +✅ houston: 11.1.1 downloaded in 110 seconds +✅ houston: version 11.1.1 has been downloaded. +🚀 houston: Performing snapshot of network state information. +🚀 houston: Attempting to capture network state snapshot (Attempt 1 of 3). +✅ houston: Network snapshot created successfully on attempt 1. +💾 houston: Network state snapshot collected and saved to assurance/snapshots/houston/pre/2024-02-04_09-19-25.json +🚀 houston: Performing readiness checks to determine if firewall is ready for upgrade. +✅ houston: Passed Readiness Check: Check if active support is available +🟨 houston: Skipped Readiness Check: Check if a given ARP entry is available in the ARP table ✅ houston: Passed Readiness Check: Check if there are pending changes on device +🟨 houston: Skipped Readiness Check: Check if the certificates' keys meet minimum size requirements +🟨 houston: Skipped Readiness Check: Running Latest Content Version +✅ houston: Passed Readiness Check: Check if any Dynamic Update job is scheduled to run within the specified time window ✅ houston: Passed Readiness Check: No Expired Licenses -✅ houston: Passed Readiness Check: Check if NTP is synchronized +✅ houston: Passed Readiness Check: Check if a there is enough space on the `/opt/panrepo` volume for downloading an PanOS image. +🟨 houston: Skipped Readiness Check: Checks HA pair status from the perspective of the current device +🟨 houston: Skipped Readiness Check: Check if a given IPsec tunnel is in active state +🟨 houston: Skipped Readiness Check: Check for any job with status different than FIN +🟨 houston: Skipped Readiness Check: Check if NTP is synchronized +🟨 houston: Skipped Readiness Check: Check if the clock is synchronized between dataplane and management plane ✅ houston: Passed Readiness Check: Check connectivity with the Panorama appliance +🟨 houston: Skipped Readiness Check: Check if a critical session is present in the sessions table ✅ houston: Readiness Checks completed -🚀 houston: Performing backup of configuration to local filesystem... -🚀 houston: Not a dry run, continue with upgrade... -🚀 houston: Performing upgrade to version 10.2.4-h4... -🚀 houston: Attempting upgrade to version 10.2.4-h4 (Attempt 1 of 3)... -Device 007054000242050 installing version: 10.2.4-h4 +🚀 houston: Performing backup of configuration to local filesystem. +📝 houston: Not a dry run, continue with upgrade. +🚀 houston: Performing upgrade to version 11.1.1. +📝 houston: The install will take several minutes, check for status details within the GUI. +🚀 houston: Attempting upgrade to version 11.1.1 (Attempt 1 of 3). +Device 007054000242050 installing version: 11.1.1 +❌ houston: Upgrade error: Device 007054000242050 attempt to install version 11.1.1 failed: ['Failed to install 11.1.1 with the following errors.\nSW version is 11.1.1\nThe software manager is currently in use. Please try again later.\nFailed to install version 11.1.1 type panos\n\n'] +🟧 houston: Software manager is busy. Retrying in 60 seconds. +🚀 houston: Attempting upgrade to version 11.1.1 (Attempt 2 of 3). +Device 007054000242050 installing version: 11.1.1 ✅ houston: Upgrade completed successfully -🚀 houston: Rebooting the standalone target device... +🚀 houston: Rebooting the target device. 📝 houston: Command succeeded with no output -🔧 houston: Target device is rebooting... -🔧 houston: Target device is rebooting... -🔧 houston: Target device is rebooting... -🔧 houston: Target device is rebooting... -🔧 houston: Target device is rebooting... -📝 houston: Target device version: 10.2.4-h4 -✅ houston: Target device rebooted in 448 seconds +🟧 houston: Retry attempt 1 due to error: URLError: reason: [Errno 60] Operation timed out +🟧 houston: Retry attempt 2 due to error: URLError: reason: [Errno 60] Operation timed out +🟧 houston: Retry attempt 3 due to error: URLError: reason: [Errno 61] Connection refused +🟧 houston: Retry attempt 4 due to error: URLError: reason: [Errno 61] Connection refused +🟧 houston: Retry attempt 5 due to error: URLError: reason: [Errno 61] Connection refused +🟧 houston: Retry attempt 6 due to error: URLError: code: 403 reason: API Error: Invalid Credential +🟧 houston: Retry attempt 7 due to error: URLError: code: 403 reason: API Error: Invalid Credential +🟧 houston: Retry attempt 8 due to error: URLError: code: 403 reason: API Error: Invalid Credential +📝 houston: Current device version: 11.1.1 +✅ houston: Device rebooted to the target version successfully. +🚀 houston: Performing backup of configuration to local filesystem. +🔧 houston: Waiting for the device to become ready for the post upgrade snapshot. +🚀 houston: Performing snapshot of network state information. +🚀 houston: Attempting to capture network state snapshot (Attempt 1 of 3). +✅ houston: Network snapshot created successfully on attempt 1. +💾 houston: Network state snapshot collected and saved to assurance/snapshots/houston/post/2024-02-04_09-44-21.json +💾 houston: Snapshot comparison PDF report saved to assurance/snapshots/houston/diff/2024-02-04_09-44-25_report.pdf ```
@@ -140,155 +192,174 @@ No settings.yaml file was found. Default values will be used. Create a settings.yaml file with 'pan-os-upgrade settings' command. =========================================================================== ✅ panorama.cdot.io: Connection to Panorama established. Firewall connections will be proxied! -📝 Woodlands-fw2: 007954001234562 192.168.255.44 -🚀 Woodlands-fw2: Getting 007954001234562 deployment information... -📝 Woodlands-fw1: 007954001234561 192.168.255.43 -🚀 Woodlands-fw1: Getting 007954001234561 deployment information... -📝 Woodlands-fw2: Target device deployment: passive -📝 Woodlands-fw2: HA mode: passive -🚀 Woodlands-fw2: Getting 007954001234562 deployment information... -📝 Woodlands-fw1: Target device deployment: active -📝 Woodlands-fw1: HA mode: active -🚀 Woodlands-fw1: Getting 007954001234561 deployment information... -📝 Woodlands-fw2: Target device deployment: passive -📝 Woodlands-fw1: Target device deployment: active -📝 Woodlands-fw2: Local state: passive, Local version: 10.1.3, Peer version: 10.1.3 -📝 Woodlands-fw1: Local state: active, Local version: 10.1.3, Peer version: 10.1.3 +📝 Woodlands-fw2: 007954000987652 192.168.255.44 +📝 Woodlands-fw1: 007954000987651 192.168.255.43 +📝 Woodlands-fw2: HA mode: active +📝 Woodlands-fw1: HA mode: passive +📝 Woodlands-fw2: Local state: active, Local version: 10.1.3, Peer version: 10.1.3 📝 Woodlands-fw2: Version comparison: equal +🔍 Woodlands-fw2: Detected active target device in HA pair running the same version as its peer. Added target device to revisit list. +📝 Woodlands-fw1: Local state: passive, Local version: 10.1.3, Peer version: 10.1.3 📝 Woodlands-fw1: Version comparison: equal -📝 Woodlands-fw2: Target device is passive -🔍 Woodlands-fw1: Detected active target device in HA pair running the same version as its peer. Added target device to revisit list. -📝 Woodlands-fw2: Current version: 10.1.3 -📝 Woodlands-fw2: Target version: 10.2.7-h3 -✅ Woodlands-fw2: Upgrade required from 10.1.3 to 10.2.7-h3 -✅ Woodlands-fw2: version 10.2.7-h3 is available for download -❌ Woodlands-fw2: Base image for 10.2.7-h3 is not downloaded. Attempting download... -🔍 Woodlands-fw2: version 10.2.0 is not on the target device -🚀 Woodlands-fw2: version 10.2.0 is beginning download -Device 007954001234562 downloading version: 10.2.0 -🔧 Woodlands-fw2: Downloading version 10.2.0 - HA will sync image - Elapsed time: 3 seconds -🔧 Woodlands-fw2: Downloading version 10.2.0 - HA will sync image - Elapsed time: 34 seconds -🔧 Woodlands-fw2: Downloading version 10.2.0 - HA will sync image - Elapsed time: 67 seconds -🔧 Woodlands-fw2: Downloading version 10.2.0 - HA will sync image - Elapsed time: 99 seconds -🔧 Woodlands-fw2: Downloading version 10.2.0 - HA will sync image - Elapsed time: 131 seconds -🔧 Woodlands-fw2: Downloading version 10.2.0 - HA will sync image - Elapsed time: 164 seconds -🔧 Woodlands-fw2: Downloading version 10.2.0 - HA will sync image - Elapsed time: 196 seconds -🔧 Woodlands-fw2: Downloading version 10.2.0 - HA will sync image - Elapsed time: 227 seconds -🔧 Woodlands-fw2: Downloading version 10.2.0 - HA will sync image - Elapsed time: 258 seconds -🔧 Woodlands-fw2: Downloading version 10.2.0 - HA will sync image - Elapsed time: 290 seconds -🔧 Woodlands-fw2: Downloading version 10.2.0 - HA will sync image - Elapsed time: 322 seconds -🔧 Woodlands-fw2: Downloading version 10.2.0 - HA will sync image - Elapsed time: 353 seconds -🔧 Woodlands-fw2: Downloading version 10.2.0 - HA will sync image - Elapsed time: 386 seconds -✅ Woodlands-fw2: 10.2.0 downloaded in 418 seconds -✅ Woodlands-fw2: Base image 10.2.0 downloaded successfully -✅ Woodlands-fw2: Pausing for 60 seconds to let 10.2.0 image load into the software manager before downloading 10.2.7-h3 -📝 Woodlands-fw2: Current version: 10.1.3 -📝 Woodlands-fw2: Target version: 10.2.7-h3 -✅ Woodlands-fw2: Upgrade required from 10.1.3 to 10.2.7-h3 -✅ Woodlands-fw2: version 10.2.7-h3 is available for download -✅ Woodlands-fw2: Base image for 10.2.7-h3 is already downloaded -🚀 Woodlands-fw2: Performing test to see if 10.2.7-h3 is already downloaded... -🔍 Woodlands-fw2: version 10.2.7-h3 is not on the target device -🚀 Woodlands-fw2: version 10.2.7-h3 is beginning download -Device 007954001234562 downloading version: 10.2.7-h3 -🔧 Woodlands-fw2: Downloading version 10.2.7-h3 - HA will sync image - Elapsed time: 3 seconds -🔧 Woodlands-fw2: Downloading version 10.2.7-h3 - HA will sync image - Elapsed time: 36 seconds -🔧 Woodlands-fw2: Downloading version 10.2.7-h3 - HA will sync image - Elapsed time: 67 seconds -🔧 Woodlands-fw2: Downloading version 10.2.7-h3 - HA will sync image - Elapsed time: 99 seconds -🔧 Woodlands-fw2: Downloading version 10.2.7-h3 - HA will sync image - Elapsed time: 132 seconds -🔧 Woodlands-fw2: Downloading version 10.2.7-h3 - HA will sync image - Elapsed time: 163 seconds -🔧 Woodlands-fw2: Downloading version 10.2.7-h3 - HA will sync image - Elapsed time: 195 seconds -🔧 Woodlands-fw2: Downloading version 10.2.7-h3 - HA will sync image - Elapsed time: 227 seconds -🔧 Woodlands-fw2: Downloading version 10.2.7-h3 - HA will sync image - Elapsed time: 260 seconds -✅ Woodlands-fw2: 10.2.7-h3 downloaded in 291 seconds -✅ Woodlands-fw2: 10.2.7-h3 has been downloaded and sync'd to HA peer. -🚀 Woodlands-fw2: Performing snapshot of network state information... -✅ Woodlands-fw2: Network snapshot created successfully -🚀 Woodlands-fw2: Performing readiness checks to determine if firewall is ready for upgrade... -✅ Woodlands-fw2: Passed Readiness Check: Check if there are pending changes on device -✅ Woodlands-fw2: Passed Readiness Check: No Expired Licenses -✅ Woodlands-fw2: Passed Readiness Check: Checks HA pair status from the perspective of the current device -✅ Woodlands-fw2: Passed Readiness Check: Check if NTP is synchronized -✅ Woodlands-fw2: Passed Readiness Check: Check if the clock is synchronized between dataplane and management plane -✅ Woodlands-fw2: Passed Readiness Check: Check connectivity with the Panorama appliance -✅ Woodlands-fw2: Readiness Checks completed -🚀 Woodlands-fw2: Checking if HA peer is in sync... -✅ Woodlands-fw2: HA peer sync test has been completed. -🚀 Woodlands-fw2: Performing backup of configuration to local filesystem... -🚀 Woodlands-fw2: Not a dry run, continue with upgrade... -🚀 Woodlands-fw2: Performing upgrade to version 10.2.7-h3... -🚀 Woodlands-fw2: Attempting upgrade to version 10.2.7-h3 (Attempt 1 of 3)... -Device 007954001234562 installing version: 10.2.7-h3 -✅ Woodlands-fw2: Upgrade completed successfully -🚀 Woodlands-fw2: Rebooting the target device... -📝 Woodlands-fw2: Command succeeded with no output -🟧 Woodlands-fw2: Retry attempt 1 due to error: 007954001234562 not connected -🟧 Woodlands-fw2: Retry attempt 2 due to error: 007954001234562 not connected -🟧 Woodlands-fw2: Retry attempt 3 due to error: 007954001234562 not connected -🟧 Woodlands-fw2: Retry attempt 4 due to error: 007954001234562 not connected -🟧 Woodlands-fw2: Retry attempt 5 due to error: 007954001234562 not connected -🟧 Woodlands-fw2: Retry attempt 6 due to error: 007954001234562 not connected -🟧 Woodlands-fw2: Retry attempt 7 due to error: 007954001234562 not connected -🟧 Woodlands-fw2: Retry attempt 8 due to error: 007954001234562 not connected -📝 Woodlands-fw2: Current device version: 10.2.7-h3 -✅ Woodlands-fw2: Device rebooted to the target version successfully. -🚀 panorama.cdot.io: Revisiting firewalls that were active in an HA pair and had the same version as their peers. -📝 Woodlands-fw1: 007954001234561 192.168.255.43 -🚀 Woodlands-fw1: Getting 007954001234561 deployment information... -📝 Woodlands-fw1: Target device deployment: active -📝 Woodlands-fw1: HA mode: active -🚀 Woodlands-fw1: Getting 007954001234561 deployment information... -📝 Woodlands-fw1: Target device deployment: active -📝 Woodlands-fw1: Local state: active, Local version: 10.1.3, Peer version: 10.1.3 -Waiting for HA synchronization to complete on Woodlands-fw1. Attempt 1/3 -🚀 Woodlands-fw1: Getting 007954001234561 deployment information... -📝 Woodlands-fw1: Target device deployment: active -HA synchronization still in progress on Woodlands-fw1. Rechecking after wait period. -Waiting for HA synchronization to complete on Woodlands-fw1. Attempt 2/3 -🚀 Woodlands-fw1: Getting 007954001234561 deployment information... -📝 Woodlands-fw1: Target device deployment: non-functional -HA synchronization complete on Woodlands-fw1. Proceeding with upgrade. -📝 Woodlands-fw1: Version comparison: older -📝 Woodlands-fw1: Target device is on an older version -📝 Woodlands-fw1: Suspending HA state of active -❌ Woodlands-fw1: Error suspending active target device HA state: argument of type 'NoneType' is not iterable +📝 Woodlands-fw1: Target device is passive 📝 Woodlands-fw1: Current version: 10.1.3 📝 Woodlands-fw1: Target version: 10.2.7-h3 ✅ Woodlands-fw1: Upgrade required from 10.1.3 to 10.2.7-h3 +🔧 Woodlands-fw1: Refreshing list of available software versions +✅ Woodlands-fw1: version 10.2.7-h3 is available for download +❌ Woodlands-fw1: Base image for 10.2.7-h3 is not downloaded. Attempting download. +🔍 Woodlands-fw1: version 10.2.0 is not on the target device +🚀 Woodlands-fw1: version 10.2.0 is beginning download +Device 007954000987651 downloading version: 10.2.0 +🔧 Woodlands-fw1: Downloading version 10.2.0 - HA will sync image - Elapsed time: 3 seconds +🔧 Woodlands-fw1: Downloading version 10.2.0 - HA will sync image - Elapsed time: 35 seconds +🔧 Woodlands-fw1: Downloading version 10.2.0 - HA will sync image - Elapsed time: 66 seconds +🔧 Woodlands-fw1: Downloading version 10.2.0 - HA will sync image - Elapsed time: 98 seconds +🔧 Woodlands-fw1: Downloading version 10.2.0 - HA will sync image - Elapsed time: 129 seconds +🔧 Woodlands-fw1: Downloading version 10.2.0 - HA will sync image - Elapsed time: 160 seconds +🔧 Woodlands-fw1: Downloading version 10.2.0 - HA will sync image - Elapsed time: 192 seconds +🔧 Woodlands-fw1: Downloading version 10.2.0 - HA will sync image - Elapsed time: 223 seconds +🔧 Woodlands-fw1: Downloading version 10.2.0 - HA will sync image - Elapsed time: 257 seconds +🔧 Woodlands-fw1: Downloading version 10.2.0 - HA will sync image - Elapsed time: 289 seconds +✅ Woodlands-fw1: 10.2.0 downloaded in 321 seconds +✅ Woodlands-fw1: Base image 10.2.0 downloaded successfully +✅ Woodlands-fw1: Pausing for 60 seconds to let 10.2.0 image load into the software manager before downloading 10.2.7-h3 +📝 Woodlands-fw1: Current version: 10.1.3 +📝 Woodlands-fw1: Target version: 10.2.7-h3 +✅ Woodlands-fw1: Upgrade required from 10.1.3 to 10.2.7-h3 +🔧 Woodlands-fw1: Refreshing list of available software versions ✅ Woodlands-fw1: version 10.2.7-h3 is available for download ✅ Woodlands-fw1: Base image for 10.2.7-h3 is already downloaded -🚀 Woodlands-fw1: Performing test to see if 10.2.7-h3 is already downloaded... -✅ Woodlands-fw1: version 10.2.7-h3 already on target device. +🚀 Woodlands-fw1: Performing test to see if 10.2.7-h3 is already downloaded. +🔍 Woodlands-fw1: version 10.2.7-h3 is not on the target device +🚀 Woodlands-fw1: version 10.2.7-h3 is beginning download +Device 007954000987651 downloading version: 10.2.7-h3 +🔧 Woodlands-fw1: Downloading version 10.2.7-h3 - HA will sync image - Elapsed time: 3 seconds +🔧 Woodlands-fw1: Downloading version 10.2.7-h3 - HA will sync image - Elapsed time: 35 seconds +🔧 Woodlands-fw1: Downloading version 10.2.7-h3 - HA will sync image - Elapsed time: 67 seconds +🔧 Woodlands-fw1: Downloading version 10.2.7-h3 - HA will sync image - Elapsed time: 103 seconds +🔧 Woodlands-fw1: Downloading version 10.2.7-h3 - HA will sync image - Elapsed time: 135 seconds +🔧 Woodlands-fw1: Downloading version 10.2.7-h3 - HA will sync image - Elapsed time: 168 seconds +🔧 Woodlands-fw1: Downloading version 10.2.7-h3 - HA will sync image - Elapsed time: 201 seconds +✅ Woodlands-fw1: 10.2.7-h3 downloaded in 233 seconds ✅ Woodlands-fw1: 10.2.7-h3 has been downloaded and sync'd to HA peer. -🚀 Woodlands-fw1: Performing snapshot of network state information... -✅ Woodlands-fw1: Network snapshot created successfully -🚀 Woodlands-fw1: Performing readiness checks to determine if firewall is ready for upgrade... +🚀 Woodlands-fw1: Performing snapshot of network state information. +🚀 Woodlands-fw1: Attempting to capture network state snapshot (Attempt 1 of 3). +✅ Woodlands-fw1: Network snapshot created successfully on attempt 1. +💾 Woodlands-fw1: Network state snapshot collected and saved to assurance/snapshots/Woodlands-fw1/pre/2024-02-04_09-15-40.json +🚀 Woodlands-fw1: Performing readiness checks to determine if firewall is ready for upgrade. +✅ Woodlands-fw1: Passed Readiness Check: Check if active support is available +🟨 Woodlands-fw1: Skipped Readiness Check: Check if a given ARP entry is available in the ARP table ✅ Woodlands-fw1: Passed Readiness Check: Check if there are pending changes on device +🟨 Woodlands-fw1: Skipped Readiness Check: Check if the certificates' keys meet minimum size requirements +🟨 Woodlands-fw1: Skipped Readiness Check: Running Latest Content Version +✅ Woodlands-fw1: Passed Readiness Check: Check if any Dynamic Update job is scheduled to run within the specified time window ✅ Woodlands-fw1: Passed Readiness Check: No Expired Licenses -✅ Woodlands-fw1: Passed Readiness Check: Check if NTP is synchronized -✅ Woodlands-fw1: Passed Readiness Check: Check if the clock is synchronized between dataplane and management plane +🟨 Woodlands-fw1: Skipped Readiness Check: Check if a there is enough space on the `/opt/panrepo` volume for downloading an PanOS image. +✅ Woodlands-fw1: Passed Readiness Check: Checks HA pair status from the perspective of the current device +🟨 Woodlands-fw1: Skipped Readiness Check: Check if a given IPsec tunnel is in active state +🟨 Woodlands-fw1: Skipped Readiness Check: Check for any job with status different than FIN +🟨 Woodlands-fw1: Skipped Readiness Check: Check if NTP is synchronized +🟨 Woodlands-fw1: Skipped Readiness Check: Check if the clock is synchronized between dataplane and management plane ✅ Woodlands-fw1: Passed Readiness Check: Check connectivity with the Panorama appliance +🟨 Woodlands-fw1: Skipped Readiness Check: Check if a critical session is present in the sessions table ✅ Woodlands-fw1: Readiness Checks completed -🚀 Woodlands-fw1: Checking if HA peer is in sync... +🚀 Woodlands-fw1: Checking if HA peer is in sync. ✅ Woodlands-fw1: HA peer sync test has been completed. -🚀 Woodlands-fw1: Performing backup of configuration to local filesystem... -🚀 Woodlands-fw1: Not a dry run, continue with upgrade... -🚀 Woodlands-fw1: Performing upgrade to version 10.2.7-h3... -🚀 Woodlands-fw1: Attempting upgrade to version 10.2.7-h3 (Attempt 1 of 3)... -Device 007954001234561 installing version: 10.2.7-h3 +🚀 Woodlands-fw1: Performing backup of configuration to local filesystem. +📝 Woodlands-fw1: Not a dry run, continue with upgrade. +🚀 Woodlands-fw1: Performing upgrade to version 10.2.7-h3. +📝 Woodlands-fw1: The install will take several minutes, check for status details within the GUI. +🚀 Woodlands-fw1: Attempting upgrade to version 10.2.7-h3 (Attempt 1 of 3). +Device 007954000987651 installing version: 10.2.7-h3 ✅ Woodlands-fw1: Upgrade completed successfully -🚀 Woodlands-fw1: Rebooting the target device... +🚀 Woodlands-fw1: Rebooting the target device. 📝 Woodlands-fw1: Command succeeded with no output -🟧 Woodlands-fw1: Retry attempt 1 due to error: 007954001234561 not connected -🟧 Woodlands-fw1: Retry attempt 2 due to error: 007954001234561 not connected -🟧 Woodlands-fw1: Retry attempt 3 due to error: 007954001234561 not connected -🟧 Woodlands-fw1: Retry attempt 4 due to error: 007954001234561 not connected -🟧 Woodlands-fw1: Retry attempt 5 due to error: 007954001234561 not connected -🟧 Woodlands-fw1: Retry attempt 6 due to error: 007954001234561 not connected -🟧 Woodlands-fw1: Retry attempt 7 due to error: 007954001234561 not connected -🟧 Woodlands-fw1: Retry attempt 8 due to error: 007954001234561 not connected +🟧 Woodlands-fw1: Retry attempt 1 due to error: 007954000987651 not connected +🟧 Woodlands-fw1: Retry attempt 2 due to error: 007954000987651 not connected +🟧 Woodlands-fw1: Retry attempt 3 due to error: 007954000987651 not connected +🟧 Woodlands-fw1: Retry attempt 4 due to error: 007954000987651 not connected +🟧 Woodlands-fw1: Retry attempt 5 due to error: 007954000987651 not connected +🟧 Woodlands-fw1: Retry attempt 6 due to error: 007954000987651 not connected +🟧 Woodlands-fw1: Retry attempt 7 due to error: 007954000987651 not connected +🟧 Woodlands-fw1: Retry attempt 8 due to error: 007954000987651 not connected +🟧 Woodlands-fw1: Retry attempt 9 due to error: 007954000987651 not connected 📝 Woodlands-fw1: Current device version: 10.2.7-h3 ✅ Woodlands-fw1: Device rebooted to the target version successfully. +🚀 Woodlands-fw1: Performing backup of configuration to local filesystem. +🔧 Woodlands-fw1: Waiting for the device to become ready for the post upgrade snapshot. +🚀 Woodlands-fw1: Performing snapshot of network state information. +🚀 Woodlands-fw1: Attempting to capture network state snapshot (Attempt 1 of 3). +✅ Woodlands-fw1: Network snapshot created successfully on attempt 1. +💾 Woodlands-fw1: Network state snapshot collected and saved to assurance/snapshots/Woodlands-fw1/post/2024-02-04_09-35-39.json +💾 Woodlands-fw1: Snapshot comparison PDF report saved to assurance/snapshots/Woodlands-fw1/diff/2024-02-04_09-35-40_report.pdf +🚀 panorama.cdot.io: Revisiting firewalls that were active in an HA pair and had the same version as their peers. +📝 Woodlands-fw2: 007954000987652 192.168.255.44 +📝 Woodlands-fw2: HA mode: non-functional +📝 Woodlands-fw2: Local state: non-functional, Local version: 10.1.3, Peer version: 10.2.7-h3 +Waiting for HA synchronization to complete on Woodlands-fw2. Attempt 1/3 +HA synchronization complete on Woodlands-fw2. Proceeding with upgrade. +📝 Woodlands-fw2: Version comparison: older +📝 Woodlands-fw2: Target device is on an older version +📝 Woodlands-fw2: Current version: 10.1.3 +📝 Woodlands-fw2: Target version: 10.2.7-h3 +✅ Woodlands-fw2: Upgrade required from 10.1.3 to 10.2.7-h3 +🔧 Woodlands-fw2: Refreshing list of available software versions +✅ Woodlands-fw2: version 10.2.7-h3 is available for download +✅ Woodlands-fw2: Base image for 10.2.7-h3 is already downloaded +🚀 Woodlands-fw2: Performing test to see if 10.2.7-h3 is already downloaded. +✅ Woodlands-fw2: version 10.2.7-h3 already on target device. +✅ Woodlands-fw2: version 10.2.7-h3 has been downloaded. +🚀 Woodlands-fw2: Performing snapshot of network state information. +🚀 Woodlands-fw2: Attempting to capture network state snapshot (Attempt 1 of 3). +✅ Woodlands-fw2: Network snapshot created successfully on attempt 1. +💾 Woodlands-fw2: Network state snapshot collected and saved to assurance/snapshots/Woodlands-fw2/pre/2024-02-04_09-36-48.json +🚀 Woodlands-fw2: Performing readiness checks to determine if firewall is ready for upgrade. +✅ Woodlands-fw2: Passed Readiness Check: Check if active support is available +🟨 Woodlands-fw2: Skipped Readiness Check: Check if a given ARP entry is available in the ARP table +✅ Woodlands-fw2: Passed Readiness Check: Check if there are pending changes on device +🟨 Woodlands-fw2: Skipped Readiness Check: Check if the certificates' keys meet minimum size requirements +🟨 Woodlands-fw2: Skipped Readiness Check: Running Latest Content Version +✅ Woodlands-fw2: Passed Readiness Check: Check if any Dynamic Update job is scheduled to run within the specified time window +✅ Woodlands-fw2: Passed Readiness Check: No Expired Licenses +🟨 Woodlands-fw2: Skipped Readiness Check: Check if a there is enough space on the `/opt/panrepo` volume for downloading an PanOS image. +🟨 Woodlands-fw2: Skipped Readiness Check: Checks HA pair status from the perspective of the current device +🟨 Woodlands-fw2: Skipped Readiness Check: Check if a given IPsec tunnel is in active state +🟨 Woodlands-fw2: Skipped Readiness Check: Check for any job with status different than FIN +🟨 Woodlands-fw2: Skipped Readiness Check: Check if NTP is synchronized +✅ Woodlands-fw2: Passed Readiness Check: Check if the clock is synchronized between dataplane and management plane +✅ Woodlands-fw2: Passed Readiness Check: Check connectivity with the Panorama appliance +🟨 Woodlands-fw2: Skipped Readiness Check: Check if a critical session is present in the sessions table +✅ Woodlands-fw2: Readiness Checks completed +🚀 Woodlands-fw2: Checking if HA peer is in sync. +🟧 Woodlands-fw2: HA peer state is not in sync. This will be noted, but the script will continue. +🚀 Woodlands-fw2: Performing backup of configuration to local filesystem. +📝 Woodlands-fw2: Not a dry run, continue with upgrade. +🚀 Woodlands-fw2: Performing upgrade to version 10.2.7-h3. +📝 Woodlands-fw2: The install will take several minutes, check for status details within the GUI. +🚀 Woodlands-fw2: Attempting upgrade to version 10.2.7-h3 (Attempt 1 of 3). +Device 007954000987652 installing version: 10.2.7-h3 +✅ Woodlands-fw2: Upgrade completed successfully +🚀 Woodlands-fw2: Rebooting the target device. +📝 Woodlands-fw2: Command succeeded with no output +🟧 Woodlands-fw2: Retry attempt 1 due to error: 007954000987652 not connected +🟧 Woodlands-fw2: Retry attempt 2 due to error: 007954000987652 not connected +🟧 Woodlands-fw2: Retry attempt 3 due to error: 007954000987652 not connected +🟧 Woodlands-fw2: Retry attempt 4 due to error: 007954000987652 not connected +🟧 Woodlands-fw2: Retry attempt 5 due to error: 007954000987652 not connected +🟧 Woodlands-fw2: Retry attempt 6 due to error: 007954000987652 not connected +🟧 Woodlands-fw2: Retry attempt 7 due to error: 007954000987652 not connected +🟧 Woodlands-fw2: Retry attempt 8 due to error: 007954000987652 not connected +🟧 Woodlands-fw2: Retry attempt 9 due to error: 007954000987652 not connected +📝 Woodlands-fw2: Current device version: 10.2.7-h3 +✅ Woodlands-fw2: Device rebooted to the target version successfully. +🚀 Woodlands-fw2: Performing backup of configuration to local filesystem. +🔧 Woodlands-fw2: Waiting for the device to become ready for the post upgrade snapshot. +🚀 Woodlands-fw2: Performing snapshot of network state information. +🚀 Woodlands-fw2: Attempting to capture network state snapshot (Attempt 1 of 3). +✅ Woodlands-fw2: Network snapshot created successfully on attempt 1. +💾 Woodlands-fw2: Network state snapshot collected and saved to assurance/snapshots/Woodlands-fw2/post/2024-02-04_09-57-36.json +💾 Woodlands-fw2: Snapshot comparison PDF report saved to assurance/snapshots/Woodlands-fw2/diff/2024-02-04_09-57-38_report.pdf ✅ panorama.cdot.io: Completed revisiting firewalls ``` @@ -320,6 +391,60 @@ pan-os-upgrade batch --hostname panorama.cdot.io --username admin --password sec If you would like to change the default settings of `pan-os-upgrade` tool, you can run the `settings` CLI argument. This will walk you through a series of options to change. +### Readiness Checks + +The following table lists the available readiness checks, their descriptions, and whether they are enabled by default. These checks are designed to ensure the device's readiness for an upgrade by validating its operational and configuration status. + +| Readiness Check | Description | Enabled by Default | +| --------------------------- | ----------------------------------------------------------------------------------------- | :----------------: | +| `active_support` | Check if active support is available | Yes | +| `arp_entry_exist` | Check if a given ARP entry is available in the ARP table | No | +| `candidate_config` | Check if there are pending changes on device | Yes | +| `certificates_requirements` | Check if the certificates' keys meet minimum size requirements | No | +| `content_version` | Running Latest Content Version | Yes | +| `dynamic_updates` | Check if any Dynamic Update job is scheduled to run within the specified time window | Yes | +| `expired_licenses` | No Expired Licenses | Yes | +| `free_disk_space` | Check if there is enough space on the `/opt/panrepo` volume for downloading a PanOS image | Yes | +| `ha` | Checks HA pair status from the perspective of the current device | Yes | +| `ip_sec_tunnel_status` | Check if a given IPsec tunnel is in active state | Yes | +| `jobs` | Check for any job with status different than FIN | No | +| `ntp_sync` | Check if NTP is synchronized | No | +| `panorama` | Check connectivity with the Panorama appliance | Yes | +| `planes_clock_sync` | Check if the clock is synchronized between dataplane and management plane | Yes | +| `session_exist` | Check if a critical session is present in the sessions table | No | + +### State Snapshots + +The following table lists the categories of state snapshots that can be captured to document essential data about the device's current state. These snapshots are crucial for diagnostics and verifying the device's operational status before proceeding with the upgrade. + +| Snapshot | Description | Enabled by Default | +| ----------------- | ----------------------------------- | :----------------: | +| `arp_table` | Snapshot of the ARP Table | Yes | +| `content_version` | Snapshot of the Content Version | Yes | +| `ip_sec_tunnels` | Snapshot of the IPsec Tunnels | No | +| `license` | Snapshot of the License Information | Yes | +| `nics` | Snapshot of the Network Interfaces | Yes | +| `routes` | Snapshot of the Routing Table | Yes | +| `session_stats` | Snapshot of the Session Statistics | No | + +### Customizing Default Settings + +The default settings for readiness checks and snapshots can be customized using the `pan-os-upgrade settings` subcommand. This interactive command guides you through a series of prompts to configure various aspects of the script's behavior, including which readiness checks and snapshots are enabled. + +To override the default settings: + +1. Run the `pan-os-upgrade settings` command. +2. Follow the prompts to enable or disable specific readiness checks and snapshots. +3. The resulting configurations are saved to a `settings.yaml` file in the current working directory. + + ```bash + pan-os-upgrade settings + ``` + +#### Note + +The `settings.yaml` file created by this command can be edited manually for further customization. +
```console @@ -406,8 +531,6 @@ Custom configuration loaded from: 📝 houston: Target version: 10.2.5 ✅ houston: Upgrade required from 10.2.4-h4 to 10.2.5 ... shortened for brevity ... -🟧 houston: Retry attempt 4 due to error: URLError: reason: [Errno 111] Connection refused -📝 houston: Current device version: 10.2.5 ✅ houston: Device rebooted to the target version successfully. ``` diff --git a/docs/user-guide/python/troubleshooting.md b/docs/user-guide/python/troubleshooting.md index 4bb4c0d..4c89682 100644 --- a/docs/user-guide/python/troubleshooting.md +++ b/docs/user-guide/python/troubleshooting.md @@ -69,6 +69,21 @@ After generating the locale, you can verify it's available by running `locale -a If you cannot request `sudo` permissions within WSL2, either use the `pan-os-upgrade` script from the Windows CMD terminal, or use the Docker container. +### 10. ARP Table Comparison Failures + +**Problem:** When capturing ARP tables for comparison, the script fails with `WrongDataTypeException: Unknown value format for key ttl`. + +**Solution:** This issue can arise when ARP table entries contain integer values for `ttl`, which the current implementation may not handle properly. To address this, consider installing a custom fork of `panos-upgrade-assurance` that includes a fix for this issue, available at [https://github.com/cdot65/pan-os-upgrade-assurance/tree/main](https://github.com/cdot65/pan-os-upgrade-assurance/tree/main). Alternatively, you can configure the script to omit ARP snapshots from the tests if modifying the script is not feasible. + +**Steps to Install Custom Fork:** + +1. Run this command: `pip install git+https://github.com/cdot65/pan-os-upgrade-assurance.git@main` + +**Steps to Omit ARP Snapshots:** + +1. If using a `settings.yaml` file, ensure ARP snapshots are disabled. +2. If running the script interactively, choose not to capture ARP snapshots when prompted. + ## General Tips - Always perform a dry run (`--dry-run`) before executing the actual upgrade.