Saturday, July 18, 2020

Palo Alto firewall - How to Upgrade an High Availability (HA) Pair

Caution: I strongly recommend you refer to the Palo Alto Networks (PAN) official site first. This article will show you how to upgrade the firewall efficiently, based on my personal experience. It might not be suitable for your environment.
I do not make any warranties about the completeness, reliability and accuracy of this information. Any action you take upon the information on this website is strictly at your own risk.


Here are two methods of how to upgrade the Palo Alto Networks (PAN) firewall in High Availability (HA) pair.

Method 1 is my way to upgrade the firewall in order to save the upgrades time overall, and Method 2 is recommended by PAN.

Before you upgrade the firewall, you should determine the upgrade path to the PAN-OS image. When you upgrade from one PAN-OS feature release version to a later feature release, you cannot skip the installation of any feature release versions in the path to your target release.

Please refer to the preferred release, and choose a target PAN-OS version. PAN marks 'P' as the preferred release in release branch.

★ Current preferred PAN-OS version for Firewalls
9.1.4 (08/05/2020), 9.0.9-h1 (07/02/2020), 8.1.15-h3 (07/02/2020)

★ Current preferred PAN-OS version for Panorama on VM / M-series
9.1.4 (08/05/2020), 9.0.9 (06/24/2020), 8.1.15 (06/24/2020)

★ Current preferred version for GlobalProtect
5.1.3 (04/22/2020), 5.0.10 (06/09/2020)

* Note: PAN-OS 9.1 is a new feature release. Support generally prefers 9.0 or 8.1 unless new features are required.
Recommend to upgrade to content version 8221 prior to upgrading to 9.1 due to issue with long boot up time. Refer to customer advisory​ for details.

 Method 1 (without HA failover testing)

Step 1. Save a backup of the current configuration file.

1) Perform these steps on each firewall in the pair: Select Device > Setup > Operations and click Export named configuration snapshot.
2) Select the XML file that contains your running configuration (for example, running-config.xml ) and click OK to export the configuration file.
3) Save the exported file to a location external to the firewall. You can use this backup to restore the configuration if you have problems with the upgrade.

Step 2. Make sure each device running the recommended Content Release version.

1) Select Device > Dynamic Updates.
2) Check the Applications and Threats or Applications section to determine what update is currently running.
3) If the firewall is not running the required update or later, click Check Now to retrieve a list of available updates.
4) Locate the desired update and click Download.
5) After the download completes, click Install.

Step 3. Determine the upgrade path. (prerequisite)

You cannot skip installing any major release versions on the path to your desired PAN-OS version. Therefore, if you plan to upgrade to a version that is more than one major release away, you must still download, install, and reboot the firewall into all interim PAN-OS versions along the upgrade path.

For example, if you want to upgrade from PAN-OS 7.1.8 to PAN-OS 9.1.1, you must:
- Download and install PAN-OS 8.0.0 and reboot.
- Download and install PAN-OS 9.0.0 and reboot.
- Download and install PAN-OS 9.1.1 and reboot.

If you have the pair in HA(active/passive) then you have to upgrade only to next version of PAN-OS then failover and proceed to upgrade for the second version of PAN-OS. Only upgrade one version at a time. Example: If you are at PAN-OS 7.1.x then you should go to 8.0.x version(let it be any version of PAN-OS) then failover and check the functionality. Otherwise you will run into the error and the HA pairs will no longer be in sync.
Additionally Remember that if there is more than 1 version of difference between the HA pairs then you will run into the "Peer version too old" issue.

[† AnalysisMan Tip]

You do not need to reboot the firewall during the intermediate upgrades even though it is not the best practice and recommended by PAN. However, it works and you can reduce the upgrade time overall as below.

PAN-OS 7.1.8 -> (8.0.0 image download & install) 9.0.0 (image download & install) -> 9.1.1 (image download & install & reboot)

Skipped steps with the intermediate PAN-OS images as below.

Step 4. Install PAN-OS 9.1 on the passive device (active/passive)

1) Click Check Now to check for the latest updates.

* CLI Command
request system software check

2) Locate the version you want to upgrade to and then click Download.

request system software download version 9.1.1

3) Verify the downloaded version.

request system software info

admin@firewall2(passive)> request system software info

Version               Size          Released on Downloaded
9.1.1                312MB 2019/09/26  11:23:05        yes
9.0.3                309MB 2019/07/12  10:30:40         no

4) After the download completes, click Install.

request system software install version 9.1.1

admin@firewall2> request system software install version 9.1.9
Executing this command will install a new version of software. It will not take effect until system is restarted. Do you want to continue? (y or n) y

Software install job enqueued with jobid 3312. Run 'show jobs id 3312' to monitor its status. Please reboot the device after the installation is done.

[† AnalysisMan Tip]

You should be able to check if the target PAN-OS image is installed on the partition by the following CLI command. It shows as 'PENDING-CHANGE', and it will be changed to 'RUNNING-ACTIVE' after rebooting.

debug swm status

admin@firewall2(passive)> debug swm status

Partition         State             Version
sysroot0          RUNNING-ACTIVE    9.0.8
sysroot1          PENDING-CHANGE    9.1.1
maint             READY             9.1.1

Step 5. After the install completes, reboot using one of the following methods:

- If you are prompted to reboot, click Yes.
- If you are not prompted to reboot, select Device > Setup > Operations and click Reboot Device in the Device Operations section.

request restart system

Step 6. Verify that the active device is passing traffic

By viewing the Monitor > Session Browser, or by running show session all from the CLI.
· You can also check the HA state on the device with show high-availability all. On the active/active configuration, check that both devices are passing traffic.

show high-availability all
show high-availability interface ha2

· To check session synchronization run show high-availability interface ha2. In the Hardware Interface counters read from CPU table check that counters are increasing. In an active/passive configuration, only the active device will show packets transmitted and the passive device will only show packets received. In the active/active configuration, you will see packets received and packets transmitted on both devices.

show high-availability interface ha2 | match bytes

admin@ firewall1(active)> show high-availability interface ha2 | match bytes
rx-bytes                      191827728
tx-bytes                      1916678815290
bytes received                           114449284
bytes transmitted                        1885650036560
bytes received                           114449284
bytes transmitted                      1885650036804

admin@ firewall2(passive)> show high-availability interface ha2 | match bytes
rx-bytes                      1916674506264
tx-bytes                      108310016
bytes received                           1885689069756
bytes transmitted                        100558008
bytes received                           1885689069756
bytes transmitted                        100558008

Step 7. Suspend the active firewall for HA failover.

1) On the active (active/passive) or active-primary (active/active) device, select Device > High Availability > Operational Commands.

2) Click Suspend local device.

request high-availability state suspend

> request high-availability state suspend

3) Select Dashboard and verify that the state of the passive device changes to active in the High Availability widget.

4) Verify that the firewall that took over as active or active-primary is passing traffic by selecting Monitor > Session Browser.

or use CLI command.

show session all

admin@firewall2(active)> show session all

Step 8. Upgrade the suspended firewall.

Upgrade the firewall1, which is now suspended from active. Follow the above Step 4 ~ 6.

The firewall1 will be failed over automatically and running as active after rebooting, HA links up, sessions sync. I presumed that you have Primary (firewall1) is active with a lower priority value (e.g., 100), and Secondary (firewall2) is passive with a high priority value (e.g., 200).

* If both firewalls have the same device priority value, the firewall with the lowest MAC address on the HA1 control link will become the active firewall.

You must enable preemptive on both the active firewall and the passive firewall.

[† AnalysisMan Tip]

I use Method 1 because I am confident that the HA failover is functional and reduce the upgrade time overall.
Also, you may install the PAN-OS image on both active/passive firewalls before the maintenance window. And reboot the Secondary (firewall2) firewall, failover to Secondary firewall, and reboot the Primary (firewall1) firewall. That's it.

 Method 2 (with HA failover testing)

Here is the Palo Alto's Best Practices for PAN-OS Upgrade. (login required)

It is recommended to upgrade the Primary firewall first and then upgrade the Secondary firewall. This is done for two reasons:

1) Ensure that HA failover is functioning properly
2) Ensure that the passive firewall is functioning properly and is able to pass traffic without issues

Follow the steps in the above link.

No comments: