Saturday, July 18, 2020

Palo Alto firewall - How to Upgrade a High Availability (HA) Pair

Caution: I strongly recommend you refer to the Palo Alto Networks (PAN) official site first. This article will show you how to upgrade the firewall efficiently, based on my personal experience. It might not be suitable for your environment.
I do not make any warranties about the completeness, reliability and accuracy of this information. Any action you take upon the information on this website is strictly at your own risk.


Here are two methods of how to upgrade the Palo Alto Networks (PAN) firewall in High Availability (HA) pair.

Method 1 is my way to upgrade the firewall in order to save the upgrades time overall, and Method 2 is recommended by PAN.

Before you upgrade the firewall, you should determine the upgrade path to the PAN-OS image. When you upgrade from one PAN-OS feature release version to a later feature release, you cannot skip the installation of any feature release versions in the path to your target release.

        ※ Please refer to the Compatibility Matrix.

        ▶ Palo Alto Networks Compatibility Matrix

        Please refer to the preferred release, and choose a target PAN-OS version. PAN marks 'P' as the preferred release in the release branch. (as of 2/26/2024)

        ★ Current preferred PAN-OS version for Firewalls
        11.1.1 (12/26/2024), 11.0.3-h3 (01/16/2024), 10.2.7-h3 (12/18/2023), 10.1.11-h5 (01/23/2024), 10.0.12-h5 (01/08/2024), 9.1.17 (12/11/2023), 9.0.17-h5 (01/08/2024), 8.1.26-h1 (01/08/2024)

        Latest version: 11.1.2 (02/26/2024), 11.0.3-h5 (02/22/2024), 10.2.8 (01/12/2024), 10.1.12 (01/29/2024), 10.0.12-h5 (01/08/2024), 9.1.17-h1 (01/08/2024), 9.0.17-h5 (01/08/2024), 8.1.26-h1 (01/08/2024)

        ★ Current preferred PAN-OS version for Panorama on VM / M-series
        11.1.1 (12/26/2024), 11.0.3-h3 (01/16/2024), 10.2.7-h3 (12/18/2023), 10.1.11-h5 (01/23/2024), 10.0.12-h5 (01/08/2024), 9.1.17 (12/11/2023), 9.0.17-h5 (01/08/2024), 8.1.26-h1 (01/08/2024)

        ★ Current preferred version for GlobalProtect
        6.2.2 (11/22/2023), 6.1.3 (11/21/2023), 6.0.7 (06/22/2023), 5.2.13 (02/22/2023), 5.1.11 (05/12/2022)

        Latest version: 6.2.2 (11/22/2023), 6.1.4 (01/29/2024), 6.0.8 (10/18/2023), 5.2.13 (02/22/2023), 5.1.12 (02/12/2024)

        * Note:

        PAN-OS & Panorama EOL

        Version Release Date End-of-Life Date
        11.1 November 3, 2023 November 3, 2026
        11.0 November 17, 2022 November 17, 2024
        10.2+ February 27, 2022 August 27, 2025
        10.1+ May 31, 2021 December 1, 2024
        10.0+ July 16, 2020 July 16, 2022
        9.1+ December 13, 2019 June 30, 2024
        9.0-XFR (VM-Series only) September 19, 2019 September 19, 2020
        9.0 February 6, 2019 March 1, 2022
        8.1+ March 1, 2018 March 1, 2022
        8.0 January 29, 2017 October 31, 2019
        7.1 March 29, 2016 June 30, 2020
        7.0 June 4, 2015 December 4, 2017
        6.1 October 25, 2014 October 25, 2018
        6.0 January 19, 2014 March 19, 2017
        5.1 (Panorama only) May 9, 2013 May 9, 2017
        5.0 November 13, 2012 November 13, 2016
        4.1 October 31, 2011 April 30, 2015
        4.0 February 22, 2011 December 31, 2014
        3.1 March 15, 2010 June 30, 2013
        3.0 June 17, 2009 December 17, 2010
        2.1 January 5, 2009 January 5, 2012
        2.0 May 20, 2008 May 20, 2009
        1.3 November 15, 2007 November 20, 2008

        GlobalProtect EOL

        GlobalProtect App version Release Date End-of-Engineering Date End-of-Life Date
        6.2 05/23/2023 05/23/2025 05/23/2025
        6.1 09/01/2022 09/01/2024 03/01/2025
        6.0 02/22/2022 02/22/2025 02/22/2025
        5.3 06/01/2021 12/01/2022 06/01/2023
        5.2 07/30/2020 08/31/2023 02/28/2024
        5.1 12/12/2019 03/12/2021 12/31/2024
        5.0 2/12/2019 5/12/2020 2/12/2021
        4.1 3/1/2018 6/1/2019 3/1/2020
        4.0 1/30/2017 5/2/2018 1/30/2019
        3.1 6/23/2016 9/23/2017 6/23/2018
        3.0 2/16/2016 5/18/2017 2/15/2018

        ▶ Related article:
        Palo Alto firewall - Releases and Addressed Issues

        * Note: PAN-OS 10.2 is a new feature release. PAN-OS 10.2 (Nebula release) is enhanced to increase reliability and robustness. Upgrading Panorama to PAN-OS 10.2 or greater requires upgrading all Panorama installed plugins to compatible versions.

        Recommend to check the important information - Important information regarding Panorama 10.2 and upgrade dependencies.

         Method 1 (without HA failover testing)

        Step 1. Save a backup of the current configuration file.

        1) Perform these steps on each firewall in the pair: Select Device > Setup > Operations and click Export named configuration snapshot.
        2) Select the XML file that contains your running configuration (for example, running-config.xml ) and click OK to export the configuration file.
        3) Save the exported file to a location external to the firewall. You can use this backup to restore the configuration if you have problems with the upgrade.

        Step 2. Make sure each device running the recommended Content Release version.

        1) Select Device > Dynamic Updates.
        2) Check the Applications and Threats or Applications section to determine what update is currently running.
        3) If the firewall is not running the required update or later, click Check Now to retrieve a list of available updates.
        4) Locate the desired update and click Download.
        5) After the download completes, click Install.

        Step 3. Determine the upgrade path. (prerequisite)

        You cannot skip installing any major release versions on the path to your desired PAN-OS version. Therefore, if you plan to upgrade to a version that is more than one major release away, you must still download, install, and reboot the firewall into all interim PAN-OS versions along the upgrade path.

        For example, if you want to upgrade from PAN-OS 7.1.8 to PAN-OS 9.1.1, you must:
        - Download and install PAN-OS 8.0.0 and reboot.
        - Download and install PAN-OS 9.0.0 and reboot. -> PAN resolved this issue. You may just need to download the base image (9.0.0) and download & install the target image.
        - Download and install PAN-OS 9.1.1 and reboot.

        If you have the pair in HA(active/passive) then you have to upgrade only to next version of PAN-OS then failover and proceed to upgrade for the second version of PAN-OS. Only upgrade one version at a time.
        Example: If you are at PAN-OS 7.1.x then you should go to 8.0.x version(let it be any version of PAN-OS) then failover and check the functionality. Otherwise you will run into the error and the HA pairs will no longer be in sync.
        Additionally Remember that if there is more than 1 version of difference between the HA pairs then you will run into the "Peer version too old" issue.

        [† AnalysisMan Tip]

        You do not need to reboot the firewall during the intermediate upgrades even though it is not the best practice and recommended by PAN. However, it works and you can reduce the upgrade time overall as below.

        PAN-OS 8.1.8 -> 9.0.0 (base image download and install) -> 9.1.1 (target image download & install & reboot)

        Skipped steps with the intermediate PAN-OS images as below.

        Step 4. Install PAN-OS 9.1 on the passive device (active/passive)

        1) Click Check Now to check for the latest updates.

        * CLI Command
        request system software check

        2) Locate the version you want to upgrade to and then click Download.

        request system software download version 9.1.1

        3) Verify the downloaded version.

        request system software info

        admin@firewall2(passive)> request system software info

        Version               Size          Released on Downloaded
        9.1.1                312MB 2019/09/26  11:23:05        yes
        9.0.3                309MB 2019/07/12  10:30:40         no

        4) After the download completes, click Install.

        request system software install version 9.1.1

        admin@firewall2> request system software install version 9.1.9
        Executing this command will install a new version of software. It will not take effect until system is restarted. Do you want to continue? (y or n) y

        Software install job enqueued with jobid 3312. Run 'show jobs id 3312' to monitor its status. Please reboot the device after the installation is done.

        [† AnalysisMan Tip]

        You should be able to check if the target PAN-OS image is installed on the partition by the following CLI command. It shows as 'PENDING-CHANGE', and it will be changed to 'RUNNING-ACTIVE' after rebooting.

        debug swm status

        admin@firewall2(passive)> debug swm status

        Partition         State             Version
        sysroot0          RUNNING-ACTIVE    9.0.8
        sysroot1          PENDING-CHANGE    9.1.1
        maint             READY             9.1.1

        Step 5. After the install completes, reboot using one of the following methods:

        - If you are prompted to reboot, click Yes.
        - If you are not prompted to reboot, select Device > Setup > Operations and click Reboot Device in the Device Operations section.

        request restart system

        Step 6. Verify that the active device is passing traffic

        By viewing the Monitor > Session Browser, or by running show session all from the CLI.
        · You can also check the HA state on the device with show high-availability all. On the active/active configuration, check that both devices are passing traffic.

        show high-availability all
        show high-availability interface ha2

        · To check session synchronization run show high-availability interface ha2. In the Hardware Interface counters read from CPU table check that counters are increasing. In an active/passive configuration, only the active device will show packets transmitted and the passive device will only show packets received. In the active/active configuration, you will see packets received and packets transmitted on both devices.

        show high-availability interface ha2 | match bytes

        admin@ firewall1(active)> show high-availability interface ha2 | match bytes
        rx-bytes                      191827728
        tx-bytes                      1916678815290
        bytes received                           114449284
        bytes transmitted                        1885650036560
        bytes received                           114449284
        bytes transmitted                      1885650036804

        admin@ firewall2(passive)> show high-availability interface ha2 | match bytes
        rx-bytes                      1916674506264
        tx-bytes                      108310016
        bytes received                           1885689069756
        bytes transmitted                        100558008
        bytes received                           1885689069756
        bytes transmitted                        100558008

        Step 7. Suspend the active firewall for HA failover.

        1) On the active (active/passive) or active-primary (active/active) device, select Device > High Availability > Operational Commands.

        2) Click Suspend local device.

        Or fail over to the passive firewall via CLI command on the active firewall as below.

        request high-availability state suspend

        > request high-availability state suspend

        3) Select Dashboard and verify that the state of the passive device changes to active in the High Availability widget.

        4) Verify that the firewall that took over as active or active-primary is passing traffic by selecting Monitor > Session Browser.

        or use CLI command.

        show session all

        admin@firewall2(active)> show session all

        Step 8. Upgrade the suspended firewall.

        Upgrade the firewall1, which is now suspended from active. Follow the above Step 4 ~ 6.

        The firewall1 will be failed over automatically and running as active after rebooting, HA links up, sessions sync. I presumed that you have Primary (firewall1) is active with a lower priority value (e.g., 100), and Secondary (firewall2) is passive with a high priority value (e.g., 200).

        * If both firewalls have the same device priority value, the firewall with the lowest MAC address on the HA1 control link will become the active firewall.

        You must enable preemptive on both the active firewall and the passive firewall.

        [† AnalysisMan Tip]

        I use Method 1 because I am confident that the HA failover is functional and reduce the upgrade time overall.
        Also, you may install the PAN-OS image on both active/passive firewalls before the maintenance window. And reboot the Secondary (firewall2) firewall, failover to Secondary firewall, and reboot the Primary (firewall1) firewall. That's it.

         Method 2 (with HA failover testing)

        Here is the Palo Alto's Best Practices for PAN-OS Upgrade. (login required)

        It is recommended to upgrade the Primary firewall first and then upgrade the Secondary firewall. This is done for two reasons:

        1) Ensure that HA failover is functioning properly
        2) Ensure that the passive firewall is functioning properly and is able to pass traffic without issues

        Follow the steps in the above link.

        ▶ Related articles:

        No comments: