Friday, August 28, 2020

Palo Alto firewall - How to clean up disk space

The disk-space full is more like to happen on the PA-200, 220 platform, which has a smaller disk size than other platforms. Symptoms may vary by filling up disk-space depends on which directory.

Here are symptoms of disk-space getting filled up are:

  • Symptom

- Web interface not loading
- Certain daemons processes not starting
- Incomplete tech support bundles
- System log alerts with "Disk usage for / exceeds limit,  X percent in use, cleaning file system"
- Unable to download PAN-OS software images or dynamic updates
- GlobalProtect HIP checking fails

  • Cause

The PAN-OS file system is divided into various directories. To view partitions and associated disk-space, use the command below:
show system disk-space 

Filesystem            Size  Used Avail Use% Mounted on
/dev/md2              3.9G  3.8G  588K  100% /                          << root partition is full
/dev/md5              7.6G  4.2G  3.0G  59% /opt/pancfg .        
/dev/md6              3.8G  3.0G  666M  82% /opt/panrepo
tmpfs                 2.0G  210M  1.8G  11% /dev/shm
cgroup_root           2.0G     0  2.0G   0% /cgroup
/dev/md8               88G  2.4G   81G   3% /opt/panlogs
tmpfs                  12M     0   12M   0% /opt/pancfg/mgmt/lcaas/ssl/private

The PAN-OS file system has a default mechanism to rotate and clear disk-space. In some cases, manual intervention may be required to clear disk space.

  • Resolution

Option 1: Enable Aggressive Cleaning

This command is available on PAN-OS 7.1.14+, PAN-OS 8.0.7+ and PAN-OS 8.1.0+.

This will automatically truncate all old log files (entries under all *var/log/pan directories matching *.1, ... *.4, *.log.old) if the 95% occupancy alarm is tripped.

> debug software disk-usage aggressive-cleaning enable
> debug software disk-usage aggressive-cleaning disable

To verify the changes or if it is already enabled use the command below.

> show system state | match aggressive-cleaning
cfg.debug-sw-du.config: { 'aggressive-cleaning': True, }

※ True means it's enabled. False means it's disabled.

In PAN-OS versions 8.1.0+, cleanup can be done manually using the command below:
> debug software disk-usage cleanup ?
+ deep        cleanup with deleting backup logfile
* threshold   percentage threshod of size of system partition to kick off cleanup

> debug software disk-usage cleanup deep threshold 90

Note that this command has to be run manually each time to bring the disk usage to below 90% and is not persistent across reboots.

Caveat: Enabling aggressive clean up may clear up logs, rendering logs unavailable for analysis.

† Tip: Create an alert for the disk-full on syslog.
The firewall generates a warning alert on the system logs. If you have a syslog system, you may parse a word like 'Disk usage for / exceeds limit' and create an alert when detected.
"Disk usage for / exceeds limit, 95 percent in use, cleaning filesystem."

Option 2: Check and Delete Unnecessary Core Files

Check the output of >show system files to see core files using up a large amount of disk space.

show system files
total 4.0K
drwxrwxrwx 2 root root 4.0K Jun 10 20:05 crashinfo

total 0

total 115M
drwxrwxrwx 2 root root 4.0K Jun 10 20:15 crashinfo
-rw-rw-rw- 1 root root 867M Jun 12 13:38 devsrvr_4.0.3-c37_1.gz
-rw-rw-rw- 1 root root  51M Jun 12 13:39 core.20053

total 16K
-rw-rw-rw- 1 root root 15K Jun 10 20:15

Delete unnecessary core files. A core file can be deemed unnecessary if investigation around the core file is complete or they are very old files.
(This example deletes a device server core file from the management-plane.)

> delete core management-plane file devsrvr_4.0.3-c37_1.gz

Option 3: Delete Rotated Files and Files with Extension .old

These files contain monitoring details and service related logs on the firewall. They can be deleted safely if you don't need them. If TAC investigates an ongoing issue, you may prefer to keep them until you upload the tech support file to the case manager.

> delete debug-log ?
> cp-log      Remove cp-log at /opt/var.cp/log/pan/
> dp0-log     Remove dp0-log at /opt/var.dp0/log/pan/
> dp1-log     Remove dp1-log at /opt/var.dp1/log/pan/
> dp2-log     Remove dp2-log at /opt/var.dp2/log/pan/
> mp-global   Remove mp-global at /opt/mp-global/
> mp-log      Remove mp-log at /var/log/pan/

delete debug-log mp-log file *.1
delete debug-log mp-log file *.2
delete debug-log mp-log file *.3
delete debug-log mp-log file *.4
delete debug-log mp-log file *.old

show system disk-space

Option 4: Clear Any packet-diag Logging If Enabled

Run > debug dataplane packet-diag show setting to check if packet-diag is enabled.

debug dataplane packet-diag show setting

DP dp0:
Packet diagnosis setting:
Packet filter
  Enabled:                   no
  Match pre-parsed packet:   no          
  Enabled:                   no     <<<  No is the default value and means not enabled.

debug dataplane packet-diag show setting

DP dp0:

Packet diagnosis setting:
Packet filter
  Enabled:                   no
  Match pre-parsed packet:   no          
  Enabled:                   yes   <<< Means packet-diag is enabled.

To clear out packet-diag setting and the logs, run below commands:
debug dataplane packet-diag clear all
Packet diagnosis setting set to default.

debug dataplane packet-diag clear log log
dataplane debug logs cleared

​​​​​​Option 5: Delete Any Debug pcaps or Debug-filter pcaps

delete debug-filter file

delete pcap directory *

Option 6: Delete Old Content and Antivirus Update Packages

delete content cache old-content

delete anti-virus update

Apart from these options, there have been various improvements in the latest PAN-OS release to best utilize disk-space.

Last resort: Open a case with Palo Alto TAC

Multiple bugs are related to the disk space, so I recommend checking the release notes and upgrading the PAN-OS. You still need to clean up the disk space before you upgrade the firewall.

After you open a case, you will need a TAC engineer who has the root permission. Some of the outsourced TAC engineers don't have the root access.

The Palo Alto TAC engineer should execute the following steps. The debug tac-login is only intended for use by a TAC engineer, and access is restricted outside of TAC.

debug tac-login challenge
admin@firewall> debug tac-login challenge
Please use the following string as your challenge:


debug tac-login response
admin@firewall> debug tac-login response 
Please enter challenge response (^C to cancel):
-----END RESPONSE-----

* The above change and response codes are modified and scrambled, so it is not the real ones.

[root@firewall ~]#    << TAC got into the root access on Linux

- Check the disk-space and directories on Linux with the root access.
[root@firewall]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3             1.9G  1.6G  206M  89% /
/dev/sda5             6.6G  5.1G  1.3G  81% /opt/pancfg
/dev/sda6             1.9G  1.2G  619M  67% /opt/panrepo
tmpfs                  1.4G  210M  1.2G  16% /dev/shm
/dev/mmcblk0p8  5.6G  5.6G  0     100% /opt/panlogs

- Go to Threat logs directory, and delete the old files and folders.
[root@firewall]# cd /opt/panlogs/logdb/threat/1

- Go to Traffic logs directory, and delete the old files and folders.

- Check the current directory.
[root@firewall]# pwd


- Check the files or directories.
[root@firewall]# ls -ah
.   20200801  20200803 20200805  20200807  20200809  20200811 20200813  20200815  20200817  20200819 20200821  20200823  20200825  20200827
..  20200802  20200804 20200806  20200808  20200810  20200812 20200814  20200816  20200818  20200820 20200822  20200824  20200826  20200828

[root@firewall]# ls -alh
total 120K
drwxrwxrwx 30 root root 4.0K Aug 28 19:55 .
drwxr-xr-x  3 root root 4.0K Oct 24  2019 ..
drwxrwxrwx  2 root root 4.0K Aug  2 02:15 20200801
drwxrwxrwx  2 root root 4.0K Aug  3 02:15 20200802
drwxrwxrwx  2 root root 4.0K Aug  4 02:15 20200803

- Check the disk-space of the folders, and sort the files or directories with numeric data present inside.
- The 'sort -n' option shows the largest file at the bottom, and 'sort -nr' shows the largest file at the top.

[root@firewall]# du -hs * | sort -n
4.4M 20200828
4.9M 20200801
5.0M 20200826
5.0M 20200827
6.4M 20200825

[root@firewall]# du -hs * | sort -nr
6.4M 20200825
5.0M 20200827
5.0M 20200826
4.9M 20200801
4.4M 20200828

- Delete the old files and folders.
[root@firewall]# rm -rf 2019*

- Logout from the Linux mode.

* Additional Information
For additional information, please review the following articles:

How to Delete Unnecessary Downloaded Software Versions
How to delete configurations through the CLI
How to Delete Saved Configuration Files

No comments: