Viewing palo statistics

system resources and traffic statistics

8 November 2022   8 min read

Useful commands to see general information on the firewall resources been used, interface and traffic statistics, and traffic counters.

System and resources

show system resources is the same as top in linux, adding follow will keep it auto refreshing until stopped with CTRL + C.

ste@HME-PAL-OEW1> show system resources follow
top - 17:41:48 up 25 days,  9:26,  1 user,  load average: 0.82, 0.43, 0.33
Tasks: 222 total,   2 running, 167 sleeping,   0 stopped,   1 zombie
%Cpu(s):  8.9 us,  1.7 sy,  0.0 ni, 87.0 id,  0.0 wa,  2.2 hi,  0.2 si,  0.0 st
KiB Mem :  6612900 total,   280932 free,  2637248 used,  3694720 buff/cache
KiB Swap:  4095996 total,  3526652 free,   569344 used.   886092 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND       
 6592 root      20   0   66.7g   2.6g   2.6g S  15.9 41.0   5349:19 pan_task      
 5341 root      20   0       0      0      0 S   3.0  0.0 968:27.46 kni_single     
 5415 root      20   0  472768  25248      0 S   0.7  0.4 147:20.85 distributord 
 2924 root       0 -20 3047528   2.6g   2.6g S   0.3 40.8  61:20.15 masterd_apps 
 2967 root      15  -5  168604   7140   1988 S   0.3  0.1  64:58.25 sysd         
 5293 nobody    20   0   52840   2200   1632 S   0.3  0.0  19:50.62 redis-server 
 5298 nobody    20   0   50280    228     72 S   0.3  0.0  19:43.07 redis-server                  
 5411 root      20   0 1764808 374444 122964 S   0.3  5.7  62:49.87 devsrvr  
 5416 root      20   0  520088  47852    208 S   0.3  0.7  45:03.03 iotd     
 5418 root      20   0  741304 150176 112504 S   0.3  2.3 218:54.33 useridd  
 5486 nobody    20   0   50280   2032   1640 S   0.3  0.0  30:57.99 redis-server                 
 5493 nobody    20   0   58984   1708   1172 S   0.3  0.0  92:21.15 redis-server                       
 5538 root      20   0 1745340 244996  12768 S   0.3  3.7  14:15.94 logrcvr  
 5547 root      20   0  721500  74572   9632 S   0.3  1.1 181:41.14 dnsproxyd
 6889 root      20   0   66.7g   2.6g   2.5g S   0.3 40.5   8:04.76 sdwand   
 7061 root      20   0   66.7g   2.6g   2.5g S   0.3 40.5  63:38.55 pan_dha  
 8730 root      20   0  160764  15512      4 S   0.3  0.2  28:41.15 envoy    
17075 ste       20   0  121636   2960   2400 R   0.3  0.0   0:00.04 top      
28650 root      20   0       0      0      0 I   0.3  0.0   3:38.61 kworker/0:1-eve             
    1 root      20   0    4388      4      0 S   0.0  0.0   0:10.67 init    

The running firewall processes and PID can be viewed using the command show system software status.

If running any tasks that are CPU intensive such as logging of the packet flows, this command is useful for keeping an eye on what resources are using up what percent of the CPUs. By default it will be for the last 60 seconds, can view the average and maximum CPU usage by filtering on the last 1-60 seconds, 1-60 minutes, 1-24 hours, 1-7 days or 1-13 weeks.

ste@HME-PAL-OEW1> show running resource-monitor hour last 3
ste@HME-PAL-OEW1> show running resource-monitor 
Resource monitoring sampling data (per second):
CPU load sampling by group:
flow_lookup                    :     1%
flow_fastpath                  :     1%
flow_slowpath                  :     1%
flow_forwarding                :     1%
flow_mgmt                      :     1%
flow_ctrl                      :     1%
nac_result                     :     0%
flow_np                        :     1%
dfa_result                     :     0%
module_internal                :     1%
aho_result                     :     0%
zip_result                     :     0%
pktlog_forwarding              :     1%
send_out                       :     1%
flow_host                      :     1%
send_host                      :     1%
fpga_result                    :     0%

CPU load (%) during last 60 seconds:
core   0   1
       *   1

Both these commands provide really nice realtime statistics on session traffic (packet rate, throughput, number of sessions) and applications (sessions, packets, bytes) that keep refreshing until quit.

ste@HME-PAL-OEW1> show system statistics session
System Statistics: ('q' to quit, 'h' for help)

Device is up          : 18 days 3 hours 12 mins 55 sec
Packet rate           : 120/s
Throughput            : 165 Kbps
Total active sessions : 61
Active TCP sessions   : 37
Active UDP sessions   : 21
Active ICMP sessions  : 3

ste@HME-PAL-OEW1> show system statistics application
Top 20 Application Statistics: ('q' to quit, 'h' for help)

Virtual System: vsys1
application                      sessions   packets      bytes
-------------------------------- ---------- ------------ ------------
stun                             1184       2581768      2093735764
itunes-base                      3181       1823844      1698765716
twitter-base                     936        1719312      1628091621
icloud-base                      7005       1240232      910446082
rtp-base                         3          735148       807039702
http-video                       23         9402047      9396393333
avira-antivir-update             277        527288       537032712

A few useful commands for looking at the dataplane interface utilisation.

show interface all                                       Interface name, speed, duplex, state, MAC, IP address and zone
show interface ethernet1/1 Settings and traffic counters including transmitted, received dropped and errors
show counter interface all Traffic counters including transmitted, received dropped and errors
show counter rate ethernet1/1 Number of tx/rx packets and Mbps for last second

Netstat shows information related to the management plane.

show netstat interfaces yes                                Shows management, loopback and tap interfaces as well as counters (*all yes* also shows tcpdump
show netstat route yes Shows the management routing table
show netstat all yes Show the ports open and the state of the connections on them
show netstat programs yes Shows ports open anD connections but now includes program and PID

I dont really understand the purpose of the ‘system state’ commands yet, seem to be hardware capabilities as well as counters.

show system state filter-pretty sys.s1.p*                Shows counters on all interfaces
show system state browser Interface stats in realtime, do Shift + L, port_stats, Y, U

Logs

System logs can viewed by severity and further filtered down based on the object, subtype and event. Juts like traffic logs can filter the time-frame and add csv-output equal yes to display the output in csv format.

show log system receive_time in [last-60-seconds | last-15-minutes | last-hour | last-6-hrs | last-12-hrs | last-24-hrs]
show log system receive_time in [last-7-days | last-30-days | last-calendar-day | last-calendar-month]
show log system start-time [equal | not-equal] <YYYY/MM/DD@hh:mm:ss>
show log system end-time [equal | not-equal] <YYYY/MM/DD@hh:mm:ss>

ste@HME-PAL-OEW1> show log system severity 
> equal                   equal 
> greater-than-or-equal   greater-than-or-equal 
> less-than-or-equal      less-than-or-equal 
> not-equal               not-equal 

ste@HME-PAL-OEW1> show log system severity greater-than-or-equal 
  critical        critical 
  high            high 
  informational   informational 
  low             low 
  medium          medium 

ste@HME-PAL-OEW1> show log system object equal "ethernet1/2" subtype equal "port"
Time                Severity Subtype Object EventID ID Description
===============================================================================
2022/09/28 08:18:06 info     port    ethern link-ch 0  Port ethernet1/2: Down Unknown duplex
2022/09/28 08:18:06 info     port    ethern link-ch 0  Port ethernet1/2: Up   10Gb/s-full duplex
2022/09/28 08:18:06 info     port    ethern link-ch 0  Port ethernet1/2: MAC Up  

The CLI doesn’t provide auto-completion on object, subtype or event, for this reason the GUI maybe a bit easier for looking at system logs.

The majority of the other non-system logs are grouped under plugins-log and mp-log (dp-log on some platforms). To view and filter use tail, less, follow and grep.

ls mp-log ?                                                  View all log files such as 
ls plugins-log ? View all log files for plugins
tail mplog pan_boot.log
less mp-log configd.log
less mp-log dhcpd.log
tail mp-log routed.log
tail follow yes mp-log bfd.log
tail follow yes mp-log pan_ifmgr.log

A disk quota is assigned to the different log types, logs will be purged when the quota is exceeded. The allocated quota and current usage levels can be checked with the command show system logdb-quota.

Global counters

All sessions traversing the firewall are tracked by the processes that touch them with global counters incremented for each step that a packet takes and for each packet in a session. show counter global provides information about the processes/actions taken on the packets passing through the device; whether they are dropped, NAT-ed, decrypted and so on. This is a list of all the possible global counters.

ste@HME-PAL-OEW1> show counter global filter 
+ aspect          Counter aspect
+ category        Counter category
+ delta           Difference from last read
+ packet-filter   Counters for packet that matches debug filter
+ severity        Counter severity
+ value           value 

Global counters can be filtered based on severity, category and aspect.

  • Severity: Are 4 levels of severity, info (default for all counters), drop (indicate something that was intentionally discarded like due to a security policy), error (packets that are malformed and are discarded), or warn (system level error or abnormality in received packets)
  • Category: Indicates which process this counter is related to (dfa (APP-ID algorithm engine), appid, dlp, flowpacket, uid, zip, nat, etc)
  • Aspect: More detail regarding which stage a packet was in when the counter was incremented (parse, session, and forward are three stages of flow)

Drop counters are normally the ones of real interests as it keeps a count of all drops, what is causing them (such as flow_policy_deny for packets that were dropped by a security rule) and how many packets were dropped. By running the command multiple times with delta yes can see the drop counters since the last time the command was run making it easier to see if counters are increasing.

show counter global                                            View all global counters
show counter global filter delta yes View only global counters that have changed since ran last command
show counter global name ? Lists all the counters that are availables
show counter global filter category Lists available processes such as cluster, device, dlp, flow, nat, proxy, ssl, tcp, url, etc
show counter global filter aspect Lists available stages such as arp, bfd, dos, forward, offload, pktproc, etc

show counter global filter value non-zero Show only counters without non-zero values
show counter global name flow_policy_deny Show this one counter with a brief description
show counter global filter severity drop Show only drop counters
show counter global filter severity drop delta yes Show only drop counters that have happened since last command was run

Global counters can be narrowed down to specific traffic flows by using the same filters that packet captures use. Before starting is best to clear all counters and unmark any sessions that were marked by the previous filter.

debug dataplane packet-diag clear all                                       Clear all capture settings, counters and filters
debug dataplane packet-diag clear filter-marked-session all Unmark any sessions that were marked by previous filters

debug dataplane packet-diag set filter match destination 8.8.8.8 Configure upto 4 filters
debug dataplane packet-diag set filter match source 8.8.8.8
show counter global filter delta yes packet-filter yes severity drop Show only drop counters for flows that match this filter

Resources

Palo KB article on global counters and filtering them
Some really good info on counters and lots of other CLI show commands
Some handy cheatsheets from Mastering Palo Alto Networks by Tom Piens