Information on how applications are identified by App-ID and following sessions and traffic flows through the firewall using the CLI.
Table Of Contents
Identifying an Application
Based on the first TCP packet received (SYN) an application will be marked as incomplete as it can not be detected until at least one data packet traverses the device. App-ID will go through several stages to identify just what something is.
- 6-Tuple: Packet is checked against the security policy to verify whether the source, destination, protocol, and port combination is allowed
- Application: The packet is checked against known application signatures and the app cache followed by a second security policy check now adding App-ID to the required set of identifiers for the security policy to allow the session through
- Decrypt: If the application is SSH, TLS, or SSL and matches the decryption policy the session is decrypted and checked again for a known application signature as the encapsulated session may be something entirely different
- Decoder: If by now the application has not been identified (maximum of 4 packets after the handshake or 2000 bytes) App-ID will use the base protocol to determine which decoder to use to analyze the packets more deeply
- Known protocol: It is decoded and the payload run against the known application signatures again. The outcome could either be a known application or an unknown generic application like unknown-tcp. The session is then again re-matched against the security policy
- Unknown protocol: App-ID will apply heuristics to try and determine which protocol is used in the session. Once it is determined which protocol is used another security policy check is performed
- Stop: Once the application has been identified or all options have been exhausted, App-ID will stop processing the packets for identification
Throughout the life of a session, the identified application may change several times as more info is learned from the session through inspecting packet after packet. Whenever an application shift happens a new security policy lookup is performed to find the closest matching rule. Once the App-ID process has settled on an application the application decoder will continuously scan the session for expected and deviant behavior.
In parallel to each packet being processed by an App-ID decoder the Content-ID stream-based engine applies the URL filtering policy and scans the session for threats like vulnerability exploits, virus or worm infections, suspicious DNS queries, command and control signatures or DoS attacks. This ‘in parallel’ design reduces latency meaning that enabling more security profiles does not come at an exponential performance cost.
HTTPS Traffic
To get the most out of App-ID in terms of application recognition the firewall should function as an SSL forward proxy and decrypt SSL traffic going to external sites. For non-decrypted HTTPS traffic the firewall can use the Server Name Indication (SNI) for URL categorization, if the client does not send the SNI then the Common Name (CN) is used (if is a wildcard the application is identified as SSL). Custom applications can be created using the SNI to control SSL traffic without the need for SSL decryption, although it is worth noting that the when the SNI is added to the TLS handshake it is not in the encrypted conversation meaning that it can be spoofed.
Applications can have either implicit or explicit dependant applications. For example facebook-base dependant applications will be implicitly used needing no explicit rules whilst ms-office365-base requires explicit rules for the applications that it depends on. This can be viewed using show predefined application <application_name> in configuration mode of the CLI as implicit-use-applications and use-applications.
If an App-ID application relies on ssl and web-browsing it is extremely likely that these are what are used in the initial part of the conversation to identify the application (some applications redirect to another website, like for authentication).
- ssl: Identifies HTTPS applications as this based on the SSL handshake. If that first ssl request is not allowed in the security policy the session will never get to identifying the application. After that request is allowed either the packet is decrypted or the firewall reads the SNI of the certificate and if possible a more specific application ID identified and the application re-evaluated against policy with that ID
- web-browsing: Identifies applications as this based typical browsing behavior such as an HTTP GET. The web-browsing app on port 443 is only matched/shown for firewalls doing SSL decryption, otherwise it shows up as the ssl application.
web-browsing and ssl only match generic HTTP/HTTPS traffic, if the firewall determines a more precise App-ID for the traffic then it will switch to using that for determining whether to allow or deny this traffic. If traffic does not match a more specific application and is only classified as ssl or web-browsing it will be allowed by any rules with these applications in. To guard against holes in the security policy allowing unsanctioned applications need to lock down the access of these applications to specific destinations and URL categories (be it pre-defined, custom or EDLs). It is a compromise between giving enough access to recognize the applications vs allowing too much unknown ssl or web-browsing traffic.
Traffic flows
The session table and traffic log are good places to start when trying to understand the traffic flows through the firewall.
- Session browser: Active sessions currently passing through the firewall, once the sessions end can see them in the traffic log
- Traffic Log: Historic information for sessions that have already ended (via the Log at Session End option in the security policy)
- Unified Log: Displays the Traffic, Threat URL Filtering, WildFire, and Data Filtering logs all in the same single-pane view
Sessions
A session is defined by two uni-directional flows each uniquely identified by a 6-tuple key of source/destination IP and port, protocol and security-zone. Each entry has a lot of information regarding the session that can not be found in the traffic logs.
show session infoNumbers of active sessions, statistics throughput timers and TCP/UDP settings show session meterDisplays maximum, current and throttled number of sessions for each VSYS show session allAll active sessions, limits number shown based session byte size show session all filterkey value Can filter based on most of the attributes of a session show session all filterkey value countyes Count the number of sessions matching filter show session idid To see a detailed view of a specific session
Each session is identified by two uni-directional flows, client-to-server(c2s) and the returning server-to-client(s2c).
ste@HME-PAL-OEW1> show session all
--------------------------------------------------------------------------------
ID Application State Type Flag Src[Sport]/Zone/Proto (translated IP[Port])
Vsys Dst[Dport]/Zone (translated IP[Port])
--------------------------------------------------------------------------------
48968 spotify ACTIVE FLOW NS 10.10.64.224[39461]/south/6 (192.168.1.202[25008])
vsys1 104.199.65.124[4070]/north (104.199.65.124[4070])
State: There is total 7 possible session states that a flow can be in. The 3 stable states represent most of the sessions seen in the session table as the other 4 transient states are shortlived and used during the transition to one of the stable states.
show session all filter state [initial | opening | active | closing | discard | closed]
- INIT (stable): A session that is ready and waiting to be used by a new flow. Every session begins in this state
- OPENING (transient): A session ID is assigned to the flow while it is being evaluated to become a full session (has aggressive timers to handle half-open TCP connections)
- ACTIVE (stable): This is the state in which everything happens – the flow is up and packets are being passed back and forth
- CLOSING (transient): Session timeout has been reached for the flow (TTL or idle-timeout) meaning the session is set to expire soon but has not yet been removed from aging process or session lookup table. New packets no longer match against this session so are instead either queued to create a new session or discarded as they no longer match an active session (non-SYN TCP)
- DISCARD (stable): Traffic that has been matched by a session but is denied due to a drop/deny security policy rule or a threat detection block. All packets matching the session will be discarded for the duration of the discard phase
- CLOSED (transient): The session has been removed from the aging process, but not from the session lookup table. No new packets can match this session so are either queued for a new session or dropped
- FREE (transient): The session has been closed and removed from the session lookup table but still needs to be made available for a new flow
Under normal conditions each state will go through the transition cycle of INIT > OPENING > ACTIVE > DISCARD/CLOSING > CLOSED > FREE. From the FREE state the session will move back to the INIT to start the next cycle.
Type: There are also five session types of which the two most common seen in the session table are FLOW or PRED.
show session all filter type [flow | forward | predict | tunnel | vni]
- FLOW: These are all the regular sessions where the flow is the same between c2s and s2c (HTTP, Telnet, SSH, etc)
- PRED: Application Layer Gateway (ALG) protocols that require a return session be set up outside of the established session use a predict session to anticipate the inbound connection and transform this into a FLOW if the return session is received (FTP active/passive, SIP, etc)
- FORW: Captive portal intercepting and redirecting browsing sessions to a login page or policy-based forwarding is applied to a flow
- Tunnel: VPN connections will be set up in a Tunnel session
- VNI: If VXLAN Tunnel Content Inspection (TCI) is enabled in Policies » Tunnel Inspection VXLAN tunnels will be vni-type sessions
Flags: Information related to the type of NAT used. A flag with an asterisk (*) indicates that the session was decrypted.
show session all filter nat [source | destination | both | none]
- NS (NAT Source): The source address was translated
- ND (NAT Destination): The destination address was translated
- NB (NAT Bidirectional): Both the source and destination addresses were translated
- No flag: There is no NAT applied on the session
Individual sessions show a lot of additional information regarding the traffic flow, some of more interesting fields are:
- Timeout: Specific timeout configured for the application, is the amount of time a session is allowed to exist
- Time to live: Time left until the session will expire and the state change
- Session in session ager: Aging process that keeps track of the lifetime of sessions, will be True for Active sessions when TTL not reached 0
- Session to be logged at the end: Configured in the security rules, can be set to the end (True) or the start (False) of the session
- Offload yes: Marks traffic for which the application has already been identified and will be processed in hardware
- Layer7 processing: App-ID has been enabled on the traffic flow and the application is constantly identified (complete means the application has been identified)
- tracker stage (firewall): Provides information on the precise reasons for mitigation actions taken on particular session
ste@HME-PAL-OEW1> show session id 173457
Session 173457
c2s flow:
source: 10.10.64.223 [south]
dst: 104.199.65.124
proto: 6
sport: 50210 dport: 80
state: ACTIVE type: FLOW
src user: unknown
dst user: unknown
s2c flow:
source: 104.199.65.124 [north]
dst: 192.168.1.202
proto: 6
sport: 80 dport: 42596
state: ACTIVE type: FLOW
src user: unknown
dst user: unknown
start time : Thu Oct 13 12:16:41 2022
timeout : 3600 sec
time to live : 3598 sec
total byte count(c2s) : 1107545
total byte count(s2c) : 656987
layer7 packet count(c2s) : 15518
layer7 packet count(s2c) : 8468
vsys : vsys1
application : spotify
rule : Green to iNET
service timeout override(index) : False
session to be logged at end : True
session in session ager : True
session updated by HA peer : False
address/port translation : source
nat-rule : Green PAT(vsys1)
layer7 processing : completed
URL filtering enabled : False
session via syn-cookies : False
session terminated on host : False
session traverses tunnel : False
session terminate tunnel : False
captive portal session : False
ingress interface : ethernet1/2
egress interface : ethernet1/1
session QoS rule : N/A (class 4)
tracker stage l7proc : ctd decoder done
end-reason : unknown
The session table is viewable in the GUI under Monitor » Session Browser, it can be exported and saved to PDF or CSV.
If a network connection failure can not found in the traffic log it is worth checking for DISCARD sessions in the session table.
show session all filter statediscard Dropped traffic has a discard state and end-reason of policy-deny show session all filter application dns destination8.8.8.8 Match all DNS traffic to google show session all filter fromtrust tountrust applicationssl stateactive Match all active SSL traffic between zones
Forcibly terminating a session will immediately set the sessions state to INIT so that any packets still arriving on the firewall will not have a session to match against meaning that they are either discarded as non-SYN TCP or evaluated as a new session. The CLI seems more flexible than the GUI (done under Clear column) for clearing sessions as sessions can be cleared based on the same filters used to view sessions.
clear session allclear session idid show session all filterkey value count yesclear session all filterkey value
Traffic
For each sessions log action (start or end) a traffic log entry is created that includes the standard 6-tuple Key (source/destination IP and port, protocol and security-zone), users, the application (or lack of), bytes, packets and the reason for the action applied by a security policy.
When viewing in the CLI the majority of filter options can be chained together. Adding csv-output equal yes displays the output in csv format.
show log traffic receive_time in [last-60-seconds | last-15-minutes | last-hour | last-6-hrs | last-12-hrs | last-24-hrs]
show log traffic receive_time in [last-7-days | last-30-days | last-calendar-day | last-calendar-month]
show log traffic start-time [equal | not-equal] <YYYY/MM/DD@hh:mm:ss>
show log traffic end-time [equal | not-equal] <YYYY/MM/DD@hh:mm:ss>
show log traffic rule [equal | not-equal] <rule_name>
show log traffic rule_uuid [equal | not-equal] <rule_uuid>
show log traffic from [equal | not-equal] <zone>
show log traffic to [equal | not-equal] <zone>
show log traffic src [in | not-in] <ip>
show log traffic dst [in | not-in] <ip>
show log traffic sport [equal | not-equal] <1-65535>
show log traffic dport [equal | not-equal] <1-65535>
show log traffic app <app_name>
show log traffic query equal <regex_query_like_in_gui>
It is best to limit the generated output to a specific time-frame as a too broad a filter makes it slightly laggy.
ste@HME-PAL-OEW1> show log traffic receive_time in last-60-seconds rule equal "Green to iNET" src in 10.10.64.203
Time App From Src Port Source
Rule Action To Dst Port Destination
Src User Dst User End Reason
Rule_UUid
====================================================================================================
2022/10/15 10:55:31 ssl south 62062 10.10.64.203
Green to iNET allow north 443 176.255.202.82
tcp-rst-from-server
88cf7e5c-2d4e-41ba-bb28-e62fa073f820
2022/10/15 10:55:31 ssl south 62060 10.10.64.203
Green to iNET allow north 443 176.255.202.82
tcp-rst-from-server
88cf7e5c-2d4e-41ba-bb28-e62fa073f820
Action: The action that was taken for the session
show log traffic action equal [allow | deny | drop | drop-icmp | reset-both | reset-client | reset-server]
- allow: Session was allowed by policy
- deny: Session was denied by policy, this indicates the rule identified a specific application. If traffic is dropped before the application is identified, such as when a rule drops all traffic for a specific service, the application will be shown as not-applicable
- drop: Session was dropped silently indicating that the security rule that blocked the traffic specified the application as any
- drop ICMP: Session was silently dropped with an ICMP unreachable message to the host or application
- reset both: Session was terminated and a TCP reset is sent to both the sides of the connection
- reset client: Session was terminated and a TCP reset is sent to the client
- reset server: Session was terminated and a TCP reset is sent to the server
Session-end-reason: The reason a session terminated. If their are multiple causes it only displays the highest priority reason (the explanations below list these in order of priority)
show log traffic session-end-reason equal [aged-out | decoder | decrypt-cert-validation | decrypt-error | decrypt-unsupport-param | n/a | policy-deny | resources-unavailable | tcp-fin | tcp-reuse | tcp-rst-from-client | tcp-rst-from-server | threat | unknown]
- threat: The firewall detected a threat associated with a reset, drop, or block (IP address) action
- policy-deny: The session matched a security rule with a deny or drop action
- decrypt-cert-validation: The session terminated because the firewall is configured to block client (certificate) authentication or if the session uses a server certificate which is expired, untrusted issuer, unknown status, status verification time-out, type bad_certificate, unsupported_certificate, certificate_revoked, access_denied, or no_certificate_RESERVED
- decrypt-unsupport-param: The firewall blocks sessions that use an unsupported protocol version, cipher, or SSH algorithm
- decrypt-error: The firewall blocks sessions when firewall resources for the hardware security module (HSM) are unavailable or are other SSL errors that are not covered by decrypt-cert-validation and decrypt-unsupport-param
- tcp-rst-from-client: The client sent a TCP reset to the server
- tcp-rst-from-server: The server sent a TCP reset to the client
- resources-unavailable: The session dropped because of a system resource limitation
- tcp-fin: Both hosts in the connection sent a TCP FIN message to close the session
- tcp-reuse: A session is reused and the firewall closes the previous session
- decoder: The decoder detects a new connection within the protocol (such as HTTP-Proxy) and ends the previous connection
- aged-out: The session aged out
- unknown: Session terminations that the preceding reasons do not cover (for example, a clear session all command)
- n/a: This value applies when the traffic log type is not end
show-tracker equal yes includes the reason why a the session was closed (Session Info), this is show as the tracker stage firewall in the session table.
ste@HME-PAL-OEW1> show log traffic receive_time in last-60-seconds rule equal "Green to iNET" src in 10.10.64.203 show-tracker equal yes
Time App From Src Port Source
Rule Action To Dst Port Destination
Src User Dst User Session Info End Reason
Rule_UUid
================================================================================================
2022/10/15 11:00:26 outlook-web-onl south 62052 10.10.64.203
Green to iNET allow north 443 52.98.207.146
ctd decoder done tcp-fin
TCP FIN
The traffic log is one of the many log databases that can be viewed under monitor » log in the GUI. All log entries have a magnifying glass to the left of them which opens a detailed log of the session that includes NAT information and the flags applied. The table at the bottom is a clickable list of related log entries from other log databases such as vulnerability, URL filtering or end (traffic log entry created by log-session end).
The same RegEx queries used in the GUI can also be used in the CLI to filter the traffic log.
show log traffic query equal "(( addr.src in 10.10.64.202 ) and ( port.dst eq 80 or port.dst eq 443 )) and (action neq allow)"
In the top left of the GUI from right to left there are icons for add filter (filter builder), save filter, load filter or export to CSV.
Reference
The Palo App-ID Wiki lists all the applications and information about them (although doesn’t mention the app dependencies)
An interesting discussion on the Palo forum regards allowing unwanted SSL and Web-browsing Internet Access
KB articles on application dependencies and a list of application implicit dependencies
KB article on data required to identify and application, how to create a custom application using the SNI, a custom application to stop application re-categorisation and this article is a more about how to allow Microsoft Office 365 so that it is recognized as well as making custom applications and URL categories
A good article on using cli to view counters to track packets through firewall
Palo KB articles on sessions and the session tracker feature
Fairly old but still relevant, some great troublehooting tips and commands from itsecworks in part1 and part2
Mastering Palo Alto Networks by Tom Piens is a well formatted book to get started and find more in depth info on Palos, there are some handy cheatsheets on the the books github page