Information on how applications are identified by App-ID and following sessions and traffic flows through the firewall using the CLI.
Table Of Contents
Identifying an Application
Based on the first TCP packet received (SYN) an application will be marked as incomplete as it can not be detected until at least one data packet traverses the device. App-ID will go through several stages to identify just what something is.
- 6-Tuple: Packet is checked against the security policy to verify whether the source, destination, protocol, and port combination is allowed
- Application: The packet is checked against known application signatures and the app cache followed by a second security policy check now adding App-ID to the required set of identifiers for the security policy to allow the session through
- Decrypt: If the application is SSH, TLS, or SSL and matches the decryption policy the session is decrypted and checked again for a known application signature as the encapsulated session may be something entirely different
- Decoder: If by now the application has not been identified (maximum of 4 packets after the handshake or 2000 bytes) App-ID will use the base protocol to determine which decoder to use to analyze the packets more deeply
- Known protocol: It is decoded and the payload run against the known application signatures again. The outcome could either be a known application or an unknown generic application like unknown-tcp. The session is then again re-matched against the security policy
- Unknown protocol: App-ID will apply heuristics to try and determine which protocol is used in the session. Once it is determined which protocol is used another security policy check is performed
- Stop: Once the application has been identified or all options have been exhausted, App-ID will stop processing the packets for identification
Throughout the life of a session, the identified application may change several times as more info is learned from the session through inspecting packet after packet. Whenever an application shift happens a new security policy lookup is performed to find the closest matching rule. Once the App-ID process has settled on an application the application decoder will continuously scan the session for expected and deviant behavior.
In parallel to each packet being processed by an App-ID decoder the Content-ID stream-based engine applies the URL filtering policy and scans the session for threats like vulnerability exploits, virus or worm infections, suspicious DNS queries, command and control signatures or DoS attacks. This ‘in parallel’ design reduces latency meaning that enabling more security profiles does not come at an exponential performance cost.
To get the most out of App-ID in terms of application recognition the firewall should function as an SSL forward proxy and decrypt SSL traffic going to external sites. For non-decrypted HTTPS traffic the firewall can use the Server Name Indication (SNI) for URL categorization, if the client does not send the SNI then the Common Name (CN) is used (if is a wildcard the application is identified as SSL). Custom applications can be created using the SNI to control SSL traffic without the need for SSL decryption, although it is worth noting that the when the SNI is added to the TLS handshake it is not in the encrypted conversation meaning that it can be spoofed.
Applications can have either implicit or explicit dependant applications. For example facebook-base dependant applications will be implicitly used needing no explicit rules whilst ms-office365-base requires explicit rules for the applications that it depends on. This can be viewed using
show predefined application <application_name> in configuration mode of the CLI as implicit-use-applications and use-applications.
If an App-ID application relies on ssl and web-browsing it is extremely likely that these are what are used in the initial part of the conversation to identify the application (some applications redirect to another website, like for authentication).
- ssl: Identifies HTTPS applications as this based on the SSL handshake. If that first ssl request is not allowed in the security policy the session will never get to identifying the application. After that request is allowed either the packet is decrypted or the firewall reads the SNI of the certificate and if possible a more specific application ID identified and the application re-evaluated against policy with that ID
- web-browsing: Identifies applications as this based typical browsing behavior such as an HTTP GET. The web-browsing app on port 443 is only matched/shown for firewalls doing SSL decryption, otherwise it shows up as the ssl application.
web-browsing and ssl only match generic HTTP/HTTPS traffic, if the firewall determines a more precise App-ID for the traffic then it will switch to using that for determining whether to allow or deny this traffic. If traffic does not match a more specific application and is only classified as ssl or web-browsing it will be allowed by any rules with these applications in. To guard against holes in the security policy allowing unsanctioned applications need to lock down the access of these applications to specific destinations and URL categories (be it pre-defined, custom or EDLs). It is a compromise between giving enough access to recognize the applications vs allowing too much unknown ssl or web-browsing traffic.
The session table and traffic log are good places to start when trying to understand the traffic flows through the firewall.
- Session browser: Active sessions currently passing through the firewall, once the sessions end can see them in the traffic log
- Traffic Log: Historic information for sessions that have already ended (via the Log at Session End option in the security policy)
- Unified Log: Displays the Traffic, Threat URL Filtering, WildFire, and Data Filtering logs all in the same single-pane view
A session is defined by two uni-directional flows each uniquely identified by a 6-tuple key of source/destination IP and port, protocol and security-zone. Each entry has a lot of information regarding the session that can not be found in the traffic logs.
show session info
Numbers of active sessions, statistics throughput timers and TCP/UDP settingsshow session meter Displays maximum, current and throttled number of sessions for each VSYSshow session all All active sessions, limits number shown based session byte sizeshow session all filter key value Can filter based on most of the attributes of a sessionshow session all filter key valuecount yes Count the number of sessions matching filtershow session id id To see a detailed view of a specific session
Each session is identified by two uni-directional flows, client-to-server(c2s) and the returning server-to-client(s2c).
ste@HME-PAL-OEW1> show session all -------------------------------------------------------------------------------- ID Application State Type Flag Src[Sport]/Zone/Proto (translated IP[Port]) Vsys Dst[Dport]/Zone (translated IP[Port]) -------------------------------------------------------------------------------- 48968 spotify ACTIVE FLOW NS 10.10.64.224/south/6 (192.168.1.202) vsys1 126.96.36.199/north (188.8.131.52)
State: There is total 7 possible session states that a flow can be in. The 3 stable states represent most of the sessions seen in the session table as the other 4 transient states are shortlived and used during the transition to one of the stable states.
show session all filter state [initial | opening | active | closing | discard | closed]
- INIT (stable): A session that is ready and waiting to be used by a new flow. Every session begins in this state
- OPENING (transient): A session ID is assigned to the flow while it is being evaluated to become a full session (has aggressive timers to handle half-open TCP connections)
- ACTIVE (stable): This is the state in which everything happens – the flow is up and packets are being passed back and forth
- CLOSING (transient): Session timeout has been reached for the flow (TTL or idle-timeout) meaning the session is set to expire soon but has not yet been removed from aging process or session lookup table. New packets no longer match against this session so are instead either queued to create a new session or discarded as they no longer match an active session (non-SYN TCP)
- DISCARD (stable): Traffic that has been matched by a session but is denied due to a drop/deny security policy rule or a threat detection block. All packets matching the session will be discarded for the duration of the discard phase
- CLOSED (transient): The session has been removed from the aging process, but not from the session lookup table. No new packets can match this session so are either queued for a new session or dropped
- FREE (transient): The session has been closed and removed from the session lookup table but still needs to be made available for a new flow
Under normal conditions each state will go through the transition cycle of INIT > OPENING > ACTIVE > DISCARD/CLOSING > CLOSED > FREE. From the FREE state the session will move back to the INIT to start the next cycle.
Type: There are also five session types of which the two most common seen in the session table are FLOW or PRED.
show session all filter type [flow | forward | predict | tunnel | vni]
- FLOW: These are all the regular sessions where the flow is the same between c2s and s2c (HTTP, Telnet, SSH, etc)
- PRED: Application Layer Gateway (ALG) protocols that require a return session be set up outside of the established session use a predict session to anticipate the inbound connection and transform this into a FLOW if the return session is received (FTP active/passive, SIP, etc)
- FORW: Captive portal intercepting and redirecting browsing sessions to a login page or policy-based forwarding is applied to a flow
- Tunnel: VPN connections will be set up in a Tunnel session
- VNI: If VXLAN Tunnel Content Inspection (TCI) is enabled in Policies » Tunnel Inspection VXLAN tunnels will be vni-type sessions
Flags: Information related to the type of NAT used. A flag with an asterisk (*) indicates that the session was decrypted.
show session all filter nat [source | destination | both | none]
- NS (NAT Source): The source address was translated
- ND (NAT Destination): The destination address was translated
- NB (NAT Bidirectional): Both the source and destination addresses were translated
- No flag: There is no NAT applied on the session
Individual sessions show a lot of additional information regarding the traffic flow, some of more interesting fields are:
- Timeout: Specific timeout configured for the application, is the amount of time a session is allowed to exist
- Time to live: Time left until the session will expire and the state change
- Session in session ager: Aging process that keeps track of the lifetime of sessions, will be True for Active sessions when TTL not reached 0
- Session to be logged at the end: Configured in the security rules, can be set to the end (True) or the start (False) of the session
- Offload yes: Marks traffic for which the application has already been identified and will be processed in hardware
- Layer7 processing: App-ID has been enabled on the traffic flow and the application is constantly identified (complete means the application has been identified)
- tracker stage (firewall): Provides information on the precise reasons for mitigation actions taken on particular session
ste@HME-PAL-OEW1> show session id 173457 Session 173457 c2s flow: source: 10.10.64.223 [south] dst: 184.108.40.206 proto: 6 sport: 50210 dport: 80 state: ACTIVE type: FLOW src user: unknown dst user: unknown s2c flow: source: 220.127.116.11 [north] dst: 192.168.1.202 proto: 6 sport: 80 dport: 42596 state: ACTIVE type: FLOW src user: unknown dst user: unknown start time : Thu Oct 13 12:16:41 2022 timeout : 3600 sec time to live : 3598 sec total byte count(c2s) : 1107545 total byte count(s2c) : 656987 layer7 packet count(c2s) : 15518 layer7 packet count(s2c) : 8468 vsys : vsys1 application : spotify rule : Green to iNET service timeout override(index) : False session to be logged at end : True session in session ager : True session updated by HA peer : False address/port translation : source nat-rule : Green PAT(vsys1) layer7 processing : completed URL filtering enabled : False session via syn-cookies : False session terminated on host : False session traverses tunnel : False session terminate tunnel : False captive portal session : False ingress interface : ethernet1/2 egress interface : ethernet1/1 session QoS rule : N/A (class 4) tracker stage l7proc : ctd decoder done end-reason : unknown
The session table is viewable in the GUI under Monitor » Session Browser, it can be exported and saved to PDF or CSV.
If a network connection failure can not found in the traffic log it is worth checking for DISCARD sessions in the session table.
show session all filter state
discard Dropped traffic has a discard state and end-reason of policy-denyshow session all filter application dns destination 18.104.22.168 Match all DNS traffic to googleshow session all filter from trustto untrustapplication sslstate active Match all active SSL traffic between zones
Forcibly terminating a session will immediately set the sessions state to INIT so that any packets still arriving on the firewall will not have a session to match against meaning that they are either discarded as non-SYN TCP or evaluated as a new session. The CLI seems more flexible than the GUI (done under Clear column) for clearing sessions as sessions can be cleared based on the same filters used to view sessions.
clear session allclear session id
idshow session all filter key valuecount yesclear session all filter key value
For each sessions log action (start or end) a traffic log entry is created that includes the standard 6-tuple Key (source/destination IP and port, protocol and security-zone), users, the application (or lack of), bytes, packets and the reason for the action applied by a security policy.
When viewing in the CLI the majority of filter options can be chained together. Adding
csv-output equal yes displays the output in csv format.
show log traffic receive_time in [last-60-seconds | last-15-minutes | last-hour | last-6-hrs | last-12-hrs | last-24-hrs] show log traffic receive_time in [last-7-days | last-30-days | last-calendar-day | last-calendar-month] show log traffic start-time [equal | not-equal] <YYYY/MM/DD@hh:mm:ss> show log traffic end-time [equal | not-equal] <YYYY/MM/DD@hh:mm:ss> show log traffic rule [equal | not-equal] <rule_name> show log traffic rule_uuid [equal | not-equal] <rule_uuid> show log traffic from [equal | not-equal] <zone> show log traffic to [equal | not-equal] <zone> show log traffic src [in | not-in] <ip> show log traffic dst [in | not-in] <ip> show log traffic sport [equal | not-equal] <1-65535> show log traffic dport [equal | not-equal] <1-65535> show log traffic app <app_name> show log traffic query equal <regex_query_like_in_gui>
It is best to limit the generated output to a specific time-frame as a too broad a filter makes it slightly laggy.
ste@HME-PAL-OEW1> show log traffic receive_time in last-60-seconds rule equal "Green to iNET" src in 10.10.64.203 Time App From Src Port Source Rule Action To Dst Port Destination Src User Dst User End Reason Rule_UUid ==================================================================================================== 2022/10/15 10:55:31 ssl south 62062 10.10.64.203 Green to iNET allow north 443 22.214.171.124 tcp-rst-from-server 88cf7e5c-2d4e-41ba-bb28-e62fa073f820 2022/10/15 10:55:31 ssl south 62060 10.10.64.203 Green to iNET allow north 443 126.96.36.199 tcp-rst-from-server 88cf7e5c-2d4e-41ba-bb28-e62fa073f820
Action: The action that was taken for the session
show log traffic action equal [allow | deny | drop | drop-icmp | reset-both | reset-client | reset-server]
- allow: Session was allowed by policy
- deny: Session was denied by policy, this indicates the rule identified a specific application. If traffic is dropped before the application is identified, such as when a rule drops all traffic for a specific service, the application will be shown as not-applicable
- drop: Session was dropped silently indicating that the security rule that blocked the traffic specified the application as any
- drop ICMP: Session was silently dropped with an ICMP unreachable message to the host or application
- reset both: Session was terminated and a TCP reset is sent to both the sides of the connection
- reset client: Session was terminated and a TCP reset is sent to the client
- reset server: Session was terminated and a TCP reset is sent to the server
Session-end-reason: The reason a session terminated. If their are multiple causes it only displays the highest priority reason (the explanations below list these in order of priority)
show log traffic session-end-reason equal [aged-out | decoder | decrypt-cert-validation | decrypt-error | decrypt-unsupport-param | n/a | policy-deny | resources-unavailable | tcp-fin | tcp-reuse | tcp-rst-from-client | tcp-rst-from-server | threat | unknown]
- threat: The firewall detected a threat associated with a reset, drop, or block (IP address) action
- policy-deny: The session matched a security rule with a deny or drop action
- decrypt-cert-validation: The session terminated because the firewall is configured to block client (certificate) authentication or if the session uses a server certificate which is expired, untrusted issuer, unknown status, status verification time-out, type bad_certificate, unsupported_certificate, certificate_revoked, access_denied, or no_certificate_RESERVED
- decrypt-unsupport-param: The firewall blocks sessions that use an unsupported protocol version, cipher, or SSH algorithm
- decrypt-error: The firewall blocks sessions when firewall resources for the hardware security module (HSM) are unavailable or are other SSL errors that are not covered by decrypt-cert-validation and decrypt-unsupport-param
- tcp-rst-from-client: The client sent a TCP reset to the server
- tcp-rst-from-server: The server sent a TCP reset to the client
- resources-unavailable: The session dropped because of a system resource limitation
- tcp-fin: Both hosts in the connection sent a TCP FIN message to close the session
- tcp-reuse: A session is reused and the firewall closes the previous session
- decoder: The decoder detects a new connection within the protocol (such as HTTP-Proxy) and ends the previous connection
- aged-out: The session aged out
- unknown: Session terminations that the preceding reasons do not cover (for example, a clear session all command)
- n/a: This value applies when the traffic log type is not end
show-tracker equal yes includes the reason why a the session was closed (Session Info), this is show as the tracker stage firewall in the session table.
ste@HME-PAL-OEW1> show log traffic receive_time in last-60-seconds rule equal "Green to iNET" src in 10.10.64.203 show-tracker equal yes Time App From Src Port Source Rule Action To Dst Port Destination Src User Dst User Session Info End Reason Rule_UUid ================================================================================================ 2022/10/15 11:00:26 outlook-web-onl south 62052 10.10.64.203 Green to iNET allow north 443 188.8.131.52 ctd decoder done tcp-fin TCP FIN
The traffic log is one of the many log databases that can be viewed under monitor » log in the GUI. All log entries have a magnifying glass to the left of them which opens a detailed log of the session that includes NAT information and the flags applied. The table at the bottom is a clickable list of related log entries from other log databases such as vulnerability, URL filtering or end (traffic log entry created by log-session end).
The same RegEx queries used in the GUI can also be used in the CLI to filter the traffic log.
show log traffic query equal "(( addr.src in 10.10.64.202 ) and ( port.dst eq 80 or port.dst eq 443 )) and (action neq allow)"
In the top left of the GUI from right to left there are icons for add filter (filter builder), save filter, load filter or export to CSV.
The Palo App-ID Wiki lists all the applications and information about them (although doesn’t mention the app dependencies)
An interesting discussion on the Palo forum regards allowing unwanted SSL and Web-browsing Internet Access
KB articles on application dependencies and a list of application implicit dependencies
KB article on data required to identify and application, how to create a custom application using the SNI, a custom application to stop application re-categorisation and this article is a more about how to allow Microsoft Office 365 so that it is recognized as well as making custom applications and URL categories
A good article on using cli to view counters to track packets through firewall
Palo KB articles on sessions and the session tracker feature
Fairly old but still relevant, some great troublehooting tips and commands from itsecworks in part1 and part2
Mastering Palo Alto Networks by Tom Piens is a well formatted book to get started and find more in depth info on Palos, there are some handy cheatsheets on the the books github page