HTTPS Inspection logs and App Control with URL Filtering

[YvheniiK]

Hello,
I'm a bit confused about the logs' reflection on URL filtering in the Check Point NGFW.

For example, I configured HTTPS inspection and App and URL filtering on the device.
go to YouTube and then see the logs about visiting youtube.com.

but I see just some general information in the application logs and URL filtering logs
In HTTPS inspection logs, I can see information about the video resource that I watched.

Why don't I see it on the application or URL filtering logs?

I mean the information about visiting rr4---sn-5hne6nsk.googlevideo.com.

[Lesley]

Both URL filtering and https inspections have different values that are in a log entry. I posted them below. It could that there is a difference between them. You could open the full log entry and compare them with the table below. In case of the youtube.com log entry I suspect it just uses the info from the certificate itself *.google.com has also youtube.com in it. 

Field Name

Field Display Name

Type

Description

Indexed

Added in Version

Security Gateway - HTTPS Inspection Fields - R81.20 and lower
https_inspection_action	Inspection Action	string	HTTPS Inspection action (Inspect/Bypass/Error)	Yes
https_inspection_rule_id	HTTPS Inspection Rule ID	string	ID of the matched rule	Yes
https_inspection_rule_name	HTTPS Inspection Rule Name	string	Name of the matched rule	Yes
app_properties	Additional Categories	string	List of all found categories (match table)	No
resource	Resource	string	HTTPS resource Possible values: SNI Domain Name	Yes
https_validation	HTTPS Validation	string	Precise error, describing the HTTPS inspection failure	Yes
description	Description	string	Additional information about the "https_validation" field	Yes
reason	Reason	string	Explains the action decision	Yes

Field Name

Field Display Name

Type

Description

Indexed

Added in Version

Security Gateway - Application Control & URL Filtering Fields
appi_name	Application Name	string	Application name (match table)	Yes
app_desc	Application Description	string	Application description (match table)	No
app_id	Application ID	int	Application ID (match table)	No
app_properties	Additional Categories	string	Application categories (match table)	Yes
app_risk	Application Risk	int	Application risk (match table) Possible values: 0 - Unknown 1 - Very low 2 - Low 3 - Medium 4 - High 5 - Critical	Yes
app_rule_id	Application Rule ID	string	Rule number	Yes
app_rule_name	Application Rule Name	string	Rule name	No
app_sig_id	Application Signature ID	string	The signature ID, by which the application was detected (match table)	Yes
categories	Categories	string	Matched categories	Yes
certificate_resource	Resource	string	HTTPS resource Possible values: SNI Domain Name (DN)	Yes	R80.40
certificate_validation	Certificate Validation	string	Precise error, describing HTTPS certificate failure under "HTTPS categorize websites" feature	Yes	R80.40
description	Description	string	Additional explanation about the certificate validation failure	Yes	R80.40
usercheck_incident_uid	UserCheck ID	string	UserCheck incident ID	No
usercheck_reference	UserCheck Reference	string	UserCheck reference	No
resource	Resource	string	HTTP connection resource	Yes
browse_time	Browse Time	time	Application session browse time	Yes
limit_requested	N/A	int	Indicates whether data limit was requested for the session	Yes
limit_applied	N/A	int	Indicates whether the session was actually date-limited	Yes
dropped_outgoing	N/A	int	Number of outgoing dropped packets	Yes
dropped_incoming	N/A	int	Number of incoming dropped packets	Yes
dropped_total	N/A	int	Number of dropped packets (both incoming and outgoing)	Yes
suppressed_logs	Suppressed Logs	int	Number of connections/HTTP sessions that were aggregated in this application session log	No
match_id	N/A	int	Mapping of matched rule to its matched application (match table)	Yes
client_type_os	N/A	string	Client OS detected in the HTTP request	Yes
referrer	N/A	string	The referrer header, if exists	Yes
name	N/A	string	Application name	Yes
properties	N/A	string	Application categories (match table)	Yes
risk	N/A	int	Application risk	Yes
sig_id	N/A	string	Application's signature ID, by which it was detected	Yes
desc	N/A	string	Override application description	Yes
referrer_self_uid	N/A	guid	UUID of the current log	Yes
referrer_parent_uid	N/A	guid	Log UUID of the referring application	Yes
needs_browse_time	N/A	int	Browse time required for the connection	Yes
security_inzone	N/A	string	Source security zone	Yes
security_outzone	N/A	string	Destination security zone	Yes
url	URL	string	Matched URL	Yes
app_byte_ps_in	Application Byte/Sec In	int	Incoming traffic of an application (Bytes per Second)	No
app_byte_ps_out	Application Byte/Sec Out/td>	int	Outgoing traffic of an application (Bytes per Second)	No
app_pack_ps_in	Application Packet/Sec In	int	Incoming traffic of an application (Packets per Second)	No
app_pack_ps_out	Application Packet/Sec Out/td>	int	Outgoing traffic of an application (Packets per Second)	No
matched_application	Matched Application	string	Name of the matched application	No

[YvheniiK]

That is clear to me, and I know that HTTPS-inspection and Application Control & URL Filtering loglines have different fields.
My question here is more about a count of these loglines.
When I have a YouTube session, I see that I have many more loglines in HTTPS-inspection than in Application Control & URL Filtering.
I have just 1 logline in Application Control & URL Filtering about youtube session, but 4-6 loglines (where specified various resources and dst IP) in HTTPS-inspection in the scope of this youtube session.
That means that if I want to get full information about web-filtering, then I need to pay attention to both logs "HTTPS-inspection" and "Application Control & URL Filtering".
And the worst thing here is that you can't correlate these two types of logs (by sessionid,loguid or somewhere else)

[Chris_Atkinson · Accepted solution]

You have not indicated how the logging/track field is currently configured for your related policy/rules?

[YvheniiK]

I configured it like this 

Also

[YvheniiK]

Oh sorry, 

enabling extended log solves this problem.
Thank you!

[the_rock]

Good job @YvheniiK 

[the_rock]

I totally see the point @Chris_Atkinson made here. The way logging options are configured may have something to do with it.

Andy

[the_rock]

I believe thats normal. I also see the same in my R82 lab as well.

Andy

[Chris_Atkinson]

Which level of logging is configured for the matching rule in the track column, remember there are additional options here i.e. Extended and Detailed.

Session vs connection logs may also be a factor...