No Resources, such as URLs, are present in connections events

[Vladimir]

Is there a way to include the Resources, such as URLs in all connections events, not just in the Sessions?

I have observed,when using a non-transparent proxy, that the destination resources, while being present in "Session" events, are absent in subsequent connections, for as long as session is maintained.

It is possible that this behavior is not unique to Proxy traffic, but I did not yet get a chance to verify that.

The absence of the resources in connections makes it very difficult to pinpoint host-to-destination calls made over existing sessions based on time.

In case of proxy traffic, all you see is the http/s call from client to proxy with no idea of the destination.

Thank you,

Vladimir

[PhoneBoy]

First of all the relevant connections have to be matched:

• In a layer with App Control / URL Filtering enabled
• Where the Log field of the rule that matches is Detailed or Complete

It may be that this information is not logged in the individual connections, only in the summary session (if you have that feature enabled).

[Vladimir]

Well, here is the situation:

It is a flat policy with Access Control and URLF blades enabled.

Highlighted rule and corresponding log entries are shown:

Note that the bottom two sessions do contain the destination URL in description fields. The top session has it present in the "Resource" field. The other log entries, however, do not have any reference to the actual destination, just the connection to the proxy.

Session:

Id: 0a32320b-f112-0000-59da-834000000000
Marker: @A@@B@1507435200@C@202615
Log Server Origin: 192.168.7.30
Time: 2017-10-08T19:57:52Z
Interface Direction: inbound
Interface Name: eth2
Connection Direction: Outgoing
Id Generated By Indexer: false
First: false
Sequencenum: 1
Hll Key: 12212875363894272126
Duration: 2100
Last Update Time: 2017-10-08T20:27:09Z
Update Count: 5
Connections: 5
Aggregated Log Count: 8
Creation Time: 2017-10-08T19:57:52Z
Source: 10.55.55.74
Destination: 10.50.50.70
Destination Port: 8080
IP Protocol: 6
Client Type Os: Unknown
Client Type: Other: Wget/1.18 (linux-gnu)
User Agent: Other: Wget/1.18 (linux-gnu)
Service ID: HTTP_proxy
Source Zone: Internal
Destination Zone: Local
Application ID: 1073741826
Application Signature ID:1073741826:1
Method: CONNECT
Packets: 110
Total Bytes: 55716
Client Inbound Packets: 44
Client Outbound Packets: 66
Server Inbound Packets: 0
Server Outbound Packets: 0
Client Inbound Bytes: 4520
Client Outbound Bytes: 51196
Server Inbound Bytes: 0
Server Outbound Bytes: 0
URLs: 4
Lastupdatetime: 2017-10-08T20:31:53Z
Action: Accept
Type: Session
Policy Name: Cluster01_Access_Contro_Policy
Policy Management: SMS8010
Db Tag: {B6D1D4A1-9A46-9B48-828B-68AF4720FDDF}
Policy Date: 2017-10-08T19:56:14Z
Blade: Application Control
Origin: Member_A
Service: TCP/8080
Product Family: Access
Received Bytes: 0
Sent Bytes: 0
Logid: 320
Application Name: GOOGLE
Application Description: Google_Main_URL
Primary Category: Custom Application/Site
Matched Category: Custom Application/Site
Additional Categories: Custom Application/Site, Medium Risk
Application Risk: Medium
Resource: https://www.google.com:443
Access Rule Number: 3
Rule UID: 882a57a2-ca32-4c60-8c36-0189f019eec5
Layer Name: Cluster01_Access_Contro_Policy Network
Interface: eth2
Description: HTTP_proxy Traffic Accepted from 10.55.55.74 to GOOGLE(10.50.50.70)
Bytes (sent\received): 54.4 KB (0 B \ 0 B)

Connection (note the session tab):

Session tab of connection:

Search query returning only sessions:

And the logging for the rule configured:

With "per Session" automatically selected.

So if the session is a long-lived and I am troubleshooting the connection now, it is difficult to match real-time traffic to the destination.

If same host has multiple sessions going through the proxy, the connections cannot be differentiated.

[Kfir_Dadosh]

With "Extended Logging" and "per connection" log generation you get all the URLs for both the session and the connection log.

In the logs view, select a specific connection or session log to the proxy, and look at the lower panel under the URLs tab.

You should see all the URLs that were accessed within the selected connection/session.

[Vladimir]

Kfir,

This is exactly what is NOT HAPPENING. Please re-read my post. Extended Logging and Per-Connection are enabled.

When logs are searched by URL, only the sessions are displayed, which may not correlate to the actual time segment I am interested in looking at.

Connection events do not contain the URL information.

[Kfir_Dadosh]

Extended logging logs each and every URL accessed (not only the domain / first URL browsed).

These URLs are stored in a different SOLR core which is not searchable from the main logs view.

The only way you can see those URLs is to find and select the relevant connection log, and see the relevant URLs logs in the bottom pane below.

Searching for the domain / first URL is only available for Sessions. You can see the relevant connection for this session in the bottom pane under "Connections" tab.

[Vladimir]

I understand how it works, just not why it doesn't work better.

As per your statement, "The only way you can see those URLs is to find and select the relevant connection log, and see the relevant URLs logs in the bottom pane below", I have to find and select the session before looking at connections.

If I am looking in the log for close to real-time troubleshooting, there is no way to identify the destination by URL, if the session was established few hours ago, and if there is an existing session which may be few hours old.

What would've been the downside of including the URLs in the individual connection events, given that the field is already there.

I.e. filter last hour of the log and then look for destinations by domain or url. What will you see if the session is older than that?

[Vladimir]

Just dug-up the reason for this in Application and URL Filtering - Advanced Settings 

Connection Unification

Application and Web site traffic generate a large quantity of logs. To make logs manageable, Application and URL Filtering consolidates logs by session. A session is a period that starts when the user first connects to an application or site. The Security Gateway generates one log entry for each application or site accessed during the session. All actions that occur during are included in the log.

To change the length of a session:

1. Go to Manage & Settings > blades > Application and URL Filtering > Advanced Settings.
2. In the Application Settings window:
   • For applications and sites that are allowed in a Rule Base, the default session is three hours (180 minutes). To change this, click Session Unification Timeout and enter a different value, in minutes.
   • For applications and sites that are blocked in the Rule Base, the default session is 30 seconds. You cannot change this setting.

This is inconvenient, as in cases when same host generates traffic to multiple destinations, the destination is not easily identifiable by URL, even if the connection is logged. What is the possible reason for excluding Application And URL data from connections, if we are already logging them (All actions that occur during are included in the log.)?