Google Searches Odd Behavior

[jberg712]

I wanted to reach out to see if anyone has been experiencing similar issues.  We've had several users indicate to us that when they do Google searches and attempt to click on a link, it just hangs and spins.  You can click the address bar and hit enter a couple of times or re-click the link a few times and it finally goes to the link.  I can take the system out from behind the firewall and everything works seamlessly.  I've even put a troubleshooting rule to allow all outbound for myself to test and I still experience the same issue.  Strangely enough, if I use firefox, I don't experience the issue.  It's in Edge and Chrome which led me to believe a recent update occurred to created a problem.  But again, when I take the system out from behind the firewall, everything works fine.  So I don't know if there's a component in Chrome/Edge that CheckPoint is struggling with parsing or what.  Again, even adding a rule that allows all outbound access doesn't change the behavior.  Has anyone else come across this behavior?

[the_rock]

Never had that issue myself, even in the lab with ssl inspection on. Do you use ssl inspection? Regardless, when did the issue happen? Is it affecting all users?

Andy

[jberg712]

We do use SSL Inspection.  It appears to be affecting all users behind the gateway.  I know some services with Google are inspected and some are not.

[the_rock]

Is it same behavior no matter what browser they use? I attached a doc with some screenshots of how I have this set up in the lab and works fine.

Andy

[AdiGH]

Are you using Harmony Browse or Harmony Endpoint products? or only the firewall? 

[jberg712]

We are not using Harmony Endpoint or Browse.  It's only Secure Gateway with Firewall, App Cntrl, URL filtering, HTTPS Inspection, along with threat protection like IPS, Antibot, Antivirus.

[PhoneBoy]

You posted this issue in a space for Harmony products, which are Endpoint based.
This sounds like a gateway issue, so I'm moving this thread to the correct area.

[Lesley]

I think this one is the issue:

https://support.checkpoint.com/results/sk/sk111754

[the_rock]

Definitely could be related...

Andy

[PhoneBoy]

Please provide version/JHF of gateway as well as any logs that occur during the time this problem is occuring.
I would explicitly block QUIC in your access policy if you have not already as this is not supported for HTTPS Inspection until R82.

[jberg712]

I actually did some more digging into this.  (We have the QUIC protocol disabled and have for sometime now across the board).  What I noticed is that there is some odd correlation between Chrome/Edge and the HTTP2 protocol.  When I disable HTTP2 by using the flag on the shortcut --disable-http2, traffic to sites works tremendously better.  The logs are indicating when I go to a site like google and do a search, I'm matched properly to the correct rule, however, at some point a Redirect log is generated and the redirect does not match properly and it's being picked up on the a compliance rule.  The best example I have so far of this is a user in marketing department who uses facebook to update our facebook page.  I'll attach the screenshots.

I've dug around and all I can see when it comes to HTTP2 is to disable strict hold on SK116022 along with SK180257 and SK180673.  I'm afraid to disable inspecting of HTTP2 with fear that leads to things being vulnerable during web browsing for the end user.  Disabling strict hold didn't help the situation.  

I have a TAC ticket open about this.  Are there any recommendations/best practices when it comes to HTTP2 protocol?  It's been out for nearly 10 years now so I can't imagine why there would still be issues with it.  Unless Chrome made some update in the latest releases that did something funky with it.  I can't find good info on release notes for Chrome/Edge releases.  I say that because in one scenario I put on Firefox for another user and his works without issue unlike Chrome/Edge.  It's like a perfect mixture of chaos between Chrome, CheckPoint, and HTTP2.

We are currently on R81.20 Take 76

[PhoneBoy]

We support HTTP/2 for HTTPS Inspection as of R81 (though I believe it requires USFW support).
What hardware is this on?
HTTP/3 will be supported for HTTPS Inspection in R82.

[jberg712]

It's on a 6600 SGW.  

[jberg712]

@PhoneBoy Question regarding USFW and KSFW.  I believe in the past we've always been recommended to use KSFW.  I believe when we were running R81.10 on our 6600, we did end up swapping to KSFW because of strange performance issues.  It may have reverted back to USFW when we did the upgrade to R81.20.  What is usually recommended for overall performance?  Should we stay with USFW or consider moving to KSFW?

[Lesley]

This depends. Sharing below output could help to answer the question. 

Best Practices

Use the factors listed below to select the best CoreXL Firewall mode for your Security Gateway - User Space (USFW) or Kernel Space (KSFW):

Factor	Testing command	Recommended Firewall mode
80% or more of the traffic undergoes the Fast / Accelerated path	fwaccel stats -s	KSFW
70% or more of the traffic undergoes the Firewall / Slow path	fwaccel stats -s	KSFW
30% or more of the traffic undergoes the PXL / Medium path	fwaccel stats -s	USFW
Security Gateway is configured with more CoreXL SND instances than CoreXL Firewall instances, or when CoreXL SND instances are the bottleneck	fw ctl affinity -l -r	KSFW
Security Gateway is configured with more than 38 CoreXL Firewall instances	fw ctl affinity -l -r	USFW

[jberg712]

@Lesley  I was looking at that as well Lesley.  The part i'm having trouble understanding is the 'fw ctl affinity' results.

Here are the fwaccel stats:

fwaccel stats -s
Accelerated conns/Total conns : 18/14309 (0%)
LightSpeed conns/Total conns : 0/14309 (0%)
Accelerated pkts/Total pkts : 10138306524/10716219296 (94%)
LightSpeed pkts/Total pkts : 0/10716219296 (0%)
F2Fed pkts/Total pkts : 577912772/10716219296 (5%)
F2V pkts/Total pkts : 157378648/10716219296 (1%)
CPASXL pkts/Total pkts : 7960499954/10716219296 (74%)
PSLXL pkts/Total pkts : 2176883828/10716219296 (20%)
CPAS pipeline pkts/Total pkts : 0/10716219296 (0%)
PSL pipeline pkts/Total pkts : 0/10716219296 (0%)
QOS inbound pkts/Total pkts : 0/10716219296 (0%)
QOS outbound pkts/Total pkts : 0/10716219296 (0%)
Corrected pkts/Total pkts : 0/10716219296 (0%)

The 'fw ctl affinity' says this:  

fw ctl affinity -l -r
CPU 0:
CPU 1:
CPU 2: fw_3 (active)
cprid lpd mta_monitor cp_file_convertd usrchkd mpdaemon fwd wsdnsd rad cserver rtmd pepd watermark_cp_file_convertd in.aftpd in.msd scrubd in.ahclientd scanengine_b in.asessiond scrub_cp_file_convertd core_uploader pdpd in.emaild.mta vpnd in.acapd cprid cpd msgd
CPU 3: fw_2 (active)
cprid lpd mta_monitor cp_file_convertd usrchkd mpdaemon fwd wsdnsd rad cserver rtmd pepd watermark_cp_file_convertd in.aftpd in.msd scrubd in.ahclientd scanengine_b in.asessiond scrub_cp_file_convertd core_uploader pdpd in.emaild.mta vpnd in.acapd cprid cpd msgd
CPU 4: fw_1 (active)
cprid lpd mta_monitor cp_file_convertd usrchkd mpdaemon fwd wsdnsd rad cserver rtmd pepd watermark_cp_file_convertd in.aftpd in.msd scrubd in.ahclientd scanengine_b in.asessiond scrub_cp_file_convertd core_uploader pdpd in.emaild.mta vpnd in.acapd cprid cpd msgd
CPU 5: fw_0 (active)
cprid lpd mta_monitor cp_file_convertd usrchkd mpdaemon fwd wsdnsd rad cserver rtmd pepd watermark_cp_file_convertd in.aftpd in.msd scrubd in.ahclientd scanengine_b in.asessiond scrub_cp_file_convertd core_uploader pdpd in.emaild.mta vpnd in.acapd cprid cpd msgd
All:
Interface eth1: has multi queue enabled
Interface eth5: has multi queue enabled
Interface eth1-01: has multi queue enabled
Interface eth1-02: has multi queue enabled
Interface Mgmt: has multi queue enabled

I'm not sure how to read how many CoreXL instances vs CoreXL SND instances from these results.  Are you able to tell?

[Lesley]

Looks like 2 SND and the res FWK's to be sure verify with cpview -> cpu

You will see either CoreXL_SND or OTHER. And CoreXL_FW

[jberg712]

CPview indicates 2 CPUs for CoreXL_SND and 4 CPUs for CoreXL_FW

[PhoneBoy]

From a performance perspective, the table @Lesley provided gives you a good idea when to use USFW.
Note that some features require USFW (e.g. HTTPS Inspection for TLS 1.3 and HTTP/2).
I suspect it will be the default mode in a future version.

[Lesley]

Hi,

Please check bug:

PRJ-51049, PMTR-97496

Security Gateway

In some scenarios, websites that use HTTP2 protocol do not load properly.

This is solved in R81.20 take 79 (released yesterday for R81.20 and today for R81.10

[jberg712]

Thank you for pointing and finding this out Lesley.  I'm going to prepare to update my management and gateways with this to see if this resolves the issue.  

[jberg712]

Update on this...  I updated to Take 79.  Still having the odd behavior in Google searches.  I have a TAC case open about this.  Since the update though, I see logs indicating the 'URL Filtering - Internal System Error (0)'.  I've done some debugging and when I did the debug for APPI and NRB I found an error line stating: [ERROR]: rad_kernel_urlf_request_serialize: string len =4288 bigger than max 4096;

I can't seem to find anything more about this in SK's.  Below is a broader snippet of the log during the failure event.

@;3832104.2122417;22Aug2024 8:49:15.448813;[vs_0];[tid_0];[fw4_0];1:{global} appi_rad_uf_cmi_handler_is_enabled_ex: with _urlf_blade_enabled=0 -> *_enabled=1;
@;3832104.2122418;22Aug2024 8:49:15.448815;[vs_0];[tid_0];[fw4_0];1:{referrer} appi_clobs_observer_http_inspect_referrer: md5 is 2100488168;
@;3832104.2122419;22Aug2024 8:49:15.448817;[vs_0];[tid_0];[fw4_0];1:{referrer} [WARNING]: appi_clobs_observer_http_inspect_referrer: transaction's application is NULL;
@;3832104.2122420;22Aug2024 8:49:15.448818;[vs_0];[tid_0];[fw4_0];1:{observer} appi_clobs_observer_notify_exe_rb_event: returned retval 0;
@;3832104.2122421;22Aug2024 8:49:15.448833;[vs_0];[tid_2];[fw4_2];1:{policy} appi_rad_uf_cmi_handler_match_cb_handle_url: call rad_kernel_api_get_cached_resource;
@;3832104.2122422;22Aug2024 8:49:15.448857;[vs_0];[tid_1];[fw4_1];1:{policy} appi_rad_uf_cmi_handler_match_cb_handle_url: call rad_kernel_api_get_cached_resource;
@;3832104.2122423;22Aug2024 8:49:15.448867;[vs_0];[tid_2];[fw4_2];[172.16.16.31:53447 -> 108.177.122.147:443] [ERROR]: rad_kernel_urlf_request_serialize: string len =4288 bigger than max 4096;
@;3832104.2122424;22Aug2024 8:49:15.448871;[vs_0];[tid_2];[fw4_2];1:[172.16.16.31:53447 -> 108.177.122.147:443] {policy} [ERROR]: appi_rad_uf_cmi_handler_match_cb_handle_url: rad_kernel_api_async_get_resource_with_status_code() failed, error: Request flat buffer error;
@;3832104.2122425;22Aug2024 8:49:15.448874;[vs_0];[tid_2];[fw4_2];1:[172.16.16.31:53447 -> 108.177.122.147:443] {policy} [ERROR]: appi_rad_uf_cmi_handler_match_cb: appi_rad_uf_cmi_handler_match_cb_handle_url() failed;
@;3832104.2122426;22Aug2024 8:49:15.448876;[vs_0];[tid_2];[fw4_2];1:{global} appi_global_policy_get_fail_action: fail action is REJECT;
@;3832104.2122427;22Aug2024 8:49:15.448877;[vs_0];[tid_2];[fw4_2];1:{connection} appi_rad_uf_cmi_handler_match_cb: perform fail action [Reject];
@;3832104.2122428;22Aug2024 8:49:15.448890;[vs_0];[tid_1];[fw4_1];[172.16.16.31:53446 -> 108.177.122.147:443] [ERROR]: rad_kernel_urlf_request_serialize: string len =4275 bigger than max 4096;
@;3832104.2122429;22Aug2024 8:49:15.448891;[vs_0];[tid_2];[fw4_2];1:{policy} appi_rad_uf_cmi_handler_match_cb: ended;
@;3832104.2122430;22Aug2024 8:49:15.448894;[vs_0];[tid_1];[fw4_1];1:[172.16.16.31:53446 -> 108.177.122.147:443] {policy} [ERROR]: appi_rad_uf_cmi_handler_match_cb_handle_url: rad_kernel_api_async_get_resource_with_status_code() failed, error: Request flat buffer error;
@;3832104.2122431;22Aug2024 8:49:15.448897;[vs_0];[tid_1];[fw4_1];1:[172.16.16.31:53446 -> 108.177.122.147:443] {policy} [ERROR]: appi_rad_uf_cmi_handler_match_cb: appi_rad_uf_cmi_handler_match_cb_handle_url() failed;
@;3832104.2122432;22Aug2024 8:49:15.448899;[vs_0];[tid_1];[fw4_1];1:{global} appi_global_policy_get_fail_action: fail action is REJECT;
@;3832104.2122433;22Aug2024 8:49:15.448900;[vs_0];[tid_1];[fw4_1];1:{connection} appi_rad_uf_cmi_handler_match_cb: perform fail action [Reject];
@;3832104.2122434;22Aug2024 8:49:15.448912;[vs_0];[tid_1];[fw4_1];1:{policy} appi_rad_uf_cmi_handler_match_cb: ended;

[Lesley]

Is the issue still there if you bypass the following categories in HTTPS inspection?

Google Services and Google Cloud Platform Services

[jberg712]

Give me a moment Lesley and I'll test

[the_rock]

Or try below.

Andy

 

[the_rock]

Very good point @Lesley 

[jberg712]

I bypassed Google services (all of them) and I do not get the error.  Everything loads like it's supposed to.  So it would indicate something with HTTPS inspection?

[the_rock]

100%, if bypass works.

[the_rock]

URLF database, does it show updated in smart console?

Andy

[jberg712]

Andy, it states it's up-to-date, but the latest version is 141202408011245 (Created on 8/1/2024 7:45 AM)  Is that the most updated DB?