Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
jberg712
Collaborator

Google Searches Odd Behavior

I wanted to reach out to see if anyone has been experiencing similar issues.  We've had several users indicate to us that when they do Google searches and attempt to click on a link, it just hangs and spins.  You can click the address bar and hit enter a couple of times or re-click the link a few times and it finally goes to the link.  I can take the system out from behind the firewall and everything works seamlessly.  I've even put a troubleshooting rule to allow all outbound for myself to test and I still experience the same issue.  Strangely enough, if I use firefox, I don't experience the issue.  It's in Edge and Chrome which led me to believe a recent update occurred to created a problem.  But again, when I take the system out from behind the firewall, everything works fine.  So I don't know if there's a component in Chrome/Edge that CheckPoint is struggling with parsing or what.  Again, even adding a rule that allows all outbound access doesn't change the behavior.  Has anyone else come across this behavior?

44 Replies
the_rock
Legend
Legend

Never had that issue myself, even in the lab with ssl inspection on. Do you use ssl inspection? Regardless, when did the issue happen? Is it affecting all users?

Andy

jberg712
Collaborator

We do use SSL Inspection.  It appears to be affecting all users behind the gateway.  I know some services with Google are inspected and some are not.

the_rock
Legend
Legend

Is it same behavior no matter what browser they use? I attached a doc with some screenshots of how I have this set up in the lab and works fine.

Andy

 

AdiGH
Employee
Employee

Are you using Harmony Browse or Harmony Endpoint products? or only the firewall? 

jberg712
Collaborator

We are not using Harmony Endpoint or Browse.  It's only Secure Gateway with Firewall, App Cntrl, URL filtering, HTTPS Inspection, along with threat protection like IPS, Antibot, Antivirus.

PhoneBoy
Admin
Admin

You posted this issue in a space for Harmony products, which are Endpoint based.
This sounds like a gateway issue, so I'm moving this thread to the correct area.

Lesley
Leader Leader
Leader

I think this one is the issue:

https://support.checkpoint.com/results/sk/sk111754

 

-------
If you like this post please give a thumbs up(kudo)! 🙂
the_rock
Legend
Legend

Definitely could be related...

Andy

PhoneBoy
Admin
Admin

Please provide version/JHF of gateway as well as any logs that occur during the time this problem is occuring.
I would explicitly block QUIC in your access policy if you have not already as this is not supported for HTTPS Inspection until R82.

jberg712
Collaborator

I actually did some more digging into this.  (We have the QUIC protocol disabled and have for sometime now across the board).  What I noticed is that there is some odd correlation between Chrome/Edge and the HTTP2 protocol.  When I disable HTTP2 by using the flag on the shortcut --disable-http2, traffic to sites works tremendously better.  The logs are indicating when I go to a site like google and do a search, I'm matched properly to the correct rule, however, at some point a Redirect log is generated and the redirect does not match properly and it's being picked up on the a compliance rule.  The best example I have so far of this is a user in marketing department who uses facebook to update our facebook page.  I'll attach the screenshots.

I've dug around and all I can see when it comes to HTTP2 is to disable strict hold on SK116022 along with SK180257 and SK180673.  I'm afraid to disable inspecting of HTTP2 with fear that leads to things being vulnerable during web browsing for the end user.  Disabling strict hold didn't help the situation.  

I have a TAC ticket open about this.  Are there any recommendations/best practices when it comes to HTTP2 protocol?  It's been out for nearly 10 years now so I can't imagine why there would still be issues with it.  Unless Chrome made some update in the latest releases that did something funky with it.  I can't find good info on release notes for Chrome/Edge releases.  I say that because in one scenario I put on Firefox for another user and his works without issue unlike Chrome/Edge.  It's like a perfect mixture of chaos between Chrome, CheckPoint, and HTTP2.

We are currently on R81.20 Take 76

PhoneBoy
Admin
Admin

We support HTTP/2 for HTTPS Inspection as of R81 (though I believe it requires USFW support).
What hardware is this on?
HTTP/3 will be supported for HTTPS Inspection in R82.

jberg712
Collaborator

It's on a 6600 SGW.  

jberg712
Collaborator

@PhoneBoy Question regarding USFW and KSFW.  I believe in the past we've always been recommended to use KSFW.  I believe when we were running R81.10 on our 6600, we did end up swapping to KSFW because of strange performance issues.  It may have reverted back to USFW when we did the upgrade to R81.20.  What is usually recommended for overall performance?  Should we stay with USFW or consider moving to KSFW?

Lesley
Leader Leader
Leader

This depends. Sharing below output could help to answer the question. 

Best Practices

Use the factors listed below to select the best CoreXL Firewall mode for your Security Gateway - User Space (USFW) or Kernel Space (KSFW):

Factor Testing command Recommended
Firewall
mode
80% or more of the traffic undergoes the Fast / Accelerated path fwaccel stats -s KSFW
70% or more of the traffic undergoes the Firewall / Slow path fwaccel stats -s KSFW
30% or more of the traffic undergoes the PXL / Medium path fwaccel stats -s USFW
Security Gateway is configured with more CoreXL SND instances than CoreXL Firewall instances,
or when CoreXL SND instances are the bottleneck
fw ctl affinity -l -r KSFW
Security Gateway is configured with more than 38 CoreXL Firewall instances fw ctl affinity -l -r USFW
-------
If you like this post please give a thumbs up(kudo)! 🙂
jberg712
Collaborator

@Lesley  I was looking at that as well Lesley.  The part i'm having trouble understanding is the 'fw ctl affinity' results.

Here are the fwaccel stats:

fwaccel stats -s
Accelerated conns/Total conns : 18/14309 (0%)
LightSpeed conns/Total conns : 0/14309 (0%)
Accelerated pkts/Total pkts : 10138306524/10716219296 (94%)
LightSpeed pkts/Total pkts : 0/10716219296 (0%)
F2Fed pkts/Total pkts : 577912772/10716219296 (5%)
F2V pkts/Total pkts : 157378648/10716219296 (1%)
CPASXL pkts/Total pkts : 7960499954/10716219296 (74%)
PSLXL pkts/Total pkts : 2176883828/10716219296 (20%)
CPAS pipeline pkts/Total pkts : 0/10716219296 (0%)
PSL pipeline pkts/Total pkts : 0/10716219296 (0%)
QOS inbound pkts/Total pkts : 0/10716219296 (0%)
QOS outbound pkts/Total pkts : 0/10716219296 (0%)
Corrected pkts/Total pkts : 0/10716219296 (0%)

The 'fw ctl affinity' says this:  

fw ctl affinity -l -r
CPU 0:
CPU 1:
CPU 2: fw_3 (active)
cprid lpd mta_monitor cp_file_convertd usrchkd mpdaemon fwd wsdnsd rad cserver rtmd pepd watermark_cp_file_convertd in.aftpd in.msd scrubd in.ahclientd scanengine_b in.asessiond scrub_cp_file_convertd core_uploader pdpd in.emaild.mta vpnd in.acapd cprid cpd msgd
CPU 3: fw_2 (active)
cprid lpd mta_monitor cp_file_convertd usrchkd mpdaemon fwd wsdnsd rad cserver rtmd pepd watermark_cp_file_convertd in.aftpd in.msd scrubd in.ahclientd scanengine_b in.asessiond scrub_cp_file_convertd core_uploader pdpd in.emaild.mta vpnd in.acapd cprid cpd msgd
CPU 4: fw_1 (active)
cprid lpd mta_monitor cp_file_convertd usrchkd mpdaemon fwd wsdnsd rad cserver rtmd pepd watermark_cp_file_convertd in.aftpd in.msd scrubd in.ahclientd scanengine_b in.asessiond scrub_cp_file_convertd core_uploader pdpd in.emaild.mta vpnd in.acapd cprid cpd msgd
CPU 5: fw_0 (active)
cprid lpd mta_monitor cp_file_convertd usrchkd mpdaemon fwd wsdnsd rad cserver rtmd pepd watermark_cp_file_convertd in.aftpd in.msd scrubd in.ahclientd scanengine_b in.asessiond scrub_cp_file_convertd core_uploader pdpd in.emaild.mta vpnd in.acapd cprid cpd msgd
All:
Interface eth1: has multi queue enabled
Interface eth5: has multi queue enabled
Interface eth1-01: has multi queue enabled
Interface eth1-02: has multi queue enabled
Interface Mgmt: has multi queue enabled

I'm not sure how to read how many CoreXL instances vs CoreXL SND instances from these results.  Are you able to tell?

Lesley
Leader Leader
Leader

Looks like 2 SND and the res FWK's to be sure verify with cpview -> cpu

You will see either CoreXL_SND or OTHER. And CoreXL_FW

-------
If you like this post please give a thumbs up(kudo)! 🙂
jberg712
Collaborator

CPview indicates 2 CPUs for CoreXL_SND and 4 CPUs for CoreXL_FW

 

CoreXLinfo_cpview.png

PhoneBoy
Admin
Admin

From a performance perspective, the table @Lesley provided gives you a good idea when to use USFW.
Note that some features require USFW (e.g. HTTPS Inspection for TLS 1.3 and HTTP/2).
I suspect it will be the default mode in a future version.

Lesley
Leader Leader
Leader

Hi,

Please check bug:

PRJ-51049,
PMTR-97496

Security Gateway

In some scenarios, websites that use HTTP2 protocol do not load properly.

 

This is solved in R81.20 take 79 (released yesterday for R81.20 and today for R81.10

-------
If you like this post please give a thumbs up(kudo)! 🙂
jberg712
Collaborator

Thank you for pointing and finding this out Lesley.  I'm going to prepare to update my management and gateways with this to see if this resolves the issue.  

jberg712
Collaborator

Update on this...  I updated to Take 79.  Still having the odd behavior in Google searches.  I have a TAC case open about this.  Since the update though, I see logs indicating the 'URL Filtering - Internal System Error (0)'.  I've done some debugging and when I did the debug for APPI and NRB I found an error line stating: [ERROR]: rad_kernel_urlf_request_serialize: string len =4288 bigger than max 4096;

I can't seem to find anything more about this in SK's.  Below is a broader snippet of the log during the failure event.

@;3832104.2122417;22Aug2024 8:49:15.448813;[vs_0];[tid_0];[fw4_0];1:{global} appi_rad_uf_cmi_handler_is_enabled_ex: with _urlf_blade_enabled=0 -> *_enabled=1;
@;3832104.2122418;22Aug2024 8:49:15.448815;[vs_0];[tid_0];[fw4_0];1:{referrer} appi_clobs_observer_http_inspect_referrer: md5 is 2100488168;
@;3832104.2122419;22Aug2024 8:49:15.448817;[vs_0];[tid_0];[fw4_0];1:{referrer} [WARNING]: appi_clobs_observer_http_inspect_referrer: transaction's application is NULL;
@;3832104.2122420;22Aug2024 8:49:15.448818;[vs_0];[tid_0];[fw4_0];1:{observer} appi_clobs_observer_notify_exe_rb_event: returned retval 0;
@;3832104.2122421;22Aug2024 8:49:15.448833;[vs_0];[tid_2];[fw4_2];1:{policy} appi_rad_uf_cmi_handler_match_cb_handle_url: call rad_kernel_api_get_cached_resource;
@;3832104.2122422;22Aug2024 8:49:15.448857;[vs_0];[tid_1];[fw4_1];1:{policy} appi_rad_uf_cmi_handler_match_cb_handle_url: call rad_kernel_api_get_cached_resource;
@;3832104.2122423;22Aug2024 8:49:15.448867;[vs_0];[tid_2];[fw4_2];[172.16.16.31:53447 -> 108.177.122.147:443] [ERROR]: rad_kernel_urlf_request_serialize: string len =4288 bigger than max 4096;
@;3832104.2122424;22Aug2024 8:49:15.448871;[vs_0];[tid_2];[fw4_2];1:[172.16.16.31:53447 -> 108.177.122.147:443] {policy} [ERROR]: appi_rad_uf_cmi_handler_match_cb_handle_url: rad_kernel_api_async_get_resource_with_status_code() failed, error: Request flat buffer error;
@;3832104.2122425;22Aug2024 8:49:15.448874;[vs_0];[tid_2];[fw4_2];1:[172.16.16.31:53447 -> 108.177.122.147:443] {policy} [ERROR]: appi_rad_uf_cmi_handler_match_cb: appi_rad_uf_cmi_handler_match_cb_handle_url() failed;
@;3832104.2122426;22Aug2024 8:49:15.448876;[vs_0];[tid_2];[fw4_2];1:{global} appi_global_policy_get_fail_action: fail action is REJECT;
@;3832104.2122427;22Aug2024 8:49:15.448877;[vs_0];[tid_2];[fw4_2];1:{connection} appi_rad_uf_cmi_handler_match_cb: perform fail action [Reject];
@;3832104.2122428;22Aug2024 8:49:15.448890;[vs_0];[tid_1];[fw4_1];[172.16.16.31:53446 -> 108.177.122.147:443] [ERROR]: rad_kernel_urlf_request_serialize: string len =4275 bigger than max 4096;
@;3832104.2122429;22Aug2024 8:49:15.448891;[vs_0];[tid_2];[fw4_2];1:{policy} appi_rad_uf_cmi_handler_match_cb: ended;
@;3832104.2122430;22Aug2024 8:49:15.448894;[vs_0];[tid_1];[fw4_1];1:[172.16.16.31:53446 -> 108.177.122.147:443] {policy} [ERROR]: appi_rad_uf_cmi_handler_match_cb_handle_url: rad_kernel_api_async_get_resource_with_status_code() failed, error: Request flat buffer error;
@;3832104.2122431;22Aug2024 8:49:15.448897;[vs_0];[tid_1];[fw4_1];1:[172.16.16.31:53446 -> 108.177.122.147:443] {policy} [ERROR]: appi_rad_uf_cmi_handler_match_cb: appi_rad_uf_cmi_handler_match_cb_handle_url() failed;
@;3832104.2122432;22Aug2024 8:49:15.448899;[vs_0];[tid_1];[fw4_1];1:{global} appi_global_policy_get_fail_action: fail action is REJECT;
@;3832104.2122433;22Aug2024 8:49:15.448900;[vs_0];[tid_1];[fw4_1];1:{connection} appi_rad_uf_cmi_handler_match_cb: perform fail action [Reject];
@;3832104.2122434;22Aug2024 8:49:15.448912;[vs_0];[tid_1];[fw4_1];1:{policy} appi_rad_uf_cmi_handler_match_cb: ended;

Lesley
Leader Leader
Leader

Is the issue still there if you bypass the following categories in HTTPS inspection?

Google Services and Google Cloud Platform Services

-------
If you like this post please give a thumbs up(kudo)! 🙂
jberg712
Collaborator

Give me a moment Lesley and I'll test

the_rock
Legend
Legend

Or try below.

Andy

 

Screenshot_1.png

the_rock
Legend
Legend

Very good point @Lesley 

jberg712
Collaborator

I bypassed Google services (all of them) and I do not get the error.  Everything loads like it's supposed to.  So it would indicate something with HTTPS inspection?

the_rock
Legend
Legend

100%, if bypass works.

the_rock
Legend
Legend

URLF database, does it show updated in smart console?

Andy

jberg712
Collaborator

Andy, it states it's up-to-date, but the latest version is 141202408011245 (Created on 8/1/2024 7:45 AM)  Is that the most updated DB?

Leaderboard

Epsum factorial non deposit quid pro quo hic escorol.

Upcoming Events

    CheckMates Events