Web traffic Strange behaviour through our 3800 device

[jovovicm]

Hello,

I am facing with Web traffic related problems through our internal Next Gen Check Point 3800 Firewall device, which is positioned at Internet Edge.

Firewall device is 3800 with R81.10 Gaia version, and is managed by Smart-1 Cloud service.

Active blades are: fw, vpn, urlf, av, appi, ips, identityServer, and anti_bot.

CP 3800 device processes 99% of traffic with slow path (F2F), only 1% is being accelerated.

70% of drops are by rulebase, and around 30% of drops are by Drop out of state TCP (First packet is not SYN). We have seen that there are many drops for web traffic with First packet is not SYN (ACK flag). Do You think it is okay?

We have our direct Internet connection to ISP over pppoe connection which is terminated directly on our Check Point device. This pppoe interface/connection is terminated on phisical interface eth1. This connection is not problem.

On inside side of our Check Point firewall (towards our internal networks) there is one bonding configured which consists of two phisical interfaces (eth2 and eth3).  This bond is configured as trunk which and it has various VLAN interfaces configured (for example VLAN11, VLAN40). All VLAN's are terminated on Check Point's VLAN interfaces, which is at the same time DGW (default gateway for all Internet clients - cable/LAN and wifi clients).

Our internal Internet clients are separated to two groups:

1. WiFi clients

- use VLAN40 (10.51.40.0/24)

- VLAN 40 interface have DHCP server configured on it which gives DNS servers IP's to clients

- WiFi Internet flow:

Wifi client > Wifi AP > Cisco 9200 Switch > Check Point 3800 > Internet

2. LAN/Cable clients

- use VLAN11 (10.51.11.0/24)

- VLAN 11 interface have DHCP server configured on it which gives DNS servers IP's to clients

- LAN Internet flow:

LAN/cable client > Cisco 9200 Switch > Check Point 3800 > Internet

Problem definition:

LAN clients gets connected to network (gets IP by Check Point DHCP server on VLAN11).

LAN clients have ICMP reply which works without any problems.

LAN clients have DNS connection and resolving works without any problems.

But problem is with TCP (80 and 443) traffic.

When connected to Internet over LAN/cable we have this problem with Web (80/443) traffic which. Web pages are not loading for approximatelly 20 seconds (when we hit Enter in browser). And then after that time page loads but not completely. Some pages can not be opened at all (yahoo.com for example). Some are opening almost like it should (with minimal content loading problems).

Facts:

Our Check Point 3800 device do not have problems with it's capacity/hardware (CPU, memory, connections), it has 10G free memory (16G is on device), and CPU utilization is preety low.

We do not have a problem with DNS service communication at all (all DNS servers are reachable and name resolving works all time).

We do not have routing issues too, our internal network do not use routing protocols (only default GW which is/points to check point's VLAN interface).

We tried with switch-ing off URLF and app control security blades on our firewall, but web traffic problem was still in place.

When we use WiFi as a way of connecting to Internet, we do not such problems. Web traffic works as a charm.

This is problematic traffic flow through our FW. From traffic capture perspective we can not find any problem:

Incoming traffic to CP bond20.11 interface (29 inspection dots, it goes from dot number 9 because secureXL is not used):

[vs_0][fw_0] 31Jan2023 19:43:07.645996 bond20.11:i9 (tcpt inbound)[44]: 10.51.11.101 -> 20.123.104.105 (TCP) len=52 id=29936 [vs_0][fw_0] 31Jan2023 19:43:07.646001 bond20.11:i10 (IP Options Strip (in))[44]: 10.51.11.101 -> 20.123.104.105 (TCP) len=52 id=29936

[vs_0][fw_0] 31Jan2023 19:43:07.646005 bond20.11:i11 (vpn multik forward in)[44]: 10.51.11.101 -> 20.123.104.105 (TCP) len=52 id=29936

[vs_0][fw_0] 31Jan2023 19:43:07.646010 bond20.11:i12 (vpn decrypt)[44]: 10.51.11.101 -> 20.123.104.105 (TCP) len=52 id=29936 [vs_0][fw_0] 31Jan2023 19:43:07.646014 bond20.11:i13 (l2tp inbound)[44]: 10.51.11.101 -> 20.123.104.105 (TCP) len=52 id=29936 [vs_0][fw_0] 31Jan2023 19:43:07.646018 bond20.11:i14 (Stateless verifications (in))[44]: 10.51.11.101 -> 20.123.104.105 (TCP) len=52 id=29936

[vs_0][fw_0] 31Jan2023 19:43:07.646022 bond20.11:i15 (fw multik misc proto forwarding)[44]: 10.51.11.101 -> 20.123.104.105 (TCP) len=52 id=29936

[vs_0][fw_0] 31Jan2023 19:43:07.646025 bond20.11:i16 (vpn tagging inbound)[44]: 10.51.11.101 -> 20.123.104.105 (TCP) len=52 id=29936

[vs_0][fw_0] 31Jan2023 19:43:07.646029 bond20.11:i17 (vpn decrypt verify)[44]: 10.51.11.101 -> 20.123.104.105 (TCP) len=52 id=29936

[vs_0][fw_0] 31Jan2023 19:43:07.646032 bond20.11:i18 (NAC Packet Inbound)[44]: 10.51.11.101 -> 20.123.104.105 (TCP) len=52 id=29936

[vs_0][fw_0] 31Jan2023 19:43:07.646035 bond20.11:i19 (fw VM inbound )[44]: 10.51.11.101 -> 20.123.104.105 (TCP) len=52 id=29936

[vs_0][fw_0] 31Jan2023 19:43:07.646041 bond20.11:I20 (vpn policy inbound)[44]: 10.51.11.101 -> 20.123.104.105 (TCP) len=52 id=29936

[vs_0][fw_0] 31Jan2023 19:43:07.646045 bond20.11:I21 (fw SCV inbound)[44]: 10.51.11.101 -> 20.123.104.105 (TCP) len=52 id=29936

[vs_0][fw_0] 31Jan2023 19:43:07.646048 bond20.11:I22 (vpn before offload)[44]: 10.51.11.101 -> 20.123.104.105 (TCP) len=52 id=29936

[vs_0][fw_0] 31Jan2023 19:43:07.646052 bond20.11:I23 (fw offload inbound)[44]: 10.51.11.101 -> 20.123.104.105 (TCP) len=52 id=29936

[vs_0][fw_0] 31Jan2023 19:43:07.646055 bond20.11:I24 (fw post VM inbound )[44]: 10.51.11.101 -> 20.123.104.105 (TCP) len=52 id=29936

[vs_0][fw_0] 31Jan2023 19:43:07.646062 bond20.11:I25 (fw accounting inbound)[44]: 10.51.11.101 -> 20.123.104.105 (TCP) len=52 id=29936

[vs_0][fw_0] 31Jan2023 19:43:07.646067 bond20.11:I26 (passive streaming (in))[44]: 10.51.11.101 -> 20.123.104.105 (TCP) len=52 id=29936

[vs_0][fw_0] 31Jan2023 19:43:07.646074 bond20.11:I27 (TCP streaming (in))[44]: 10.51.11.101 -> 20.123.104.105 (TCP) len=52 id=29936

[vs_0][fw_0] 31Jan2023 19:43:07.646079 bond20.11:I28 (IP Options Restore (in))[44]: 10.51.11.101 -> 20.123.104.105 (TCP) len=52 id=29936

[vs_0][fw_0] 31Jan2023 19:43:07.646082 bond20.11:I29 (Chain End)[44]: 10.51.11.101 -> 20.123.104.105 (TCP) len=52 id=29936

Outgoing traffic from CP, over pppoe1 interface (22 inspection dots, dots are from 0 to 9, because it do not use VPN/encryption nor secureXL):

[vs_0][fw_0] 31Jan2023 19:43:07.646118 pppoe1:o0 (IP Options Strip (out))[44]: 10.51.11.101 -> 20.123.104.105 (TCP) len=52 id=29936

[vs_0][fw_0] 31Jan2023 19:43:07.646123 pppoe1:o1 (vpn multik forward out)[44]: 10.51.11.101 -> 20.123.104.105 (TCP) len=52 id=29936

[vs_0][fw_0] 31Jan2023 19:43:07.646127 pppoe1:o2 (vpn nat outbound)[44]: 10.51.11.101 -> 20.123.104.105 (TCP) len=52 id=29936 [vs_0][fw_0] 31Jan2023 19:43:07.646131 pppoe1:o3 (TCP streaming (out))[44]: 10.51.11.101 -> 20.123.104.105 (TCP) len=52 id=29936

[vs_0][fw_0] 31Jan2023 19:43:07.646134 pppoe1:o4 (passive streaming (out))[44]: 10.51.11.101 -> 20.123.104.105 (TCP) len=52 id=29936

[vs_0][fw_0] 31Jan2023 19:43:07.646138 pppoe1:o5 (vpn tagging outbound)[44]: 10.51.11.101 -> 20.123.104.105 (TCP) len=52 id=29936

[vs_0][fw_0] 31Jan2023 19:43:07.646140 pppoe1:o6 (Stateless verifications (out))[44]: 10.51.11.101 -> 20.123.104.105 (TCP) len=52 id=29936

[vs_0][fw_0] 31Jan2023 19:43:07.646143 pppoe1:o7 (NAC Packet Outbound)[44]: 10.51.11.101 -> 20.123.104.105 (TCP) len=52 id=29936

[vs_0][fw_0] 31Jan2023 19:43:07.646146 pppoe1:o8 (fw VM outbound)[44]: 10.51.11.101 -> 20.123.104.105 (TCP) len=52 id=29936 [vs_0][fw_0] 31Jan2023 19:43:07.646151 pppoe1:O9 (fw post VM outbound )[44]: 10.51.11.101 -> 20.123.104.105 (TCP) len=52 id=29936 

Additional facts:

All traffic is almost F2F, fwaccell entries donot help as traffic is not accelerated at all (zero hits):

I created pcap files for problematic traffic (web port 80, test web page is http://vikom.tv which represents all our web traffic).

I have chosen this particular web page because it is some legacy page with only single IP behind it (simplifies fwmonitor capture).

If someone can help it I can send it for t-shoot.

Prior to fwmonitor I cleared general firewall connection table to this IP for 80 and 443 ports:

99% of our traffic is F2F, only 1% is accelerated.

Accelerated connections (only 15):

F2F connections (ove 7K):

What I have seen that all web connections (wifi and cable) arelways being sent over F2F path.

This is for non working cable connections:

It is the same for web connection which work from wifi:

This is connection particularly towards vikom.tv web server:80 (connection which works - from wifi):

MSS size is same for working Wifi Internet clients/connections:

and for LAN/cable Internet clients/connections:

Any thoughts are welcome.

Regards,

Milos

[Chris_Atkinson]

PPPoE will cause F2F.

MSS clamping is likely required.

Which JHF version is used for GW?

[jovovicm]

Hello Chris,

Thanks for Your prompt hint.

pppoe interface has mtu 1492 and inside bond is with 1500 MTU (ethernet).

MSS (payload) change is sensible, can You provide exact steps which is usable for our case? 

This device have VPN (S2S traffic) which does not have problems. Why web traffic works for wifi clients? 

Installed Jumbo is 66.

Thank You.

[Chris_Atkinson]

For steps please see sk61221 and similar SKs, using Smart-1 Cloud some steps may require the involvement of TAC.

Note later Jumbo takes have fixes that may assist with the "First Packet isn't Syn" depending on exact cause.

Perhaps the WiFi controller itself already does mss-clamping, common for some to avoid fragmentation where there is tunneling between APs and controller.

[jovovicm]

Indeed we use Smart-1 cloud and it is used for 3 GW's on 3 locations (not just ours). And if MSS configuration is changed, it has to be done not only on GW side, it needs to be done on Smart-1 Cloud side. This part is tricky as Smart-1 cloud do not allow us to use SSH to MGMT server and configuration is sensible (database is not as easy as in on prem: 3 policies, ...).

We use WiFi AP 9100.

[jovovicm]

Any additional thoughts are welcome. What is strange that on another location we had the same pppoe Internet connection,

and SMB device 1590, and our cable Internet worked.

[PhoneBoy]

If you need to make specific changes on the management side, that can be done by the TAC.