topic Re: R80.20 MTU and SecureXL Problem in Firewall and Security Management

R80.20 MTU and SecureXL Problem

Jan_Kleinhans — Fri, 21 Jun 2019 10:45:02 GMT

Hello,

we have a Ethernet-Link (no VPN from Checkpoint) to a network where the MTU is 1422. If we set the mtu on the interface and disable SecureXL the Clients (with default MTU of 1500) get the ICMP Fragmentation Packet and start to send packets with smaller MTU.

When we reactivate SecureXL the Clients starts to send 1500 byte packets again and do not get an ICMP Fragmentation paket from the Firewall.

We are using an Checkpoint 5600 Cluster with R80.20 with latest HFA.

Did anybody had the same problem?

Jan

Re: R80.20 MTU and SecureXL Problem

Timothy_Hall — Fri, 21 Jun 2019 13:18:03 GMT

Yep, see this SK: sk98070: Traffic sent over a VPN tunnel does not reach its destination because SecureXL does not start fragmenting the packets

For additional information: sk98074: MTU and Fragmentation Issues in IPsec VPN

Re: R80.20 MTU and SecureXL Problem

Jan_Kleinhans — Fri, 21 Jun 2019 14:57:58 GMT

Thanks for your quick answer.

But in our case the Firewall is not using IPSEC. It is only an interface with a smaller mtu. So changing the parameter to sim_ipsec_dont_fragment=1

should not make any difference or am I wrong?

Re: R80.20 MTU and SecureXL Problem

Timothy_Hall — Sat, 22 Jun 2019 01:18:57 GMT

How are you disabling SecureXL in R80.20? fwaccel off? vpn accel off?

If you disable SecureXL selectively for the involved IP addresses as specified here:

sk104468: How to disable SecureXL for specific IP addresses

but leave SecureXL active otherwise, do the ICMP Frag Needed packets still get sent?

Re: R80.20 MTU and SecureXL Problem

Jan_Kleinhans — Mon, 24 Jun 2019 06:12:34 GMT

Hello,

I disabled it with fwaccel off. This seems to work without problems.
I cannot exclude specific IP addresses as this seems to be a problem with all IP Adresses behind this network Interface (Possible IPs 192.168.0.0/16).
At the moment I will let the acceleration disabled and open a TAC.
I'm surprised, that I am the only one to have this problem with R80.20.

Thanks,

Jan

Re: R80.20 MTU and SecureXL Problem

Matt_Killeen — Wed, 26 Jun 2019 13:13:49 GMT

I don't think you are the only one with this problem. I think we have it as well.

We recently upgraded from R77.30 to R80.20 on a 15600 cluster and, prior to the upgrade to R80.20, communications from a public internet source to one of our web servers on port TCP 443 (HTTPs) completed as expected.

Following the upgrade, HTTPs communications stall due to TLS v1.2 Server Hello messages being sent with a length of 1514 and the client then sends ICMP TYPE 3 CODE 4 Destination Unreachable, Fragmentation messages which we appear to ignore.

The initial TCP SYN and SYN-ACK also contains the TCP Option : Maximum Segment Size : 1460 bytes so we shouldn't really be sending a 1514 byte packets back to the client.

I have a rolling packet capture on our public interfaces and there is evidence to confirm that the client was sending ICMP TYPE 3 CODE 4 Destination Unreachable, Fragmentation messages prior to the CheckPoint upgrade.

At the moment I’ve no idea whether r77.30 ignored the DF flag and fragmented the packets as requested or whether r77.30 forwarded the ICMP message to the server or whether it was handled differently elsewhere. So, we've escalated to vendor support.

Without a resolution to hand, we had to set the MTU to 1400 on the Web Server to take the firewall out of the equation and allow us to investigate further.

Over the next couple of days it became apparent that some private wan traffic was also affected and that turning off fwaccel resolved the issue and allowed the traffic to complete.

I'll keep a close watch on this thread and update if I have any further information.

Re: R80.20 MTU and SecureXL Problem

Matt_Killeen — Thu, 27 Jun 2019 11:03:59 GMT

UPDATE: turning Secure XL off caused intermittent connectivty problems for at least one identified web service for multiple public clients. Followed sk104468 in order to disable SecureXL for traffic sent from/to specific IP addresses - this allowed us to address the issue for specific private WAN traffic and switch Secure XL back on for everything else whilst we continue to troubleshoot.

Re: R80.20 MTU and SecureXL Problem

Matt_Killeen — Fri, 05 Jul 2019 11:11:57 GMT

UPDATE: I have been able to successfully recreate this issue using a lab client to the live service and compare r80.20 to r77.30 behaviour.

As we still have our work-a-round in place to protect our client wher the web server MTU is set at 1400, the numbers are reduced but the effect is the same.

The Lab is an Ubuntu client with an MTU of 1500 and a Cisco router with the LAN interface MTU set at 1360 (this is significant as it is this interface that doesn't want to transmit anything larger than 1360).

The Web server MTU is set at 1400 and is behind a CheckPoint r80.20 15600 Cluster.

During tests, the packet anaysis shows the SYN and SYN-ACK negotiate their preferred Maximum Segment Size (MSS) and that when the Cisco router receives a packets larger than it likes (due to the MTU set at 1360) , the packets are dropped and ICMP TYPE 3 CODE 4 messages are created and transmitted. CheckPoint r80.20 appears to ignore these ICMP messages and packets are re-transmitted until the HTTPs connection times out.

As we also have another CheckPoint 15600 cluster still running r77.30, the same lab test was run against a live web service NAT'd behind that cluster (the MTU of the live web server is 1500 here but the effect is the same).

In this test, the packet anaysis shows the SYN and SYN-ACK again negotiate their preferred Maximum Segment Size (MSS) and that when the Cisco router receives a packet larger than it likes (due to the MTU set at 1360) , the packet is dropped and an ICMP TYPE 3 CODE 4 message is created and transmitted. The difference being is that, from that point on, packets sent are lower than the Cisco's 1360 MTU and the HTTPs transaction completes successfully.

These results have being passed to CheckPoint support and we are scheduling further debug and testing with CheckPoint earl next week in an out of hours window.

Will keep this thread updated.

Matt.

Re: R80.20 MTU and SecureXL Problem

HeikoAnkenbrand — Fri, 05 Jul 2019 18:40:15 GMT

SecureXL has been significantly revised in R80.20.

There are new fw monitor chain (SecureXL) objects that do not run in the virtual machine.

SecureXL offloading chain modules

# fw ctl chain

The new fw monitor chain modules (SecureXL) do not run in the virtual machine (vm).

SecureXL inbound (sxl_in) > Packet received in SecureXL from network
SecureXL inbound CT (sxl_ct) > Accelerated packets moved from inbound to outbound processing (post routing)
SecureXL outbound (sxl_out) > Accelerated packet starts outbound processing
SecureXL deliver (sxl_deliver) > SecureXL transmits accelerated packet

More see here:
- R80.x Security Gateway Architecture (Logical Packet Flow)
- R80.20 - New FW Monitor inspection points

If there are problems with the MTU size, you should open a TAC ticket.

@Timothy_Hall

CUT>>>
If you disable SecureXL selectively for the involved IP addresses as specified here:

sk104468: How to disable SecureXL for specific IP addresses
<<<CUT

This solution does not resolve MTU Size problems. The MTU Size always affects the interface and does not affect single IP addresses in sk104468

Re: R80.20 MTU and SecureXL Problem

Matt_Killeen — Wed, 10 Jul 2019 10:49:51 GMT

Following remote session with checkpoint we were able to determine our issue is most likely patched in the latest GA hotfix. (see https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk147492).

In order to resolve this we will be scheduling the installation of the latest hotfix version (87 is GA as of yesterday).

Once scheduled, installed and tested, I will update this thread.

Matt.

Re: R80.20 MTU and SecureXL Problem

Jan_Kleinhans — Tue, 16 Jul 2019 12:57:21 GMT

Update:

We have already installed the HFA Take 87. It does not help with our problem.

Also we got a Hotfix from Checkpoint for Take 87 that does not work at the moment.

Checkpoint is working on a solution.

Re: R80.20 MTU and SecureXL Problem

Vladimir — Tue, 16 Jul 2019 15:53:03 GMT

@PhoneBoy , can we get someone from Check point to weigh-in on this issue?

Additionally, I'd like to know if those running R80.30 GA are experiencing the same.

I suspect that at least one of my larger clients was battling the issues caused by this behavior for months without resolution after the upgrade to R80.20.

Re: R80.20 MTU and SecureXL Problem

PhoneBoy — Tue, 16 Jul 2019 17:15:17 GMT

It would help for those who have the issue to send me the SR number in a PM or via email to my username AT checkpoint DOT com.

Re: R80.20 MTU and SecureXL Problem

Hellen5394 — Wed, 17 Jul 2019 09:26:25 GMT

did you get any proper solution?, I am also facing the same issue.

Re: R80.20 MTU and SecureXL Problem

Peter_Lyndley — Thu, 18 Jul 2019 14:12:33 GMT

Hi All, this may be slightly off topic, however we are also seeing something similar on latest versions of embedded Gaia (R77.20.70+), havent seen it before that version but dont know exactly when it came in. Turning off SecureXL works , or lowering external interface MTU

Re: R80.20 MTU and SecureXL Problem

Khalid_Aftas — Wed, 31 Jul 2019 09:16:21 GMT

We encountered the same issue after upgrading to R80.20.

CAPWAP tunnel traffic between a WLC and the Anchor is impacted, the FW drops packet because fragmentation table is full.

CP suggested to SK65074 to increase IP Fragment table size from 200 to 2000.

Fixed the issue only for some few hours, next day we escalated the SR and they told us to pute 4000, fixed the issue atm, but i think it will happen again.

Something has changed in R80.20 that's for sure, Support could not provide more info as they need to check with r&d.

Any news from your side ?

Re: R80.20 MTU and SecureXL Problem

Timothy_Hall — Thu, 01 Aug 2019 03:04:23 GMT

What changed in R80.20 is the ability of the SecureXL driver to perform virtual defragmentation of packets. In R80.10 and earlier, any fragmented packets received by SecureXL would be instantly sent to the F2F path on a Firewall Worker core for handling. It is possible that the new SecureXL virtual defragmention code is somehow leaking table entries and not freeing them back up in a timely fashion (or ever) once the virtual defrag timer has expired. If that is indeed the case, increasing the size as specified in sk65074 will only delay the inevitable. This assumes of course that changing that value from the IPS signature actually applies to the SecureXL virtual defragmentation code, and not just the F2F virtual defragmention code which is what that setting was originally created to modify.

Re: R80.20 MTU and SecureXL Problem

Khalid_Aftas — Thu, 01 Aug 2019 07:20:21 GMT

Thank you Timothy for your valuable explanation.

Applying a new value, instantly freed up the drops from some time, i indeed was thinking the same that it will only delay the problem, and suspect something with the table being stuck or something.

That's a bummer for us, we need to see what solution CP will provide, at the mean time r&d is not even involved.

Is there anyway to bypass this by disableing Secure XL for specific IPs ?

Re: R80.20 MTU and SecureXL Problem

Matt_Killeen — Thu, 01 Aug 2019 10:15:22 GMT

Follow sk104468 in order to disable SecureXL for traffic sent from/to specific IP addresses

Re: R80.20 MTU and SecureXL Problem

Ilya_Yusupov — Thu, 01 Aug 2019 13:55:36 GMT

Hi all,

we found the issue and we have a fix that will be included on next JHF.

if someone want the fix immediately please open case in TAC and we will provide the fix.

Thanks,

Ilya