Issues with 19200 Appliance migration: ICMP over VPN (UPPAK) and Driver Incompatibility (KPPAK)

[Jakub_K]

Hello Community,

We are currently in the middle of a hardware refresh, moving from an older cluster (16200) to a new 19200 appliance setup. However, we’ve hit a significant roadblock that has rendered the new appliance unusable in production. I’m curious if anyone else has encountered this specific issue between UPPAK and KPPAK modes on the 19000 series.

The Migration Path:

We followed the standard cluster replacement procedure (similar to this guide). After swapping the first node, we immediately ran into two issues:

1. ICMP over VPN Failure: With the default UPPAK mode, ICMP traffic through the VPN stopped working entirely. This mirrors the issue discussed in this thread.
2. High Idle CPU: The appliance was idling at roughly 20% CPU with almost no load.

The KPPAK Attempt:

Because we had experienced stability issues with UPPAK on our previous hardware, we decided to switch the new 19200 to KPPAK mode.

• The Good: Switching to KPPAK immediately fixed the ICMP VPN issue and the CPU stabilized. We moved the node into production for testing.
• The Bad: The next morning, as user load increased, performance tanked. It turns out our 10/25/40/100G QSFP28 (Intel) NICs use the ICE driver, which is known to have major performance limitations when running in KPPAK on these appliances (as per sk183525).

Current Status:

We are essentially stuck

• In UPPAK: VPN traffic (ICMP) is broken.
• In KPPAK: The ICE drivers cause severe performance issues

We have two open TAC cases, and they are currently looking into debugs for the UPPAK ICMP issue, but we are effectively unable to use the new hardware.

Has anyone successfully resolved the ICMP/UPPAK issue on the 19000 series? Or, has anyone found a workaround for the ICE driver performance bottleneck in KPPAK mode?

Any insights or similar experiences would be greatly appreciated!

Best regards,
Kuba

[PhoneBoy]

What release/JHF are you on?

[Jakub_K]

It's R81.20 JHF 120

[Lesley]

Are you sure you really had high cpu on uppak mode or you might get confused because of https://support.checkpoint.com/results/sk/sk180299?

You really only can see the load via cpview 

[Jakub_K]

We were monitoring that node via Skyline and it was iding at 30% CPU usage with about 300Mbps traffic via VPN

[Chris_Atkinson]

For context is their anything unique about the ICMP traffic (is it standard ping - what was seen in debugs?) and how are your global properties configured relative to sk172546 / sk41093?

[Jakub_K]

This was just a simple ping, and we have a dedicated rule to allow icmp traffic

[Timothy_Hall]

The high idle CPU is expected with UPPAK mode.

Need code/Jumbo HFA version.

We will need to see the results of a fw ctl zdebug + drop being run while the ping traffic is attempted in the RA VPN.  The drop reason should provide a clue.

When I've encountered ping issues in a VPN with UPPAK, it seems to be one of these things:

1) The IPSec/ESP traffic is dropped and not recognized as part of the tunnel

2) UPPAK attempts to handle the ICMP outside of the slowpath, which it shouldn't do, vpn accel off (VPN Peer IP) for the remote peer IP might be worth a try to ensure it stays slowpath where it should be.

3) Been running into this more often: sk184455: Traffic is randomly dropped due to loop prevention

[Jakub_K]

It's R81.20 JHF 120

Currently we can't check "fw ctl zdebug + drop" as it's all in production and we have old 16200 apliance actvie. We would have to create a maintenance window to test that.

As for those three possibilities:
1. We could see that traffic being accepted and logged on smart console
2. For test we did turn off vpn accel but it didn't solve this issue
3. This one we would have to testduring that maintenance window