Let's discuss: Important Notification to Customers with SecureXL User Mode (UPPAK) enabled, on R81.2

[Lesley]

We have received the following message today, I assume more people did. Let's discuss this topic. 

Anyone experienced this issue yet? I have some setups running this take but have not seen this issue. 

I see SK already has been updated with more information so that is good. I assume that the fix will be included in upcoming takes? And with a crash, gateway does full reboot? Or some processes are restarting and there is a ''hickup''? 

We would like to bring the following to your immediate attention. We have identified that configuration of your environment with SecureXL User Mode (UPPAK) enabled on your Security Gateway may have compatibility issues with Take 96 / Take 98.

Action required :
Please follow sk183181 for remediation steps.

Symptoms

• Security Gateway with the R81.20 Jumbo Hotfix Accumulator Takes 96 and Take 98 may crash frequently when SecureXL works in the User Mode (UPPAK).

Cause

Race condition may occur in the ADP acceleration driver when updating network routes for SecureXL in the User Mode (UPPAK):

• In Take 96, the process "usim_x86" crashes with a core dump file in the /var/log/dump/usermode/ directory.
• In Take 98, the Security Gateway crashes with a VMcore dump file in the /var/log/crash/<DATE>/vmcore/ directory.

Affected versions:

• R81.20 Jumbo Hotfix Accumulator Take 96
• R81.20 Jumbo Hotfix Accumulator Take 98

How to check if your Security Gateway may be affected?

1. Connect to the command line on the Security Gateway.
2. Check the current R81.20 Jumbo Hotfix Accumulator Take:
   cpinfo -y all
3. Check the current SecureXL mode:
   1. Run:
      fwaccel stat
   2. Examine the column "Name":
      • "KPPAK" means Kernel Mode - this SK article does not apply.
      • "UPPAK" means User Mode - this SK article applies.

Solution

Contact Check Point Support to get a Hotfix for this issue.

[the_rock]

Thanks for that Lesley. Lets hope fix is rolled into next jumbo release.

Andy

[Thomas_Eichelbu]

Hello Team, 

yes i just saw it ...
the question i have ...

if you run Quantum Force or QLS gateways with Light Speed NICs. What will happen to those Light Speed functionalities and most important the speed you gain by Light Speed cards if you disable UPPAK for SXL?
Is the performance boost of Light Speed also available if SXL runs in KPPAK or just at UPPAK?
when you check the limitations on UPPAK you will be surprised how much is not supported.
https://sc1.checkpoint.com/documents/Appliances/100G_Ports_AdminGuide/Content/Topics-100G-Card-AG/Kn...

For Maestro UPPAK is supported since HFA 89.
But some TAC say they dont recommend UPPAK for Maestro ... the reasons are not yet known to me ... someone has any clues?

[Thomas_Eichelbu]

aha, people who can read have an advantage .. it is written:



so only UPPAK brings the promoted performance advertised by the datasheet, without UPPAK its not the case ... 

[Timothy_Hall]

Interesting that this notification came out on the same day I delivered my "Be Your Own Tac Part Deux" presentation at CPX Vegas which extensively discusses the limitations imposed by UPPAK.   I think @PhoneBoy  will be posting my slide deck today; the list of UPPAK limitations I compiled is quite extensive.  F2F/slowpath performance is definitely lower than KPPAK, however I believe Medium Path and Fastpath performance is better but I can't quantify the amount of difference as I do not have access to Quantum Force or Lightspeed hardware which is required to work with UPPAK.

My current understanding is that UPPAK does improve performance mainly by using poll mode instead of being interrupt-driven for packet handling on the SND cores.  The UPPAK framework will also permit hardware-based acceleration in the Mellanox/Nvidia SmartNICs but I do not believe that part is actually in use yet.

[PhoneBoy]

https://community.checkpoint.com/t5/Member-Exclusive-Content/Be-your-own-TAC-Part-Deux-Advanced-Gate... 

[the_rock]

Excellent! Just had a quick look and I find table on page 51 super useful.

Andy

[OxO]

Hi,

I think I had this problem...

A few weeks ago I had to exchange about nine clusters and with/to R81.20 JHF96

Appliance types that were replaced:

6x00 -> 9x00
6x00 -> 6900
3x00 -> 6x00

Affected rate:

9000 clusters: affected: 100% 
6000 clusters: affected: 0%

...and always/only at the same cluster change/implementation step:

Cluster Node1: Replaced and Running as HA Status: Active (Policy: Access + TP)
Cluster Node2: Replaced and Running as HA Status: Standby (Policy: Access)

so we can say: the cluster is "full" replaced and layer2/3 connected (the accessibility of router gateways and arp tables were checked)... what is still missing is the last policy install, so that the second node also gets the TP policy...

...and only with this "last" policy install I got the core dumps (vmcore_zero64.gz)

impact (varies):

9000-cluster#1: both new-members died at the same time!
9000-cluster#2: "only" the new-active node died (which has already been running as active for 2 hours (differen cluster-node locations))

After the dump(s), both cluster nodes were rebooted one after the other and then the last policy install worked... since then, policy installs (regardless of whether: full / tp only / access only) worked without new dumps.

But now jhf98 is installed everywhere...

I therefore had several core dumps and CP (Case# 6-0004180684) said:

PRJ-59644, PRHF-38332
----------------------
jhf96: ontop fix -> accel_HOTFIX_R81_20_JHF_T96_828_MAIN_GA_FULL.tar
jhf98: informed me that the fix would be integrated and it was released immediately

that's why I'm surprised that jhf98 is listed here again....

I reopened my case and wait for an answer whether it is an typo in the SK, or there are further/other problems in this direction.

Best
Michael

[Alex-]

I seem to have encountered that issue on new Force appliances and the IKE crash as well. Fun times.

When we look at Take 98, it was released and set to recommended in a very short timeframe, for what happens to be a serious issue. I don't see how much QA and feedback from production can be obtained in just a few days.

Probably a though call to make by the release group who must be under quite some pressure to articulate all of this. Anyway, we're stopping new deployments of Take 98 and stay on Take 92 which seems stable for now.

[the_rock]

I had few customers also asked me about it, since it showed jumbo 98 came out Feb 12th and it was recommended take already on Feb 19th, just 1 week later. Seems way to fast to do so, in my humble opinion.

Andy

[Lesley]

Take 96 was already affected by this bug. It is true that 98 was put to recommended quickly but it only contained one fix. So it makes sense to make it more quickly recommended then when it was a big patch. 

Release notes take 98:

PRJ-59644, PRHF-38332

Security Gateway

Security Gateway may crash when route lookups encounter an unresolved next hop. See the Important Notes section.

[OxO]

according cp-suppport to my case and sk183181 - sk183181 might be a different problem than JH98 (PRJ-59644, PRHF-38332) -> so JHF98 + ONTOP is needed (at the moment)

PRJ-59644, PRHF-38332 -->  Security Gateway may crash when route lookups encounter an unresolved next hop.
???? (sk183181) -->  Race condition may occur in the ADP acceleration driver when updating network routes for SecureXL in the User Mode (UPPAK):

currently I am waiting for the JH98 - ONTOP and the info bugid inforamation for this problem

[CiscoMate]

We've had "fun" with this issue too on R81.20 HF 98, which is still listed as "recommended" 🤐. 

At one instant, "only" the primary cluster node crashed and the secondary took over. OK, not nice but **bleep** happens...

But at the other instant - and this is actually really crazy!! - both cluster nodes crashed simultaneously so that we had been completely down. Once we realized what was going on, we ran to the FW rack and pulled the power cables.

Please, I don't ever want to do this again. 😲

[the_rock]

Sorry you had to go through that...not fun 😞

I really hope this is fully fixed in next jumbo take.

Andy

[Alex-]

OK so we had two times the case where the 9x00 crashed and wouldn't reboot as usual, being reachable upon restart.

The solution was to go in LOM or console, run 'fw unloadlocal' and then install the policy for reactivating the cluster, after which we could install the hotfix.

This makes us a bit nervous about applying this level to the KPPAK appliances we have around. Some are not readily reachable, or the customer didn't bother to activate the LOM but with R81.10 nearing EoS, we need to move forward.

In short, are we confident that KPPAK running Take 98 won't have that issue? R81.10 clusters would upgrade with Blink so immediately to that version. Otherwise, we need to go through the transition of R81.20 base and then Take 92.

[Lesley]

Have you stared a ticket with CP for this crash? I had 1 customer with a crash, created a ticket for this. Have not got a confirm yet if we hit the bug. The gateway just rebooted came back came with some crash files and that is it. I did not had to use LOM or stuff like that. 

[Alex-]

Yes, cores provided and received the HF.

First time was a regular crash/reboot, subsequent were crash/reboot/no connectivity.