topic Re: VSX UPPAK issues in General Topics

VSX UPPAK issues

sebfuuu — Fri, 26 Sep 2025 07:57:12 GMT

Hello everyone,

I noticed a few interesting issues with VSX and UPPAK.

Appliance 9300
R81.20 JHF 111

* First issue:

When traffic goes through two different VS in the same VSX hardware via a Virtual Switch we see 1-2% "packet-loss"

With zdebug we see this output that matches the traffic:

@;628145396.20082;[uspace];[tid_1];[SIM4];prepare_cut_through:do_routing returned invalid out_ifn 65535, conn:<x.x.x.x,53,y.y.y.y,46771,17>;

@;628145396.20083;[uspace];[tid_1];[SIM4];sim_pkt_send_drop_notification:(5,0) received drop, reason: Interface Down (8), conn:<y.y.y.y,46771,x.x.x.x,53,17>;

@;628145396.20084;[uspace];[tid_1];[SIM4];sim_pkt_send_drop_notification:no track is needed for this drop - not sending a notificaion, conn:<y.y.y.y,46771,x.x.x.x,53,17>;

It is always dropped on the first VS, it does not matter what traffic and which VS.

(More than two VS see this behavior)

After change of the SecureXL mode from UPPAK to KPPAK this issues goes away.

TAC case was raised but the customer was not willing to share the data needed to proceed with the case at that time.

* Second issue:

After SecureXL mode change to KPPAK we get RX errors on a lot on 10g fiber interfaces.

We have confirmed this on four 9300 hardware (two diffrent clusters).

If you switch to UPPAK, the errors goes away.

If you go back to KPPAK you get the same amount of errors on the same interfaces.

Reboot of FW/Switch or disconnect/reconnect of SFP and cable does not have any impact.

No TAC case raised yet.

I have not seen these issues on larger 9xxx appliances with VSX.

So one guess would be the Intel E-cores / P-cores architecture. (Like sk183438)

USFW is used.

The questions I have for the community.

Have anyone seen any issues like this?

Thanks again to everyone for this great community.

/Seb

Re: VSX UPPAK issues

Henrik_Noerr1 — Fri, 26 Sep 2025 08:55:33 GMT

Your experiences mirrors ours.

Furthermore we have a 9400 vsx cluster waiting to be onboarded with production traffic still with no VSs built.

One node becomes unresponsive every 2-4 days and needs to be power cycled from LOM. We made RMA of both nodes. Same result. The node crashing in the cluster varies. jumbo t113

We are investigating options to replace the model to 9700+

/Henrik

Re: VSX UPPAK issues

Lesley — Sat, 27 Sep 2025 19:10:40 GMT

RX buffers could be related with: https://support.checkpoint.com/results/sk/sk182825

does not 100% match

Re: VSX UPPAK issues

the_rock — Sun, 28 Sep 2025 12:01:00 GMT

You could install jumbo 113, which is recommended and see if it helps. If not, I would certainly open TAC case for this.

Andy

Re: VSX UPPAK issues

Timothy_Hall — Sun, 28 Sep 2025 13:37:22 GMT

For the first issue, are you seeing TX errors in the output of netstat -ni? I practically never saw TX errors with KPPAK, but with UPPAK they seem much more likely now. If you run ifconfig -a are you seeing any carrier transitions on any interfaces? That "interface down" message is weird.

For the second issue what specific type of RX errors are you seeing (OVR,ERR,DRP)? Can you please post the netstat -ni output including the interface seeing the RX errors along with ethtool -S (interface) for the affected 10Gbit interface? If the RX problems are RX-DRP, it is possible it is "junk" traffic for invalid VLAN tags and/or invalid EtherType protocols. UPPAK might not increment the RX-DRP counter for this junk traffic like KPPAK does. See here:

sk166424: Number of RX packet drops on interfaces increases on a Security Gateway R80.30 and higher with Gaia kernel 3.10

sk183847: RX-ERR and RX-DRP Counters on Bonded Ports

Re: VSX UPPAK issues

Timothy_Hall — Sun, 28 Sep 2025 13:42:30 GMT

Your crashes are almost certainly related to P-Cores/E-Cores on the 9300/9400. Not sure what the heck Intel was thinking implementing this in a server-based architecture, as opposed to just mobile architectures where battery life is important. However, one could argue that the same criticism applies to many of Intel's recent leadership decisions, which is why the company is currently in such big trouble.

sk183438: Stability issue in Check Point appliances 9300 and 9400

Re: VSX UPPAK issues

sebfuuu — Tue, 30 Sep 2025 07:57:23 GMT

For the first issue, no TX errors or carrier transitions on any interfaces.

As for the second issue, see the attached outputs.
RX-ERR, rx_length_errors.nic

Re: VSX UPPAK issues

sebfuuu — Tue, 30 Sep 2025 08:17:42 GMT

Found this SK: sk183040

" This not a Check Point issue. The issue is cosmetic and does not have an effect on network traffic flows. The rx_length_errors statistic is not meaningful and can be safely ignored.

According to Linux kernel commit f9f9de23dc88, the statistic was removed from the ice driver in the upstream Linux kernel due to its lack of relevance.
"

I have checked and "ice" driver is used on the interface

Re: VSX UPPAK issues

the_rock — Thu, 02 Oct 2025 00:00:05 GMT

Thanks for letting us know!

Andy