Facing tremendous packet loss on R82.10 on 3920 Appliances

[Blason_R]

Hello,

It appears that the 3900-based appliance series does not support R82 and earlier versions. It seems to only support R82.10. The problem I am encountering is that I have just configured a new 3920 cluster on R82.10, which is managed by R82 management, along with a specific hotfix that supports the R82.10 gateways. Currently, I am experiencing significant packet loss for traffic traversing through the firewall. Even when I attempt to generate packets from the firewall, I observe the same issue. I have debugged numerous aspects, but to no avail, and I eventually reached out to TAC; however, as is often the case, the response has been slow, and this situation is becoming increasingly critical.

Do you have any insights regarding R82.10? I have tried disabling fwaccel, but there has been no success. Is it possible to disable the user space mode firewall? Given that it is R82.10, I am uncertain if many installations have been completed and whether the community can provide assistance.

[Chris_Atkinson]

UPPAK cannot be disabled in R82.10 and higher.

How is the machine connected, which JHF is installed?

[Blason_R]

JHF T22

```

This is Check Point CPinfo Build 914000219 for GAIA
[CPshared]
HOTFIX_R82_10_JUMBO_HF_T271_MAIN Take: 22

```

[the_rock]

What does basic zdebug show?

[Blason_R]

zdebug does not show any drops

[simonemantovani]

How is the load of the firewall? did you check the counters of the interfaces, if there are any errors increasing?

Do you have installed latest JHF? For 3920 and R82.10 the latest JHF is 467. https://support.checkpoint.com/results/sk/sk183199

[Blason_R]

How do I identify the take and yes I went ahead with whatever has been loaded on device

[Chris_Atkinson]

Which OS build is reported with "show version all" ?

[Duane_Toler]

You can run the command "take.info" to get the build name and version.  Yours will say "jess_main;271".  As others said, there is a new R82.10 build and JHF available ("jess_opt_main;467") and JHF 6.  On your current build, "cpinfo" is also broken due to a incorrect libcpopenssl version that's missing.

You will need to "upgrade" to the new build in CLISH (or Gaia Portal). 

installer download  Blink_image_1.1_Check_Point_R82.10_T467_JHF_T6_aarch64_SecurityGateway.tgz

installer verify  Blink_image_1.1_Check_Point_R82.10_T467_JHF_T6_aarch64_SecurityGateway.tgz

installer upgrade  Blink_image_1.1_Check_Point_R82.10_T467_JHF_T6_aarch64_SecurityGateway.tgz

This is the Blink installer, so it'll do all the work into a new disk volume without any downtime.  The only downtime is to do the reboot, then you'll need to install policy in SmartConsole.

If you do SNMP monitoring, also note that this new R82.10 build has changed several SNMP MIB names (I have a TAC case open about this, but they insist it was "by design").  The OIDs are the same, but the translated names changed; if your SNMP manager is loading the MIB and polling by the original name, like for svnMem64, be warned.

[Chris_Atkinson]

It sounds as though you are on the initial 3900 build and haven't yet completed the upgrade path outlined in sk183199?

Recommended doing this to help move the troubleshooting / case forward.

[Blason_R]

This remains a problem. We have upgraded to the recommended path, yet I do not believe the issue has been resolved.

[PhoneBoy]

Super Seven commands output will probably be helpful here: https://community.checkpoint.com/t5/Scripts/S7PAC-Super-Seven-Performance-Assessment-Commands/m-p/40... 

[Timothy_Hall]

Make sure you have the correct code on your 3900, including its special Jumbo HFAs, which were separate from the maintrain at one point.  Your issue sounds like the resolved mbuf problem, which would cause major packet loss. Check logfile /var/log/usim_x86.elg for mbuf allocation failures.

sk183771: Significant packet drops on interfaces configured with jumbo frames on 9800 appliances

sk184419: UPPAK memory buffer leak may occur when MDPS enabled and IPv6 disabled

[Blason_R]

These are the logs I am seeing in usim_x86.elg

mbuf_pool_socket_0 size:87040 cache_size:256 (1 attempts) data size: 2176 pool_0:1004eddc0
mbuf_jumbo_pool_socket_0 size:6400 jumbo cache_size:256 (1 attempts) data size: 9344 pool_0:100428c00
mbuf_pool_socket_0 size:157696 cache_size:256 (1 attempts) data size: 2176 pool_0:1004eddc0
mbuf_jumbo_pool_socket_0 size:12800 jumbo cache_size:256 (1 attempts) data size: 9344 pool_0:10083e400
Apr 29 09:45:20.534501 [uspace];[tid_0];[UPPAK];fwk_snd_fwk_event:fwk_snd_init_fwk_mbuf_caches SUCCEEDED for vsid 0
mbuf_pool_socket_0 size:157696 cache_size:256 (1 attempts) data size: 2176 pool_0:1004ed800
mbuf_jumbo_pool_socket_0 size:12800 jumbo cache_size:256 (1 attempts) data size: 9344 pool_0:10083e400
Apr 29 10:04:01.384594 [uspace];[tid_0];[UPPAK];fwk_snd_fwk_event:fwk_snd_init_fwk_mbuf_caches SUCCEEDED for vsid 0
mbuf_pool_socket_0 size:157696 cache_size:256 (1 attempts) data size: 2176 pool_0:1004ed800
mbuf_jumbo_pool_socket_0 size:12800 jumbo cache_size:256 (1 attempts) data size: 9344 pool_0:10083e400
Apr 29 10:30:38.390917 [uspace];[tid_0];[UPPAK];fwk_snd_fwk_event:fwk_snd_init_fwk_mbuf_caches SUCCEEDED for vsid 0
mbuf_pool_socket_0 size:157696 cache_size:256 (1 attempts) data size: 2176 pool_0:1004ed800
mbuf_jumbo_pool_socket_0 size:12800 jumbo cache_size:256 (1 attempts) data size: 9344 pool_0:10083e400
May 05 22:18:10.136475 [uspace];[tid_0];[UPPAK];fwk_snd_fwk_event:fwk_snd_init_fwk_mbuf_caches SUCCEEDED for vsid 0
mbuf_pool_socket_0 size:157696 cache_size:256 (1 attempts) data size: 2176 pool_0:1004ed800
mbuf_jumbo_pool_socket_0 size:12800 jumbo cache_size:256 (1 attempts) data size: 9344 pool_0:10083e400
May 13 20:49:32.556365 [uspace];[tid_0];[UPPAK];fwk_snd_fwk_event:fwk_snd_init_fwk_mbuf_caches SUCCEEDED for vsid 0
mbuf_pool_socket_0 size:157696 cache_size:256 (1 attempts) data size: 2176 pool_0:1004ed800
mbuf_jumbo_pool_socket_0 size:12800 jumbo cache_size:256 (1 attempts) data size: 9344 pool_0:10083e400
May 14 11:15:01.965327 [uspace];[tid_0];[UPPAK];fwk_snd_fwk_event:fwk_snd_init_fwk_mbuf_caches SUCCEEDED for vsid 0
May 21 12:17:42.365750 [uspace];[tid_0];mbuf_pool_socket_0 size:157696 cache_size:256 (1 attempts) data size: 2176 pool_0:1004ed800
May 21 12:17:42.372822 [uspace];[tid_0];mbuf_jumbo_pool_socket_0 size:12800 jumbo cache_size:256 (1 attempts) data size: 9344 pool_0:10083e400
May 21 11:18:32.062123 [uspace];[tid_0];[UPPAK];fwk_snd_fwk_event:fwk_snd_init_fwk_mbuf_caches SUCCEEDED for vsid 0
May 21 12:25:51.858972 [uspace];[tid_0];mbuf_pool_socket_0 size:157696 cache_size:256 (1 attempts) data size: 2176 pool_0:1004ed800
May 21 12:25:51.866067 [uspace];[tid_0];mbuf_jumbo_pool_socket_0 size:12800 jumbo cache_size:256 (1 attempts) data size: 9344 pool_0:10083e400
May 21 12:26:40.941838 [uspace];[tid_0];[UPPAK];fwk_snd_fwk_event:fwk_snd_init_fwk_mbuf_caches SUCCEEDED for vsid 0

Dont think any mbuf issue