GAIA Upgrade from R77.30 to R80.40 - Firewall Drop...

whoa · ‎2023-04-03

I was able to upgrade the SMS to R81.10 via the offline method successfully. I recently upgraded the firewall from R77.30 to R80.40 with the upgraded DA and the HFA (Take 192). I have tried different ports on the switch, different SFP's and cables. The firewall continues to drop packets badly and internet access is severely degraded.

This just seems like a software issue because the Firewall has never dropped packets before, nor has the switch ever had any issue of this type before. On the switch side I do not see any kind of packet issue (loss, crc errors, collision...etc all 0).

I have un-installed the HFA and installed the one porior to it as well as re-installed the current HFA. No matter what I do the firewall drops packets.

Any Insight would be greatly appreciated.

PhoneBoy · ‎2023-04-03

Please provide Super Seven output so we can see where the issue might lie.
https://community.checkpoint.com/t5/Scripts/S7PAC-Super-Seven-Performance-Assessment-Commands/td-p/4...

whoa · ‎2023-04-04

So while waiting I went ahead and upgraded the Firewall to R81.10 (with current DA and HFA) which did seem to resolve the dropped packets issue, but the issue now seems to be streaming. If users require streams it seems that the Firewall only capably now supports about a dozen streams before streams are either glitchy or flat out lags out. Inevitably any stream has to be refreshed in order to continue running correctly.

In reading the powerpoint slide deck on the Super 7 script it mentions disabling fwaccel. After disabling that streams got better but after multiple streams older streams will lag out and stop all together. Further if file downloads are in the mix that also will kill streams and multiple downloads will cause older downloads to stop.

So when I try to run the script I get this error:
[Expert@fw:0]# curl_cli -o /usr/bin/s7pac tools.checkpoint.engineer/s7pac && chmod +x /usr/bin/s7pac
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 162 100 162 0 0 105 0 0:00:01 0:00:01 --:--:-- 127
[Expert@fw:0]# /usr/bin/s7pac
/usr/bin/s7pac: line 1: html: No such file or directory
/usr/bin/s7pac: line 2: syntax error near unexpected token `<'
'usr/bin/s7pac: line 2: `<head><title>301 Moved Permanently</title></head>
[Expert@fw:0]#

With that said I ran all of the commands individually:

Accept Templates : enabled
Drop Templates : disabled
NAT Templates : enabled
LightSpeed Accel : disabled
[Expert@fw:0]# fwaccel stats -s
Accelerated conns/Total conns : 25/25 (100%)
LightSpeed conns/Total conns : 0/25 (0%)
Accelerated pkts/Total pkts : 1244374/11513639 (10%)
LightSpeed pkts/Total pkts : 0/11513639 (0%)
F2Fed pkts/Total pkts : 10269265/11513639 (89%)
F2V pkts/Total pkts : 5888/11513639 (0%)
CPASXL pkts/Total pkts : 0/11513639 (0%)
PSLXL pkts/Total pkts : 0/11513639 (0%)
CPAS pipeline pkts/Total pkts : 0/11513639 (0%)
PSL pipeline pkts/Total pkts : 0/11513639 (0%)
CPAS inline pkts/Total pkts : 0/11513639 (0%)
PSL inline pkts/Total pkts : 0/11513639 (0%)
QOS inbound pkts/Total pkts : 0/11513639 (0%)
QOS outbound pkts/Total pkts : 0/11513639 (0%)
Corrected pkts/Total pkts : 0/11513639 (0%)
[Expert@fw:0]# grep -c ^processor /proc/cpuinfo
8
[Expert@fw:0]# /sbin/cpuinfo
HyperThreading=disabled
[Expert@fw:0]# fw ctl affinity -l -r
CPU 0: eth0 eth1 eth3
CPU 1:
CPU 2: fw_5 (active)
cprid lpd mpdaemon fwd wsdnsd in.asessiond in.msd core_uploader vpnd iked cprid cpd
CPU 3: fw_4 (active)
cprid lpd mpdaemon fwd wsdnsd in.asessiond in.msd core_uploader vpnd iked cprid cpd
CPU 4: fw_3 (active)
cprid lpd mpdaemon fwd wsdnsd in.asessiond in.msd core_uploader vpnd iked cprid cpd
CPU 5: fw_2 (active)
cprid lpd mpdaemon fwd wsdnsd in.asessiond in.msd core_uploader vpnd iked cprid cpd
CPU 6: fw_1 (active)
cprid lpd mpdaemon fwd wsdnsd in.asessiond in.msd core_uploader vpnd iked cprid cpd
CPU 7: fw_0 (active)
cprid lpd mpdaemon fwd wsdnsd in.asessiond in.msd core_uploader vpnd iked cprid cpd
All:
[Expert@fw:0]# fw ctl multik stat
ID | Active | CPU | Connections | Peak
----------------------------------------------
0 | Yes | 7 | 37 | 72
1 | Yes | 6 | 52 | 95
2 | Yes | 5 | 55 | 84
3 | Yes | 4 | 71 | 106
4 | Yes | 3 | 77 | 110
5 | Yes | 2 | 67 | 104
[Expert@fw:0]# cpstat os -f multi_cpu -o 1

Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1| 0| 6| 94| 6| ?| 17775|
| 2| 0| 0| 100| 0| ?| 17775|
| 3| 1| 2| 97| 3| ?| 17775|
| 4| 1| 1| 98| 2| ?| 17775|
| 5| 0| 2| 98| 2| ?| 17775|
| 6| 1| 5| 95| 5| ?| 17775|
| 7| 0| 3| 97| 3| ?| 17775|
| 8| 1| 5| 95| 5| ?| 17775|
---------------------------------------------------------------------------------

Chris_Atkinson · ‎2023-04-04

So far this stands out:

F2Fed pkts/Total pkts : 10269265/11513639 (89%)

What blades are enabled on this gateway?

CCSM R77/R80/ELITE

the_rock · ‎2023-04-03

Hey mate,

We definitely need more details. What sort of connections are droppe3d exactly? Specific services? Please provide below outputs:

top

free -m

ps -auxw

cpwd_admin list

cpview (in cpview, tab between different fields and check connection tabs as well)

fw ctl multik print_heavy_conn

Hope those help.

Andy

whoa · ‎2023-04-04

Its alot of output but here it is:

[Expert@fw:0]# top
top - 11:41:46 up 2:08, 1 user, load average: 0.43, 0.29, 0.23
Tasks: 245 total, 2 running, 243 sleeping, 0 stopped, 0 zombie
%Cpu0 : 0.0 us, 0.0 sy, 0.0 ni, 94.1 id, 0.0 wa, 1.0 hi, 5.0 si, 0.0 st
%Cpu1 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu2 : 0.0 us, 6.8 sy, 0.0 ni, 92.2 id, 0.0 wa, 1.0 hi, 0.0 si, 0.0 st
%Cpu3 : 0.0 us, 2.0 sy, 0.0 ni, 98.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu4 : 0.0 us, 0.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 1.0 si, 0.0 st
%Cpu5 : 1.0 us, 1.0 sy, 0.0 ni, 98.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu6 : 0.0 us, 6.6 sy, 0.0 ni, 92.5 id, 0.0 wa, 0.9 hi, 0.0 si, 0.0 st
%Cpu7 : 0.0 us, 2.0 sy, 0.0 ni, 98.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 7902260 total, 3841924 free, 1902872 used, 2157464 buff/cache
KiB Swap: 16378260 total, 16378260 free, 0 used. 5383760 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ P COMMAND
6633 admin 20 0 0 0 0 S 4.0 0.0 0:47.15 3 fw_worker_4
6629 admin 20 0 0 0 0 S 3.0 0.0 0:53.68 7 fw_worker_0
32120 admin 20 0 0 0 0 S 3.0 0.0 0:22.93 2 kworker/2:3
6630 admin 20 0 0 0 0 S 2.0 0.0 0:43.71 6 fw_worker_1
6634 admin 20 0 0 0 0 S 2.0 0.0 0:48.77 2 fw_worker_5
6631 admin 20 0 0 0 0 R 1.0 0.0 1:14.42 5 fw_worker_2
8932 admin 20 0 267072 54108 25640 S 1.0 0.7 0:03.40 4 iked
1 admin 20 0 2628 716 612 S 0.0 0.0 0:00.50 0 init
2 admin 20 0 0 0 0 S 0.0 0.0 0:00.00 2 kthreadd
3 admin 20 0 0 0 0 S 0.0 0.0 0:00.02 0 ksoftirqd/0
5 admin 0 -20 0 0 0 S 0.0 0.0 0:00.00 0 kworker/0:0H
7 admin rt 0 0 0 0 S 0.0 0.0 0:00.00 0 migration/0
8 admin 20 0 0 0 0 S 0.0 0.0 0:00.00 0 rcu_bh
9 admin 20 0 0 0 0 S 0.0 0.0 0:02.07 2 rcu_sched
10 admin 20 0 0 0 0 S 0.0 0.0 0:00.00 0 rcuob/0
11 admin 20 0 0 0 0 S 0.0 0.0 0:00.13 5 rcuos/0
12 admin 0 -20 0 0 0 S 0.0 0.0 0:00.00 0 lru-add-drain
13 admin rt 0 0 0 0 S 0.0 0.0 0:00.01 0 watchdog/0
14 admin rt 0 0 0 0 S 0.0 0.0 0:00.00 1 watchdog/1
15 admin rt 0 0 0 0 S 0.0 0.0 0:00.00 1 migration/1
16 admin 20 0 0 0 0 S 0.0 0.0 0:00.00 1 ksoftirqd/1
17 admin 20 0 0 0 0 S 0.0 0.0 0:00.40 1 kworker/1:0
18 admin 0 -20 0 0 0 S 0.0 0.0 0:00.00 1 kworker/1:0H
19 admin 20 0 0 0 0 S 0.0 0.0 0:00.00 0 rcuob/1
20 admin 20 0 0 0 0 S 0.0 0.0 0:00.13 0 rcuos/1
21 admin rt 0 0 0 0 S 0.0 0.0 0:00.12 2 watchdog/2

[Expert@fw:0]# free -m
total used free shared buff/cache available
Mem: 7717 1858 3751 7 2106 5256
Swap: 15994 0 15994

[Expert@fw:0]# ps -auxw
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
admin 1 0.0 0.0 2628 716 ? Ss 09:33 0:00 init [3]
admin 2 0.0 0.0 0 0 ? S 09:33 0:00 [kthreadd]
admin 3 0.0 0.0 0 0 ? S 09:33 0:00 [ksoftirqd/0]
admin 5 0.0 0.0 0 0 ? S< 09:33 0:00 [kworker/0:0H]
admin 7 0.0 0.0 0 0 ? S 09:33 0:00 [migration/0]
admin 8 0.0 0.0 0 0 ? S 09:33 0:00 [rcu_bh]
admin 9 0.0 0.0 0 0 ? S 09:33 0:02 [rcu_sched]
admin 10 0.0 0.0 0 0 ? S 09:33 0:00 [rcuob/0]
admin 11 0.0 0.0 0 0 ? S 09:33 0:00 [rcuos/0]
admin 12 0.0 0.0 0 0 ? S< 09:33 0:00 [lru-add-drain]
admin 13 0.0 0.0 0 0 ? S 09:33 0:00 [watchdog/0]
admin 14 0.0 0.0 0 0 ? S 09:33 0:00 [watchdog/1]
admin 15 0.0 0.0 0 0 ? S 09:33 0:00 [migration/1]
admin 16 0.0 0.0 0 0 ? S 09:33 0:00 [ksoftirqd/1]
admin 17 0.0 0.0 0 0 ? S 09:33 0:00 [kworker/1:0]
admin 18 0.0 0.0 0 0 ? S< 09:33 0:00 [kworker/1:0H]
admin 19 0.0 0.0 0 0 ? S 09:33 0:00 [rcuob/1]
admin 20 0.0 0.0 0 0 ? S 09:33 0:00 [rcuos/1]
admin 21 0.0 0.0 0 0 ? S 09:33 0:00 [watchdog/2]
admin 22 0.0 0.0 0 0 ? S 09:33 0:00 [migration/2]
admin 23 0.0 0.0 0 0 ? S 09:33 0:00 [ksoftirqd/2]
admin 25 0.0 0.0 0 0 ? S< 09:33 0:00 [kworker/2:0H]
admin 26 0.0 0.0 0 0 ? S 09:33 0:00 [rcuob/2]
admin 27 0.0 0.0 0 0 ? S 09:33 0:01 [rcuos/2]
admin 28 0.0 0.0 0 0 ? S 09:33 0:00 [watchdog/3]
admin 29 0.0 0.0 0 0 ? S 09:33 0:00 [migration/3]
admin 30 0.0 0.0 0 0 ? S 09:33 0:00 [ksoftirqd/3]
admin 32 0.0 0.0 0 0 ? S< 09:33 0:00 [kworker/3:0H]
admin 33 0.0 0.0 0 0 ? S 09:33 0:00 [rcuob/3]
admin 34 0.0 0.0 0 0 ? S 09:33 0:01 [rcuos/3]
admin 35 0.0 0.0 0 0 ? S 09:33 0:00 [watchdog/4]
admin 36 0.0 0.0 0 0 ? S 09:33 0:00 [migration/4]
admin 37 0.0 0.0 0 0 ? S 09:33 0:00 [ksoftirqd/4]
admin 39 0.0 0.0 0 0 ? S< 09:33 0:00 [kworker/4:0H]
admin 40 0.0 0.0 0 0 ? S 09:33 0:00 [rcuob/4]
admin 41 0.0 0.0 0 0 ? S 09:33 0:00 [rcuos/4]
admin 42 0.0 0.0 0 0 ? S 09:33 0:00 [watchdog/5]
admin 43 0.0 0.0 0 0 ? S 09:33 0:00 [migration/5]
admin 44 0.0 0.0 0 0 ? S 09:33 0:00 [ksoftirqd/5]
admin 46 0.0 0.0 0 0 ? S< 09:33 0:00 [kworker/5:0H]
admin 47 0.0 0.0 0 0 ? S 09:33 0:00 [rcuob/5]
admin 48 0.0 0.0 0 0 ? S 09:33 0:00 [rcuos/5]
admin 49 0.0 0.0 0 0 ? S 09:33 0:00 [watchdog/6]
admin 50 0.0 0.0 0 0 ? S 09:33 0:00 [migration/6]
admin 51 0.0 0.0 0 0 ? S 09:33 0:00 [ksoftirqd/6]
admin 53 0.0 0.0 0 0 ? S< 09:33 0:00 [kworker/6:0H]
admin 54 0.0 0.0 0 0 ? S 09:33 0:00 [rcuob/6]
admin 55 0.0 0.0 0 0 ? S 09:33 0:01 [rcuos/6]
admin 56 0.0 0.0 0 0 ? S 09:33 0:00 [watchdog/7]
admin 57 0.0 0.0 0 0 ? S 09:33 0:00 [migration/7]
admin 58 0.0 0.0 0 0 ? S 09:33 0:00 [ksoftirqd/7]
admin 60 0.0 0.0 0 0 ? S< 09:33 0:00 [kworker/7:0H]
admin 61 0.0 0.0 0 0 ? S 09:33 0:00 [rcuob/7]
admin 62 0.0 0.0 0 0 ? S 09:33 0:01 [rcuos/7]
admin 63 0.0 0.0 0 0 ? S 09:33 0:00 [kdevtmpfs]
admin 64 0.0 0.0 0 0 ? S< 09:33 0:00 [netns]
admin 65 0.0 0.0 0 0 ? S 09:33 0:00 [slab_mcd]
admin 66 0.0 0.0 0 0 ? S< 09:33 0:00 [writeback]
admin 67 0.0 0.0 0 0 ? S< 09:33 0:00 [kintegrityd]
admin 68 0.0 0.0 0 0 ? S< 09:33 0:00 [bioset]
admin 69 0.0 0.0 0 0 ? S< 09:33 0:00 [bioset]
admin 70 0.0 0.0 0 0 ? S< 09:33 0:00 [bioset]
admin 71 0.0 0.0 0 0 ? S< 09:33 0:00 [kblockd]
admin 72 0.0 0.0 0 0 ? S< 09:33 0:00 [md]
admin 73 0.0 0.0 0 0 ? S< 09:33 0:00 [edac-poller]
admin 74 0.0 0.0 0 0 ? S< 09:33 0:00 [watchdogd]
admin 77 0.0 0.0 0 0 ? S 09:33 0:01 [kworker/2:1]
admin 78 0.0 0.0 0 0 ? S 09:33 0:00 [kworker/3:1]
admin 80 0.0 0.0 0 0 ? S 09:33 0:00 [kworker/5:1]
admin 81 0.0 0.0 0 0 ? S 09:33 0:00 [kworker/6:1]
admin 83 0.0 0.0 0 0 ? S< 09:33 0:00 [rpciod]
admin 84 0.0 0.0 0 0 ? S< 09:33 0:00 [xprtiod]
admin 88 0.0 0.0 0 0 ? S 09:33 0:00 [kswapd0]
admin 89 0.0 0.0 0 0 ? SN 09:33 0:00 [ksmd]
admin 90 0.0 0.0 0 0 ? SN 09:33 0:00 [khugepaged]
admin 91 0.0 0.0 0 0 ? S< 09:33 0:00 [nfsiod]
admin 92 0.0 0.0 0 0 ? S< 09:33 0:00 [crypto]
admin 101 0.0 0.0 0 0 ? S< 09:33 0:00 [kthrotld]
admin 102 0.0 0.0 0 0 ? S< 09:33 0:00 [kmpath_rdacd]
admin 103 0.0 0.0 0 0 ? S< 09:33 0:00 [kaluad]
admin 117 0.0 0.0 0 0 ? S< 09:33 0:00 [deferwq]
admin 122 0.0 0.0 0 0 ? S 09:33 0:00 [kworker/4:2]
admin 126 0.0 0.0 0 0 ? S< 09:33 0:00 [bioset]
admin 127 0.0 0.0 0 0 ? S< 09:33 0:00 [xfsalloc]
admin 128 0.0 0.0 0 0 ? S< 09:33 0:00 [xfs_mru_cache]
admin 137 0.0 0.0 0 0 ? S< 09:33 0:00 [ata_sff]
admin 139 0.0 0.0 0 0 ? S 09:33 0:00 [scsi_eh_0]
admin 140 0.0 0.0 0 0 ? S< 09:33 0:00 [scsi_tmf_0]
admin 141 0.0 0.0 0 0 ? S 09:33 0:00 [scsi_eh_1]
admin 142 0.0 0.0 0 0 ? S< 09:33 0:00 [scsi_tmf_1]
admin 145 0.0 0.0 0 0 ? S 09:33 0:00 [scsi_eh_2]
admin 146 0.0 0.0 0 0 ? S< 09:33 0:00 [scsi_tmf_2]
admin 147 0.0 0.0 0 0 ? S 09:33 0:00 [scsi_eh_3]
admin 148 0.0 0.0 0 0 ? S< 09:33 0:00 [scsi_tmf_3]
admin 152 0.0 0.0 0 0 ? S 09:33 0:00 [cciss_scan]
admin 161 0.0 0.0 0 0 ? S< 09:33 0:00 [dm_bufio_cache]
admin 164 0.0 0.0 0 0 ? S< 09:33 0:00 [kworker/4:1H]
admin 165 0.0 0.0 0 0 ? S< 09:33 0:00 [kworker/0:1H]
admin 171 0.0 0.0 0 0 ? S< 09:33 0:00 [kworker/3:1H]
admin 172 0.0 0.0 0 0 ? S< 09:33 0:00 [kworker/1:1H]
admin 174 0.0 0.0 0 0 ? S< 09:33 0:00 [kworker/2:1H]
admin 175 0.0 0.0 0 0 ? S< 09:33 0:00 [kdmflush]
admin 176 0.0 0.0 0 0 ? S< 09:33 0:00 [bioset]
admin 177 0.0 0.0 0 0 ? S< 09:33 0:00 [kdmflush]
admin 178 0.0 0.0 0 0 ? S< 09:33 0:00 [bioset]
admin 180 0.0 0.0 0 0 ? S< 09:33 0:00 [kworker/6:1H]
admin 326 0.0 0.0 0 0 ? S< 09:34 0:00 [kworker/7:1H]
admin 328 0.0 0.0 0 0 ? S< 09:34 0:00 [xfs-buf/dm-0]
admin 329 0.0 0.0 0 0 ? S< 09:34 0:00 [xfs-data/dm-0]
admin 330 0.0 0.0 0 0 ? S< 09:34 0:00 [xfs-conv/dm-0]
admin 331 0.0 0.0 0 0 ? S< 09:34 0:00 [xfs-cil/dm-0]
admin 332 0.0 0.0 0 0 ? S< 09:34 0:00 [xfs-reclaim/dm-]
admin 333 0.0 0.0 0 0 ? S< 09:34 0:00 [xfs-log/dm-0]
admin 334 0.0 0.0 0 0 ? S< 09:34 0:00 [xfs-eofblocks/d]
admin 335 0.0 0.0 0 0 ? S 09:34 0:00 [xfsaild/dm-0]
admin 376 0.0 0.0 0 0 ? S< 09:34 0:00 [kworker/5:1H]
admin 398 0.0 0.0 2540 836 ? S<s 09:34 0:00 /sbin/udevd -d
admin 871 0.0 0.0 0 0 ? S 09:39 0:00 [kworker/0:13]
admin 1566 0.0 0.0 0 0 ? SN 09:34 0:00 [kipmi0]
admin 1801 0.0 0.0 0 0 ? S< 09:34 0:00 [kpsmoused]
admin 2066 0.0 0.0 0 0 ? S< 09:34 0:00 [ttm_swap]
admin 2394 0.0 0.0 0 0 ? S< 09:34 0:00 [kmpathd]
admin 2395 0.0 0.0 0 0 ? S< 09:34 0:00 [kmpath_handlerd]
admin 2430 0.0 0.0 0 0 ? S 09:34 0:00 [kjournald]
admin 2431 0.0 0.0 0 0 ? S< 09:34 0:00 [xfs-buf/dm-1]
admin 2432 0.0 0.0 0 0 ? S< 09:34 0:00 [xfs-data/dm-1]
admin 2433 0.0 0.0 0 0 ? S< 09:34 0:00 [xfs-conv/dm-1]
admin 2434 0.0 0.0 0 0 ? S< 09:34 0:00 [xfs-cil/dm-1]
admin 2435 0.0 0.0 0 0 ? S< 09:34 0:00 [xfs-reclaim/dm-]
admin 2436 0.0 0.0 0 0 ? S< 09:34 0:00 [xfs-log/dm-1]
admin 2437 0.0 0.0 0 0 ? S< 09:34 0:00 [xfs-eofblocks/d]
admin 2438 0.0 0.0 0 0 ? S 09:34 0:00 [xfsaild/dm-1]
admin 3131 0.0 0.0 0 0 ? S 09:39 0:00 [kworker/1:2]
admin 4592 0.0 0.0 21960 2948 ? Ss 09:34 0:00 lvmetad
admin 4594 0.0 0.0 0 0 ? S 09:39 0:00 [kworker/5:0]
admin 4612 0.0 0.0 4528 380 ? Ss 09:34 0:00 lvmpolld
admin 5480 0.0 0.0 0 0 ? S 09:34 0:00 [kissd]
nobody 5717 0.0 0.0 17508 5488 ? S 09:39 0:00 /web/cpshared/web/Apache/2.2.0/bin/httpd2 -k start -f /web/conf/httpd2.conf -D FOREGROUND
admin 6629 0.7 0.0 0 0 ? S 09:34 0:55 [fw_worker_0]
admin 6630 0.5 0.0 0 0 ? R 09:34 0:44 [fw_worker_1]
admin 6631 0.9 0.0 0 0 ? S 09:34 1:14 [fw_worker_2]
admin 6632 0.5 0.0 0 0 ? S 09:34 0:44 [fw_worker_3]
admin 6633 0.6 0.0 0 0 ? S 09:34 0:48 [fw_worker_4]
admin 6634 0.6 0.0 0 0 ? S 09:34 0:49 [fw_worker_5]
admin 6897 0.0 0.0 10768 532 ? S<sl 09:34 0:00 auditd
admin 6900 0.0 0.0 0 0 ? S 09:34 0:00 [kauditd]
admin 7476 0.0 0.1 30004 8128 ? Ss 09:34 0:00 /bin/pm
admin 7494 0.0 0.1 52208 15776 ? Ssl 09:34 0:04 /bin/confd
admin 7495 0.0 0.0 27172 5312 ? SNsl 09:34 0:01 /bin/searchd -niceboost 10
admin 7496 0.0 0.0 7504 2192 ? Ss 09:34 0:00 /usr/sbin/ntpd -n -g -c /etc/ntp.conf
admin 7497 0.0 0.0 3068 1276 ? Ss 09:34 0:00 /sbin/dhclient -d
admin 7506 0.0 0.1 90268 9988 ? Ssl 09:34 0:01 /bin/rconfd /etc/actions_mapping.xml
admin 7508 0.0 0.0 9096 4668 ? Ss 09:34 0:04 /bin/monitord
admin 7624 0.0 0.0 29848 7752 ? Ss 09:35 0:00 /bin/cloningd
admin 7625 0.0 0.0 16572 7052 pts/1 Ss+ 09:35 0:01 /bin/clishd default server
admin 7626 0.1 0.3 37452 25972 ? Ss 09:35 0:08 /opt/CPsuite-R81.10/fw1/Python/bin/python3.7 /bin/celery -A taskmanager.taskManager.celery
admin 7628 0.0 0.1 61132 9096 ? Ssl 09:35 0:00 /opt/CPCoreUploader/bin/core_uploader
admin 7631 0.0 0.0 21452 3992 ? Ssl 09:35 0:00 /bin/clish -p
admin 7632 0.0 0.1 20560 12504 ? Ss 09:35 0:00 /opt/CPsuite-R81.10/fw1/Python/bin/python3.7 -m gunicorn.app.wsgiapp -c /rest_api/rest-api-
admin 7633 0.0 0.0 29476 2208 ? Ssl 09:35 0:03 /bin/redis-server 127.0.0.1:6379
admin 7635 0.0 0.0 11672 2756 ? Ss 09:35 0:00 /usr/bin/dynamic_server
admin 7636 0.0 0.1 21128 13144 ? Ss 09:35 0:00 /opt/CPsuite-R81.10/fw1/Python/bin/python3.7 -m gunicorn.app.wsgiapp -c /rest_api/rest-api-
admin 7657 0.0 0.1 57568 9692 ? S<sl 09:35 0:00 /bin/routed -N
admin 7669 0.0 0.0 2412 532 ? Ss 09:35 0:00 /usr/sbin/acpid
admin 7696 0.0 0.1 76216 11844 ? S<l 09:35 0:00 /bin/routed -i default -f /etc/routed0.conf -h 0
admin 7712 0.0 0.0 8228 2012 ? Ss 09:35 0:00 /usr/sbin/sshd
admin 7740 0.0 0.0 3956 996 ? Ss 09:35 0:00 xinetd -stayalive -pidfile /var/run/xinetd.pid
admin 7779 0.0 0.0 4712 1360 ? Ss 09:35 0:00 crond
admin 7822 0.0 0.0 3320 1232 ? S 09:35 0:00 /bin/csh -fb /opt/CPshrd-R81.10/bin/cprid_wd
admin 7833 0.0 0.0 21228 5988 ? S 09:35 0:00 /opt/CPshrd-R81.10/bin/cprid
admin 7842 0.0 0.0 11912 4384 ? Ss 11:18 0:00 sshd: admin [priv]
admin 7848 0.0 0.0 11912 2744 ? S 11:18 0:00 sshd: admin@pts/2
admin 7849 0.0 0.0 23420 7332 pts/2 Ssl 11:18 0:00 /bin/clish
admin 7854 0.0 0.1 24988 14900 ? S 09:35 0:00 /opt/CPsuite-R81.10/fw1/Python/bin/python3.7 -m gunicorn.app.wsgiapp -c /rest_api/rest-api-
admin 7855 0.0 0.4 55124 33772 ? Sl 09:35 0:02 /opt/CPsuite-R81.10/fw1/Python/bin/python3.7 -m gunicorn.app.wsgiapp -c /rest_api/rest-api-
admin 7857 0.0 0.0 24768 6088 pts/1 Sl+ 11:18 0:00 /bin/clishd serving 7849
admin 7860 0.0 0.2 37024 22732 ? S 09:35 0:00 /opt/CPsuite-R81.10/fw1/Python/bin/python3.7 /bin/celery -A taskmanager.taskManager.celery
admin 7861 0.0 0.2 37024 22740 ? S 09:35 0:00 /opt/CPsuite-R81.10/fw1/Python/bin/python3.7 /bin/celery -A taskmanager.taskManager.celery
admin 7906 0.0 0.0 19380 3304 ? Ss 09:36 0:05 /opt/CPshrd-R81.10/bin/cpwd
admin 7919 0.0 0.6 218844 48192 ? Ss 09:36 0:05 cpviewd
admin 7939 0.0 0.2 40608 19184 ? Ss 09:36 0:07 cpview_services
admin 7944 0.0 0.2 38576 16452 ? Ss 09:36 0:01 cpview_api_service
admin 7947 0.0 0.1 29172 9636 ? Ss 09:36 0:00 sxl_statd
admin 7959 0.2 1.1 334848 90100 ? Ssl 09:36 0:19 cpd
admin 7972 0.0 0.1 160420 9672 ? Ss 09:36 0:00 mpdaemon /opt/CPshrd-R81.10/log/mpdaemon.elg /opt/CPshrd-R81.10/conf/mpdaemon.conf
admin 8017 0.0 0.3 975332 26256 ? Ssl 09:36 0:00 tp_conf_service --conf=tp_conf.json --log=error
admin 8073 0.0 0.0 17260 6912 ? Ss 09:36 0:00 /web/cpshared/web/Apache/2.2.0/bin/httpd2 -k start -f /web/conf/httpd2.conf -D FOREGROUND
admin 8090 0.0 0.0 3320 1244 ? Ss 09:36 0:00 /bin/csh -f /opt/CPsuite-R81.10/fw1/bin/avi_del_tmp_files
nobody 8094 0.0 0.0 25436 7180 ? Ss 09:36 0:00 ci_http_server -j -f /opt/CPsuite-R81.10/fw1/conf/cihs.conf
admin 8130 0.1 1.0 629688 86380 ? Ssl 09:36 0:13 fwd
admin 8137 0.0 0.0 33220 5352 ? Ss 09:36 0:05 spike_detective
admin 8171 0.0 0.0 0 0 ? S 09:36 0:00 [snd]
admin 8172 0.0 0.0 0 0 ? S 09:36 0:00 [snd]
admin 8173 0.0 0.0 0 0 ? S 09:36 0:00 [snd]
admin 8174 0.0 0.0 0 0 ? S 09:36 0:00 [snd]
admin 8175 0.0 0.0 0 0 ? S 09:36 0:00 [snd]
admin 8176 0.0 0.0 0 0 ? S 09:36 0:00 [snd]
admin 8177 0.0 0.0 0 0 ? S 09:36 0:00 [snd]
admin 8178 0.0 0.0 0 0 ? S 09:36 0:00 [snd]
admin 8179 0.0 0.0 0 0 ? S 09:36 0:00 [snd_c]
admin 8705 0.0 0.0 24628 5304 ? Ss 09:36 0:00 lpd
admin 8852 0.0 0.1 35096 9348 ? Ss 09:36 0:00 wsdnsd
admin 8887 0.0 0.7 361064 55572 ? Sl 09:36 0:01 in.asessiond 0
admin 8891 0.0 0.7 369656 56140 ? Sl 09:36 0:01 in.msd 0
admin 8893 0.0 0.3 117788 25516 ? S 09:36 0:00 tpd 0
admin 8894 0.0 0.2 50016 22596 ? SL 09:36 0:01 wstlsd 0 0
admin 8895 0.0 0.2 50008 22572 ? SL 09:36 0:02 wstlsd 0 1
admin 8896 0.0 0.2 50008 22540 ? SL 09:36 0:02 wstlsd 0 2
admin 8897 0.0 0.2 49996 22644 ? SL 09:36 0:02 wstlsd 0 3
admin 8898 0.4 0.2 50024 22628 ? SL 09:36 0:32 wstlsd 0 4
admin 8899 0.0 0.2 50016 22636 ? SL 09:36 0:02 wstlsd 0 5
nobody 8902 0.0 0.0 17508 5700 ? S 09:36 0:00 /web/cpshared/web/Apache/2.2.0/bin/httpd2 -k start -f /web/conf/httpd2.conf -D FOREGROUND
admin 8922 0.1 0.7 270636 57548 ? Sl 09:36 0:10 vpnd 0
admin 8932 0.0 0.6 267072 54108 ? Sl 09:36 0:03 iked 0
admin 9663 0.0 0.0 20756 2216 ? SLs 09:36 0:00 /opt/CPshrd-R81.10/bin/pkxld /cp/pkxl_8898_4114090432_1
admin 11874 0.0 0.0 0 0 ? S 09:46 0:00 [kworker/3:2]
admin 11933 0.0 0.0 0 0 ? S 11:31 0:00 [kworker/7:2]
admin 12272 0.0 0.0 3636 1348 ? Ss 09:36 0:00 /bin/bash /opt/CPda/bin/DAService_script
admin 12285 0.0 0.1 69796 11728 ? Sl 09:36 0:02 /opt/CPda/bin/DAService
admin 12287 0.0 0.0 3636 1356 ? Ss 09:36 0:00 /bin/bash /opt/AutoUpdater/latest/bin/AutoUpdaterService.sh
admin 12302 0.0 0.0 3636 1132 ? S 09:36 0:00 /bin/bash /opt/AutoUpdater/latest/bin/AutoUpdaterRun.sh
admin 12310 0.0 0.1 89760 13108 ? SNl 09:36 0:01 /opt/AutoUpdater/latest/bin/AutoUpdater
admin 12476 0.0 0.0 2748 1024 ? Ss 09:37 0:00 syslogd -m 0 -z 515 -P info -f /var/run/syslog.conf
admin 12487 0.0 0.0 2388 416 ? Ss 09:37 0:00 klogd -x -c 1
admin 12620 0.0 0.0 4536 780 tty1 Ss+ 09:37 0:00 /sbin/agetty --nohostname 9600 tty1
admin 12621 0.0 0.0 4536 780 tty2 Ss+ 09:37 0:00 /sbin/agetty --nohostname 9600 tty2
admin 12622 0.0 0.0 4536 780 tty3 Ss+ 09:37 0:00 /sbin/agetty --nohostname 9600 tty3
admin 12623 0.0 0.0 3636 1304 ? Ss 09:37 0:00 /bin/bash /bin/console_agetty
admin 12644 0.0 0.0 2488 828 ? S 09:37 0:00 /sbin/agetty --nohostname 9600 ttyS0 vt100
nobody 13222 0.0 0.0 17508 5680 ? S 09:37 0:00 /web/cpshared/web/Apache/2.2.0/bin/httpd2 -k start -f /web/conf/httpd2.conf -D FOREGROUND
admin 13301 0.0 0.0 2392 296 ? S 11:36 0:00 sleep 3600
admin 13341 0.0 0.0 0 0 ? S 11:36 0:00 [kworker/7:0]
admin 13855 0.0 0.0 0 0 ? S 11:37 0:00 [kworker/u16:2]
admin 14840 0.0 0.0 0 0 ? S 11:41 0:00 [kworker/7:1]
admin 15046 0.0 0.0 6336 2416 pts/2 S 11:41 0:00 /bin/bash
admin 15153 0.0 0.0 2392 296 ? S 11:42 0:00 /bin/sleep 5
admin 15163 0.0 0.0 2392 296 ? S 11:42 0:00 sleep 5
admin 15164 0.0 0.0 5944 1892 pts/2 R+ 11:42 0:00 ps -auxw
admin 15473 0.0 0.0 0 0 ? S 09:37 0:00 [kworker/0:11]
nobody 15593 0.0 0.0 17588 5720 ? S 09:37 0:00 /web/cpshared/web/Apache/2.2.0/bin/httpd2 -k start -f /web/conf/httpd2.conf -D FOREGROUND
admin 20054 0.0 0.0 0 0 ? S 10:10 0:00 [kworker/u16:1]
admin 20075 0.0 0.0 0 0 ? S 09:38 0:00 [kworker/6:2]
admin 24123 0.0 0.0 0 0 ? S 10:24 0:00 [kworker/u16:0]
admin 32040 0.0 0.0 0 0 ? S 09:38 0:00 [kworker/4:0]
admin 32120 0.3 0.0 0 0 ? S 09:38 0:23 [kworker/2:3]

| CPVIEW.Overview 04Apr2023 11:43:57 |
|----------------------------------------------------------------------------------------------------------------------------------------------------------|
| Overview SysInfo Network CPU I/O Software-blades Hardware-Health Advanced |
|----------------------------------------------------------------------------------------------------------------------------------------------------------|
| CPU: |
| |
| Num of CPUs: 8 |
| |
| CPU Used |
| 2 14% |
| 7 10% |
| 3 8% |
| -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Memory: |
| |
| Total MB Used MB Free MB |
| Physical 7,717 2,468 5,248 |
| FW 5,710 514 5,196 |
| Swap 15,994 0 15,994 |
| -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Network: |
| |
| Bits/sec 54,323K |
| Packets/sec 6,322 |
| Connections/sec 1 |
| Concurrent connections 282 |
| -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Disk space (top 3 used partitions): |
| |
| Partition Total MB Used MB Free MB |
| / 32,752 8,043 24,708 |
| /boot 290 29 245 |
| /var/log 450,340 5,863 444,476 |
| -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Events: |
| |
| # of monitored daemons crashes since last cpstart 0

| CPVIEW.SysInfo 04Apr2023 11:44:23 |
|----------------------------------------------------------------------------------------------------------------------------------------------------------|
| Overview SysInfo Network CPU I/O Software-blades Hardware-Health Advanced |
|----------------------------------------------------------------------------------------------------------------------------------------------------------|
| Configuration Information: |
| |
| Platform Gaia 64Bit |
| Configuration Check Point Security Gateway |
| CoreXL Status on |
| CoreXL instances 6 |
| Dynamic Balancing Status Off |
| SecureXL Status Off |
| USFW Status off |
| UPPAK Status Off |
| -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| General information: |
| |
| System uptime 0 days, 02:10:46 |
| Last policy install time 04Apr2023 9:36:32 |
| Last policy name Standard_New |
| -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Version Information: |
| |
| fw1_wrapper package version R81.10 |
| |
| Branch Name Build Number |
| FW User Mode R81_10_jumbo_hf_main 996000035 |
| FW Kernel R81_10_jumbo_hf_main 996000036 |
| Accel Module R81_10_jumbo_hf_main 996000016 |
| Adpdrv Module R81_10_jumbo_hf_main 996000009 |
| SIM Module R81_10_jumbo_hf_main 996000016 |
| -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Hardware Information: |
| |
| HW Model Open Server |
| |
| Devices Type Slot Status Firmware |
| - - - - -

| CPVIEW.Network 04Apr2023 11:44:49 |
|----------------------------------------------------------------------------------------------------------------------------------------------------------|
| Overview SysInfo Network CPU I/O Software-blades Hardware-Health Advanced |
|----------------------------------------------------------------------------------------------------------------------------------------------------------|
| Traffic Distribution Interfaces Top-Protocols Top-Connections LightSpeed-Top-Connections |
|----------------------------------------------------------------------------------------------------------------------------------------------------------|
| Traffic Rate: |
| |
| Inbound packets/sec Outbound packets/sec Inbound bits/sec Outbound bits/sec Connections/sec |
| Host 0 8,090 0 69,643K 0 |
| LightSpeed 0 0 0 0 0 |
| -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Concurrent Connections: |
| |
| Connections Non-TCP TCP handshake TCP established TCP closed |
| FW 320 75 49 191 5 |
| Host Streaming 0 0 0 0 0 |
| Host SXL (including LightSpeed) 19 0 0 19 0 |
| LightSpeed 0 0 0 0 0 |
| -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Templates: |
| |
| % Connections from templates 0% |
| % Unused templates 0% |
| -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Drops: |
| |
| Software Blades 292 |
| Interface incoming drops 0 |
| Instance high CPU 0 |
| Rulebase 21,628 |
| Capacity 0 |
| SecureXL 0 |
| Drop out of state TCP enabled |
| First packets not SYN 8,612

| CPVIEW.CPU 04Apr2023 11:45:12 |
|----------------------------------------------------------------------------------------------------------------------------------------------------------|
| Overview SysInfo Network CPU I/O Software-blades Hardware-Health Advanced |
|----------------------------------------------------------------------------------------------------------------------------------------------------------|
| Overview Top-Protocols Top-Connections Spikes |
|----------------------------------------------------------------------------------------------------------------------------------------------------------|
| Host |
|----------------------------------------------------------------------------------------------------------------------------------------------------------|
| Overview: |
| |
| CPU type CPUs Avg utilization |
| CoreXL_SND 1 7% |
| OTHER 1 0% |
| CoreXL_FW 6 8% |
| -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| CPU: |
| |
| CPU Type User System Idle I/O wait Interrupts |
| 0 CoreXL_SND 0% 7% 93% 0% 26,916 |
| 1 OTHER 0% 0% 100% 0% 26,918 |
| 2 CoreXL_FW 5% 8% 88% 0% 26,919 |
| 3 CoreXL_FW 2% 5% 93% 0% 26,915 |
| 4 CoreXL_FW 0% 2% 98% 0% 26,916 |
| 5 CoreXL_FW 1% 3% 97% 0% 26,917 |
| 6 CoreXL_FW 2% 12% 86% 0% 26,918 |
| 7 CoreXL_FW 4% 5% 91% 0% 26,918

| CPVIEW.I/O 04Apr2023 11:45:35 |
|----------------------------------------------------------------------------------------------------------------------------------------------------------|
| Overview SysInfo Network CPU I/O Software-blades Hardware-Health Advanced |
|----------------------------------------------------------------------------------------------------------------------------------------------------------|
| Overview |
|----------------------------------------------------------------------------------------------------------------------------------------------------------|
| Requests Per Second: |
| |
| Device Merged read Merged write Read Write |
| sda 0.00 0.00 0.00 1.00 |
| dm-0 0.00 0.00 0.00 0.00 |
| dm-1 0.00 0.00 0.00 1.00 |
| -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Read/Write Per Second: |
| |
| Device Read KB Write KB |
| sda 0.00 6.00 |
| dm-0 0.00 0.00 |
| dm-1 0.00 6.00 |
| -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Average: |
| |
| Device Request Size Queue Size Wait Service |
| sda 6.00 0.00 0.00 0.00 |
| dm-0 0.00 0.00 0.00 0.00 |
| dm-1 6.00 0.00 0.00 0.00 |
| -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| CPU Utilization: |
| |
| Device I/O |
| sda 0.00% |
| dm-0 0.00% |
| dm-1 0.00%

| CPVIEW.Software-blades 04Apr2023 11:46:07 |
|----------------------------------------------------------------------------------------------------------------------------------------------------------|
| Overview SysInfo Network CPU I/O Software-blades Hardware-Health Advanced |
|----------------------------------------------------------------------------------------------------------------------------------------------------------|
| Overview VPN SSL-Inspection IDA DLP Threat-Prevention Threat-Emulation Advanced Content-Awareness QoS URLF Application-Control |
|----------------------------------------------------------------------------------------------------------------------------------------------------------|
| Updates Information: |
| |
| Blade status Last update Number Update Time |
| Application Control disabled N/A N/A |
| Anti-Virus disabled 1109220741 03Apr2023 10:40:15 |
| Anti-Bot disabled N/A N/A |
| IPS disabled N/A N/A

| CPVIEW.Hardware-Health 04Apr2023 11:47:19 |
|----------------------------------------------------------------------------------------------------------------------------------------------------------|
| Overview SysInfo Network CPU I/O Software-blades Hardware-Health Advanced |
|----------------------------------------------------------------------------------------------------------------------------------------------------------|
| Temperature Voltage Power-Supply Fan Bios |
|----------------------------------------------------------------------------------------------------------------------------------------------------------|
| Information |
| |
| Name Value Unit Min Max Status |
| - - - - - -

| CPVIEW.Advanced 04Apr2023 11:47:43 |
|----------------------------------------------------------------------------------------------------------------------------------------------------------|
| Overview SysInfo Network CPU I/O Software-blades Hardware-Health Advanced |
|----------------------------------------------------------------------------------------------------------------------------------------------------------|
| Logging CPU-Profiler Memory Network SDWAN SecureXL ClusterXL CoreXL PrioQ Streaming NAT MUX Routed RAD Conn-Tracker UP HTTP-Parser SSH-Parser CPAQ >>
|----------------------------------------------------------------------------------------------------------------------------------------------------------|
| Log-Rates |
|----------------------------------------------------------------------------------------------------------------------------------------------------------|
| |
| |
| Generated Log Rate 5 |
| Local Logging Rate 0 |
| |
| IP Log Sending Rate |
| Log Server 10.10.1.25 5

[Expert@fw:0]# fw ctl multik print_heavy_conn
[Expert@fw:0]#

whoa · ‎2023-04-04

.

the_rock · ‎2023-04-04

Looks like you may have memory issue, based on free -m output.

whoa · ‎2023-04-04

total: 7717

used: 1858

free: 3751

shared: 7

buff/cache: 2106

available: 5256

Please explain where the problem lay? Memory looks fine to me.

the_rock · ‎2023-04-04

My bad, I calculated wrong, I looked at the wrong column, apologies mate. By the way, if you run cpview, what does memory usage show there?

Andy

whoa · ‎2023-04-04

It took a few hours, but a very large post finally posted (up above). The memory is found there. FYI - I couldnt be down for another day so I have downgraded to R77.30. I wont be able to re-visit R81.10 until this weekend. With that said, I very much believe that this is a matter of software settings since at R77.30 everything works perfectly...

Are you a member of CheckMates?

GAIA Upgrade from R77.30 to R80.40 - Firewall Dropping Packets