Cluster Capacity - peak/concurrent connections

[Sorin_Gogean]

Hello everyone,

Not sure why, lately we had seen an increase in memory utilization (like it doubled) and I was able to determine that it's due to some traffic spikes. 

Memory utilization, it jumped from ~45% utilization to ~80% . Our GWs are 15600 with 32Gb memory (and quite some blades). 

So, I tried to identify what traffic caused that, see some sources/destinations or anything that can get us close to a conclusion.

Sadly I wasn't lucky enough to get anywhere, therefore I come here asking for some guidance.

In order to prevent this, I looked for a way to limit concurrent connections per IP/client, but I'm not yet there (using fwaccel dos rate ) so any hints are wellcomed.

Here is how fw ctl pstat results show on a node... that "1145453 peak concurrent" bothers me 😁 - wth 1mil ?!?!?!?!

Roughly, I look for a way to get some reports, either from the Manager or from the box itself when the connections are over 500K (some value) to get the list of the connection table that I can work with and get some data out of it - still 500K or 1Mil .... 

ALVA-FW01

ALVA-FW01> fw ctl pstat System Capacity Summary:   Memory used: 48% (11578 MB out of 23889 MB) - below watermark   Concurrent Connections: 54553 (Unlimited)   Aggressive Aging is enabled, not active Hash kernel memory (hmem) statistics:   Total memory allocated: 13925134336 bytes in 3399691 (4096 bytes) blocks using 11 pools   Initial memory allocated: 2503999488 bytes (Hash memory extended by 11421134848 bytes)   Memory allocation  limit: 20039335936 bytes using 512 pools   Total memory bytes  used:        0   unused: 13925134336 (100.00%)   peak: 14058217444   Total memory blocks used:        0   unused:  3399691 (100%)   peak:  3592449   Allocations: 3826885158 alloc, 0 failed alloc, 3801372538 free System kernel memory (smem) statistics:   Total memory  bytes  used: 19378365776   peak: 20195144584   Total memory bytes wasted: 95203288     Blocking  memory  bytes   used: 69845532   peak: 110230372     Non-Blocking memory bytes used: 19308520244   peak: 20084914212   Allocations: 580197892 alloc, 0 failed alloc, 580126896 free, 0 failed free   vmalloc bytes  used: 19216527896 expensive: no Kernel memory (kmem) statistics:   Total memory  bytes  used: 8419234052   peak: 16326533036   Allocations: 112078525 alloc, 0 failed alloc                86508537 free, 0 failed free   External Allocations:     Packets: 66761920, SXL: 0, Reorder: 0     Zeco: 0, SHMEM: 94392, Resctrl: 0     ADPDRV: 0, PPK_CI: 0, PPK_CORR: 0 Cookies:         397638576 total, 394223007 alloc, 394212203 free,         4272844296 dup, 621658599 get, 2526281133 put,         2705746389 len, 2027218867 cached len, 0 chain alloc,         0 chain free Connections:         673523638 total, 296395981 TCP, 359631398 UDP, 17496203 ICMP,         56 other, 39952 anticipated, 195487 recovered, 54554 concurrent,         1145453 peak concurrent Fragments:         8688744 fragments, 4341654 packets, 14 expired, 0 short,         0 large, 0 duplicates, 0 failures NAT:         2579202207/0 forw, 2673121164/0 bckw, 6811102365 tcpudp,         33611286 icmp, 358817824-291829883 alloc Sync: Run "cphaprob syncstat" for cluster sync statistics. ALVA-FW01>

A TAC will be opened on Monday....