Hi CheckMates,
I'm encountering an issue since 4 months, after a migration from old 14000 boxes running on r77.30 to 15000series on r80.30.
CPU cores were saturation could be seen imediatly after the migration, 4 Months of investigation with TAC, first issue identified was a bug that most of tcp 443/445 tarfic was taking the medium path, we used fast accel feature to staticly accelerate it, this bug should have been fixed in JHF 219, and indeed it seems to as we dont use fast accel today.
But the performance is still an issue, our FWKs are maxed out, and thus all Cores related during bussiness day.
Technical infos :
VSX cluster running r80.30 take 219 new kernel.
Blades : FW + IPS
Stats :
[Expert@EU93A0391-OSS:1]# fwaccel stats -s
Accelerated conns/Total conns : 27355/185090 (14%)
Accelerated pkts/Total pkts : 981191859249/1466990930587 (66%)
F2Fed pkts/Total pkts : 8887485442/1466990930587 (0%)
F2V pkts/Total pkts : 2802683200/1466990930587 (0%)
CPASXL pkts/Total pkts : 0/1466990930587 (0%)
PSLXL pkts/Total pkts : 476911585896/1466990930587 (32%)
QOS inbound pkts/Total pkts : 0/1466990930587 (0%)
QOS outbound pkts/Total pkts : 0/1466990930587 (0%)
Corrected pkts/Total pkts : 0/1466990930587 (0%)
[Expert@xxxxxxx:0]# fw ctl affinity -l
eth3-01: CPU 0
eth3-02: CPU 0
Mgmt: CPU 0
VS_0: CPU 4 5 6 7 8 9 10 11 12 13 14 15 20 21 22 23 24 25 26 27 28 29 30 31
VS_0 fwk: CPU 4 5 6 7 8 9 10 11 12 13 14 15 20 21 22 23 24 25 26 27 28 29 30 31
VS_1: CPU 4 5 6 7 8 9 10 11 12 13 14 15 20 21 22 23 24 25 26 27 28 29 30 31
VS_1 fwk: CPU 4 5 6 7 8 9 10 11 12 13 14 15 20 21 22 23 24 25 26 27 28 29 30 31
VS_1 pepd: CPU 16
VS_1 pdpd: CPU 17
VS_2: CPU 4 5 6 7 8 9 10 11 12 13 14 15 20 21 22 23 24 25 26 27 28 29 30 31
VS_2 fwk: CPU 4 5 6 7 8 9 10 11 12 13 14 15 20 21 22 23 24 25 26 27 28 29 30 31
VS_3: CPU 4 5 6 7 8 9 10 11 12 13 14 15 20 21 22 23 24 25 26 27 28 29 30 31
VS_3 fwk: CPU 4 5 6 7 8 9 10 11 12 13 14 15 20 21 22 23 24 25 26 27 28 29 30 31
Interface eth1-01: has multi queue enabled
Interface eth1-02: has multi queue enabled
[Expert@EU93A0391-OSS:1]# fw ctl multik stat
ID | Active | CPU | Connections | Peak
----------------------------------------------
0 | Yes | 4-15+ | 11859 | 16320
1 | Yes | 4-15+ | 10365 | 14918
2 | Yes | 4-15+ | 11278 | 14150
3 | Yes | 4-15+ | 10394 | 14274
4 | Yes | 4-15+ | 9037 | 12512
5 | Yes | 4-15+ | 9715 | 14493
6 | Yes | 4-15+ | 10179 | 13757
7 | Yes | 4-15+ | 10707 | 15184
8 | Yes | 4-15+ | 9731 | 13823
9 | Yes | 4-15+ | 10668 | 12959
10 | Yes | 4-15+ | 10530 | 15033
11 | Yes | 4-15+ | 9930 | 14691
12 | Yes | 4-15+ | 1673 | 13807
13 | Yes | 4-15+ | 9023 | 12942
14 | Yes | 4-15+ | 8952 | 14260
15 | Yes | 4-15+ | 9620 | 13936
16 | Yes | 4-15+ | 10442 | 13535
17 | Yes | 4-15+ | 10177 | 13863
18 | Yes | 4-15+ | 2597 | 13612
19 | Yes | 4-15+ | 9941 | 13543
20 | Yes | 4-15+ | 7191 | 14152
21 | Yes | 4-15+ | 9821 | 12975
22 | Yes | 4-15+ | 9721 | 12999
23 | Yes | 4-15+ | 6742 | 13279
Main VS runing 18 fwk instances.
All cpu cores handeling VS traffics are on avg 80% usage, and most of them maxing out during bussiness hours.
We have no lead atm with TAC (prety slow response, and knowledge to be honest).
