topic Re: CPSG 23800 Performance Issue in Firewall and Security Management

CPSG 23800 Performance Issue

dj0Nz — Thu, 15 Nov 2018 10:02:31 GMT

I am currently working on an issue with a brand new CPSG-23800 device:

- Two 10g Interfaces

- One Firewall Rule (Any-Any-Accept-Log)

- CP R80.10 without HF

As we tested throughput, there is a maximum of 1.6 gbps going though the box, no matter how much traffic we try to generate. We also tried with multiple systems/flow but it seems like the firewall has a fixed 1.6 gbps Limit.

We enabled multi queueing, configured more SND cores, no change.

Log message (drop debug) is:

"fwmultik_enqueue_packet_kernel Reason: Instance is currently fully utilized"

But: Buffer is already at 16k...

Any ideas?

Cheers

Michael

Re: CPSG 23800 Performance Issue

dj0Nz — Thu, 15 Nov 2018 10:04:34 GMT

BTW: The system does not report ANY error. No receive queue drops. Not a single error visible with ethtool...

Re: CPSG 23800 Performance Issue

Kaspars_Zibarts — Thu, 15 Nov 2018 11:43:19 GMT

We are pushing 10G on our production 23800 VSX and it's nowhere near the limit Could easily double it. Just to ease your mind.

We would need more info on your CoreXL and SXL set up. Start with:

fw ctl affinity -l

cpmq get -vv

top (with all cores visible, so press 1, during the test)

fwaccel stats -s (when you are running your test)

Re: CPSG 23800 Performance Issue

dj0Nz — Thu, 15 Nov 2018 12:05:31 GMT

Currently, we are working with a traffic generator which generates 10 sessions of ip protocol 254 traffic (alltogether 6 gbps) which is not getting accelerated. But nevertheless I think the box should be able to push about 6 to 7 gbps.

We did the following:

- disable hyperthreading

- configure 12 snd cores

- reconfigure cpmq

cpmq get -vv Output:

Active ixgbe interfaces:
eth3-01 [On]
eth3-02 [On]

Active igb interfaces:
Mgmt [Off]

The rx_num for ixgbe is: 12 (default)

multi-queue affinity for ixgbe interfaces:
CPU | TX | Vector                      | RX Packets          | RX Bytes
-----------------------------------------------------------------------------------
0   | 0 | eth3-01-TxRx-0 (68)         | 184                 | 11040
    |    | eth3-02-TxRx-0 (228)        |                     |
1   | 1 | eth3-01-TxRx-1 (76)         | 0                   | 0
    |    | eth3-02-TxRx-1 (236)        |                     |
2   | 2 | eth3-01-TxRx-2 (84)         | 0                   | 0
    |    | eth3-02-TxRx-2 (53)         |                     |
3   | 3 | eth3-01-TxRx-3 (92)         | 0                   | 0
    |    | eth3-02-TxRx-3 (61)         |                     |
4   | 4 | eth3-01-TxRx-4 (100)        | 0                   | 0
    |    | eth3-02-TxRx-4 (69)         |                     |
5   | 5 | eth3-01-TxRx-5 (108)        | 0                   | 0
    |    | eth3-02-TxRx-5 (77)         |                     |
6   | 6 | eth3-01-TxRx-6 (116)        | 0                   | 0
    |    | eth3-02-TxRx-6 (85)         |                     |
7   | 7 | eth3-01-TxRx-7 (124)        | 99358895            | 45794052878
    |    | eth3-02-TxRx-7 (93)         |                     |
8   | 8 | eth3-01-TxRx-8 (132)        | 0                   | 0
    |    | eth3-02-TxRx-8 (101)        |                     |
9   | 9 | eth3-01-TxRx-9 (140)        | 0                   | 0
    |    | eth3-02-TxRx-9 (109)        |                     |
10 | 10 | eth3-01-TxRx-10 (148)       | 0                   | 0
    |    | eth3-02-TxRx-10 (117)       |                     |
11 | 11 | eth3-01-TxRx-11 (156)       | 0                   | 0
    |    | eth3-02-TxRx-11 (125)       |                     |
12 | 12 |                             |                     |
13 | 13 |                             |                     |
14 | 14 |                             |                     |
15 | 15 |                             |                     |
16 | 16 |                             |                     |
17 | 17 |                             |                     |
18 | 18 |                             |                     |
19 | 19 |                             |                     |
20 | 20 |                             |                     |
21 | 21 |                             |                     |
22 | 22 |                             |                     |
23 | 23 |                             |                     |

Top Output:

top - 13:05:22 up 6 min, 2 users, load average: 7.62, 5.70, 2.62
Tasks: 318 total, 7 running, 311 sleeping, 0 stopped, 0 zombie
Cpu0 : 0.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.3%hi, 99.7%si, 0.0%st
Cpu1 : 0.0%us, 0.0%sy, 0.0%ni, 85.3%id, 0.0%wa, 0.0%hi, 14.7%si, 0.0%st
Cpu2 : 0.0%us, 0.0%sy, 0.0%ni, 94.4%id, 0.0%wa, 0.7%hi, 5.0%si, 0.0%st
Cpu3 : 0.0%us, 0.0%sy, 0.0%ni, 86.3%id, 0.0%wa, 0.3%hi, 13.3%si, 0.0%st
Cpu4 : 0.0%us, 0.0%sy, 0.0%ni, 96.3%id, 0.0%wa, 0.3%hi, 3.3%si, 0.0%st
Cpu5 : 0.0%us, 0.0%sy, 0.0%ni, 96.3%id, 0.0%wa, 0.7%hi, 3.0%si, 0.0%st
Cpu6 : 0.0%us, 0.0%sy, 0.0%ni, 84.4%id, 0.0%wa, 0.7%hi, 15.0%si, 0.0%st
Cpu7 : 0.0%us, 0.0%sy, 0.0%ni, 24.3%id, 0.0%wa, 0.0%hi, 75.7%si, 0.0%st
Cpu8 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu9 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu10 : 0.0%us, 0.0%sy, 0.0%ni, 96.3%id, 0.0%wa, 0.3%hi, 3.3%si, 0.0%st
Cpu11 : 0.0%us, 0.0%sy, 0.0%ni, 96.0%id, 0.0%wa, 0.3%hi, 3.7%si, 0.0%st
Cpu12 : 0.0%us, 0.7%sy, 0.0%ni, 57.8%id, 0.0%wa, 0.0%hi, 41.5%si, 0.0%st
Cpu13 : 0.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi,100.0%si, 0.0%st
Cpu14 : 0.3%us, 1.0%sy, 0.0%ni, 67.8%id, 0.0%wa, 0.0%hi, 30.9%si, 0.0%st
Cpu15 : 0.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi,100.0%si, 0.0%st
Cpu16 : 0.0%us, 0.7%sy, 0.0%ni, 79.7%id, 0.0%wa, 0.0%hi, 19.6%si, 0.0%st
Cpu17 : 0.3%us, 0.7%sy, 0.0%ni, 77.7%id, 0.0%wa, 0.0%hi, 21.3%si, 0.0%st
Cpu18 : 0.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi,100.0%si, 0.0%st
Cpu19 : 0.0%us, 0.7%sy, 0.0%ni, 59.5%id, 0.0%wa, 0.0%hi, 39.9%si, 0.0%st
Cpu20 : 0.3%us, 0.0%sy, 0.0%ni, 99.3%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st
Cpu21 : 0.0%us, 0.0%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st
Cpu22 : 0.0%us, 1.0%sy, 0.0%ni, 73.1%id, 0.0%wa, 0.0%hi, 25.9%si, 0.0%st
Cpu23 : 0.0%us, 0.7%sy, 0.0%ni, 75.7%id, 0.0%wa, 0.0%hi, 23.7%si, 0.0%st

fw ctl affinity -l:

Mgmt: CPU 0
Kernel fw_0: CPU 23
Kernel fw_1: CPU 22
Kernel fw_2: CPU 21
Kernel fw_3: CPU 20
Kernel fw_4: CPU 19
Kernel fw_5: CPU 18
Kernel fw_6: CPU 17
Kernel fw_7: CPU 16
Kernel fw_8: CPU 15
Kernel fw_9: CPU 14
Kernel fw_10: CPU 13
Kernel fw_11: CPU 12
Daemon mpdaemon: CPU 12 13 14 15 16 17 18 19 20 21 22 23
Daemon fwd: CPU 12 13 14 15 16 17 18 19 20 21 22 23
Daemon lpd: CPU 12 13 14 15 16 17 18 19 20 21 22 23
Daemon in.asessiond: CPU 12 13 14 15 16 17 18 19 20 21 22 23
Daemon cprid: CPU 12 13 14 15 16 17 18 19 20 21 22 23
Daemon cpd: CPU 12 13 14 15 16 17 18 19 20 21 22 23
Interface eth3-01: has multi queue enabled
Interface eth3-02: has multi queue enabled

Re: CPSG 23800 Performance Issue

Kaspars_Zibarts — Thu, 15 Nov 2018 12:15:15 GMT

you forgot the fwaccel stats -s

but there are couple of things sticking out: MQ is running everything through one CPU core (#7), so you are not really using MQ

have you reset affinity after you changed it to 12?

And for pure throughput tests you will need to get acceleration working for that traffic, else you are hammering CPU core on system interrupts

Re: CPSG 23800 Performance Issue

Kaspars_Zibarts — Thu, 15 Nov 2018 12:17:50 GMT

I'm also not too sure about whole MQ and SXL behaviour with IP proto 254.. I would switch to something more regular as UDP or TCP

Re: CPSG 23800 Performance Issue

dj0Nz — Thu, 15 Nov 2018 12:24:03 GMT

I didn't forget it, it's useless because the ip protocol 254 traffic the traffic generator generates is not accelerated in any way. But that's not the point. I think, neither MQ nor dynamic patcher is working for any reason...

Re: CPSG 23800 Performance Issue

dj0Nz — Thu, 15 Nov 2018 12:25:17 GMT

Oh, I forgot to mention that cpmq reconfigure should deal with affinities. Isn't it?

Re: CPSG 23800 Performance Issue

Kaspars_Zibarts — Thu, 15 Nov 2018 13:21:20 GMT

Well, not accelerating traffic will make tremendous impact on throughput. All I can suggest is try to set up your traffic generator to use standard protocols

Re: CPSG 23800 Performance Issue

Kaspars_Zibarts — Thu, 15 Nov 2018 13:23:24 GMT

From memory, if you change number of cores handling specific interface type, you must run set affinity. I'm not entirely sure what you meant by cpmq reconfigure? there is no such option

Re: CPSG 23800 Performance Issue

PhoneBoy — Fri, 16 Nov 2018 19:37:26 GMT

How many source/destination IPs are being used here?

If you're using only a single source and a single destination, particularly with something that isn't UDP/TCP, you've got yourself an elephant flow.

Dynamic Dispatcher won't help in this case.

Re: CPSG 23800 Performance Issue

dj0Nz — Mon, 17 Dec 2018 11:32:17 GMT

Hm indeed maybe this is related to the nature of the traffic. We've been running tests witch multiple flows starting from 100 mbit and increasing traffic after 5 seconds in 10 mbit steps. During one of these tests, the firewall began to drop packets (without logging) at a rate of 600 mbit...

Nevertheless, we advised the customer to use two standard linux boxes with 10g interfaces and repeat the tests with iperf3. With that, we were able to push about 9 Gbps. Enough in that case to prove the Checkpoint box is not the problem here.

Thanks to all of you for your help and effort.