topic Re: Latency on Links in General Topics

Latency on Links

Terry — Wed, 16 Jan 2019 21:08:00 GMT

Hello,

We have been experiencing latency on our Bonded links. It has had a major impact on product and caused us to move our bonded gateway back to our core switch. I simply am not familiar enough with the different core configurations to make an assessment here. Originally I setup our bond with multi-queuing. Can someone assist? Here is the Super 7 output:

[Expert@CSBFW-PROD-A:0]# /usr/bin/s7pac
+-----------------------------------------------------------------------------+
| Super Seven Performance Assessment Commands v0.3 (Thanks to Timothy Hall) |
+-----------------------------------------------------------------------------+
| Inspecting your environment: OK |
| This is a firewall....(continuing) |
| |
| Referred pagenumbers are to be found in the following book: |
| Max Power: Check Point Firewall Performance Optimization - Second Edition |
| Available at http://www.maxpowerfirewalls.com/ |
| |
+-----------------------------------------------------------------------------+
| Command #1: fwaccel stat |
| |
| Check for : Accelerator Status must be enabled (R77.xx/R80.10 versions) |
| Status must be enabled (R80.20) |
| Accept Templates must be enabled |
| Message "disabled" from (low rule number) = bad |
| |
| Chapter 9: SecureXL throughput acceleration |
| Page 278 |
+-----------------------------------------------------------------------------+
| Output: |
Accelerator Status : on
Accept Templates : enabled
Drop Templates : disabled
NAT Templates : disabled by user

Accelerator Features : Accounting, NAT, Cryptography, Routing,
HasClock, Templates, Synchronous, IdleDetection,
Sequencing, TcpStateDetect, AutoExpire,
DelayedNotif, TcpStateDetectV2, CPLS, McastRouting,
WireMode, DropTemplates, NatTemplates,
Streaming, MultiFW, AntiSpoofing, Nac,
ViolationStats, AsychronicNotif, ERDOS,
NAT64, GTPAcceleration, SCTPAcceleration,
McastRoutingV2
Cryptography Features : Tunnel, UDPEncapsulation, MD5, SHA1, NULL,
3DES, DES, CAST, CAST-40, AES-128, AES-256,
ESP, LinkSelection, DynamicVPN, NatTraversal,
EncRouting, AES-XCBC, SHA256

+-----------------------------------------------------------------------------+
| Command #2: fwaccel stats -s |
| |
| Check for : Accelerated conns/Totals conns: >25% good, >50% great |
| Accelerated pkts/Total pkts : >50% great |
| PXL pkts/Total pkts : >50% OK |
| F2Fed pkts/Total pkts : <30% good, <10% great |
| |
| Chapter 9: SecureXL throughput acceleration |
| Page 287, Packet/Throughput Acceleration: The Three Kernel Paths |
+-----------------------------------------------------------------------------+
| Output: |
Accelerated conns/Total conns : 0/35 (0%)
Accelerated pkts/Total pkts : 0/3983956 (0%)
F2Fed pkts/Total pkts : 3983956/3983956 (100%)
PXL pkts/Total pkts : 0/3983956 (0%)
QXL pkts/Total pkts : 0/3983956 (0%)

+-----------------------------------------------------------------------------+
| Command #3: grep -c ^processor /proc/cpuinfo && /sbin/cpuinfo |
| |
| Check for : If number of cores is roughly double what you are excpecting, |
| hyperthreading may be enabled |
| |
| Chapter 7: CorexL Tuning |
| Page 239 |
+-----------------------------------------------------------------------------+
| Output: |
6
HyperThreading=disabled

+-----------------------------------------------------------------------------+
| Command #4: fw ctl affinity -l -r |
| |
| Check for : SND/IRQ/Dispatcher Cores, # of CPU's allocated to interface(s) |
| Firewall Workers/INSPECT Cores, # of CPU's allocated to fw_x |
| R77.30: Support processes executed on ALL CPU's |
| R80.xx: Support processes only executed on Firewall Worker Cores|
| |
| Chapter 7: CoreXL Tuning |
| Page 221 |
+-----------------------------------------------------------------------------+
| Output: |
CPU 0: eth1-04
CPU 1: Sync Mgmt
CPU 2:
CPU 3:
CPU 4:
CPU 5:
All:

+-----------------------------------------------------------------------------+
| Command #5: netstat -ni |
| |
| Check for : RX/TX errors |
| RX-DRP % should be <0.1% calculated by (RX-DRP/RX-OK)*100 |
| TX-ERR might indicate Fast Ethernet/100Mbps Duplex Mismatch |
| |
| Chapter 2: Layers 1&2 Performance Optimization |
| Page 28-35 |
| |
| Chapter 7: CoreXL Tuning |
| Page 204 |
+-----------------------------------------------------------------------------+
| Output: |
Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
Mgmt 1500 0 43600162 2 0 0 36013528 0 0 0 BMRU
Sync 1500 0 61606193 0 0 0 283637876 0 0 0 BMRU
eth1-04 1500 0 5426765033 0 495 495 4930747640 0 0 0 BMRU
lo 16436 0 8398237 0 0 0 8398237 0 0 0 LRU

interface eth1-04: There are no RX drops

Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1| 8| 8| 84| 16| ?| 4|
| 2| 5| 6| 89| 11| ?| 4|
| 3| 4| 5| 91| 9| ?| 4|
| 4| 4| 4| 92| 8| ?| 4|
| 5| 4| 4| 92| 8| ?| 4|
| 6| 4| 4| 93| 7| ?| 4|
---------------------------------------------------------------------------------

Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1| 14| 16| 70| 30| ?| 2726|
| 2| 4| 11| 86| 14| ?| 2726|
| 3| 5| 8| 87| 13| ?| 5452|
| 4| 5| 6| 89| 11| ?| 2726|
| 5| 8| 9| 84| 16| ?| 2726|
| 6| 10| 9| 81| 19| ?| 5452|
---------------------------------------------------------------------------------

Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1| 10| 26| 64| 36| ?| 694|
| 2| 3| 9| 89| 11| ?| 694|
| 3| 5| 8| 87| 13| ?| 694|
| 4| 5| 10| 86| 14| ?| 694|
| 5| 6| 14| 80| 20| ?| 695|
| 6| 7| 13| 80| 20| ?| 697|
---------------------------------------------------------------------------------

+-----------------------------------------------------------------------------+
| Thanks for using s7pac |

Re: Latency on Links

Timothy_Hall — Wed, 16 Jan 2019 22:05:41 GMT

Question: Why is CoreXL disabled?

Everything seems to be going F2F/slowpath through only one core, not sure if that high F2F is due to CoreXL being off or if you are experiencing fragmentation issues. Not surprised at all that you are seeing high latency.

Please provide output of:

enabled_blades

fwaccel stats -p

CheckMates Break Out Sessions Speaker

CPX 2019 Las Vegas & Vienna - Tuesday@13:30

Re: Latency on Links

Alessandro_Marr — Thu, 17 Jan 2019 11:41:29 GMT

I guess he should check HyperThreading too...

Re: Latency on Links

Terry — Thu, 17 Jan 2019 14:19:37 GMT

It appears that I have 5 physical cores. So I plan to enable corexl and set 2 of these cores for the FW instances if further research proves this to be the best path. This system has been a bit of a lesson in hard knocks so I really appreciate your taking the time to assist.

[Expert@a:0]# vsenv 10
Context is set to Virtual Device a:10_PRODFW01 (ID 10).
[Expert@a:10]# enabled_blades
fw vpn cvpn urlf av appi identityServer
[Expert@a:10]# vsenv 15
Context is set to Virtual Device a:15_PRODFW02 (ID 15).
[Expert@a:15]# enabled_blades
fw identityServer

F2F packets:
--------------
Violation Packets Violation Packets
-------------------- --------------- -------------------- ---------------
pkt is a fragment 0 pkt has IP options 4335
ICMP miss conn 40 TCP-SYN miss conn 3578
TCP-other miss conn 39 UDP miss conn 1930086
other miss conn 3603 VPN returned F2F 0
ICMP conn is F2Fed 0 TCP conn is F2Fed 4831
UDP conn is F2Fed 51060 other conn is F2Fed 0
uni-directional viol 0 possible spoof viol 0
TCP state viol 0 out if not def/accl 0
bridge, src=dst 0 routing decision err 0
sanity checks failed 0 temp conn expired 0
fwd to non-pivot 0 broadcast/multicast 0
cluster message 0 partial conn 0
PXL returned F2F 0 cluster forward 782
chain forwarding 0 general reason 0

Re: Latency on Links

Terry — Thu, 17 Jan 2019 14:27:19 GMT

I'm reading that hyper threading on the 12400 series needs to be enabled in the bios by support.

Re: Latency on Links

Timothy_Hall — Fri, 21 Jun 2019 09:12:54 GMT

You have 6 cores. Based on your traffic path distribution I'd recommend that 4 of them be Firewall Workers and 2 be SND/IRQ cores. Doesn't look like the high F2F is caused by IP frags, so no concerns there.

I wouldn't worry about hyperthreading for the moment, I'd focus on getting CoreXL enabled & configured first, run production traffic for a few days, then run the Super Seven again. The only legit reasons you might have CoreXL disabled are:

- Still using the ancient Traditional Mode VPN's (encrypt actions will be present in the rulebase)

- Using route-based VPNs/VTIs (resolved in R80.10 gateway)

- Using QoS blade (resolved in R77.10 gateway)

There are a few other corner cases that require disabling CoreXL, but these are the big ones. See sk61701: CoreXL Known Limitations for the complete list. Since you are using VSX, also please see Kaspars Zibarts great VSX tuning post below; he will be presenting this VSX tuning material at CPX Vienna 2019 right after my presentation: https://community.checkpoint.com/message/29117-security-gateway-performance-optimization-vsx

CheckMates Break Out Sessions Speaker

CPX 2019 Las Vegas & Vienna - Tuesday@13:30

Re: Latency on Links

Terry — Thu, 17 Jan 2019 19:01:58 GMT

Thank you. Considering that this is VSX, would we use the cpconfig method or assign new firewall instances via smart dashboard?

Also, I have been looking for a way to assign addition cores to the SND. Could you point me in the right direction?

Finally, based on your post above, we should be ok to enable corexl. The one thing that confuses me is this output:

Does this mean that 2 of our cores are already in use? eth1-04 is a trunk.

Re: Latency on Links

Kaspars_Zibarts — Thu, 17 Jan 2019 19:46:33 GMT

ok, now i understand why do we see 100% f2f - its traffic originating from vs0 not beeing accelerated, thats totally normal.

can you share fw ctl affinity -l from vs0 context pls?

sounds like your system is underpowered in all honesty if i see 15 virtual systems on 6 cores.. that will be tough

Re: Latency on Links

Terry — Thu, 17 Jan 2019 19:49:54 GMT

A:0> fw ctl affinity -l
Sync: CPU 1
Mgmt: CPU 1
eth1-04: CPU 0
VS_0 fwk: CPU 2 3 4 5
VS_1 fwk: CPU 2 3 4 5
VS_2 fwk: CPU 2 3 4 5
VS_4 fwk: CPU 2 3 4 5
VS_5 fwk: CPU 2 3 4 5
VS_6 fwk: CPU 2 3 4 5
VS_7 fwk: CPU 2 3 4 5
VS_8 fwk: CPU 2 3 4 5
VS_9 fwk: CPU 2 3 4 5
VS_10 fwk: CPU 2 3 4 5
VS_11 fwk: CPU 2 3 4 5
VS_12 fwk: CPU 2 3 4 5
VS_13 fwk: CPU 2 3 4 5
VS_14 fwk: CPU 2 3 4 5
VS_15 fwk: CPU 2 3 4 5
VS_16 fwk: CPU 2 3 4 5
VS_17 fwk: CPU 2 3 4 5
VS_18 fwk: CPU 2 3 4 5
VS_19 fwk: CPU 2 3 4 5
VS_20 fwk: CPU 2 3 4 5
VS_21 fwk: CPU 2 3 4 5
VS_22 fwk: CPU 2 3 4 5
VS_23 fwk: CPU 2 3 4 5
VS_24 fwk: CPU 2 3 4 5

Two of these are firewalls, the others are virtual switches.

Re: Latency on Links

Kaspars_Zibarts — Thu, 17 Jan 2019 19:50:01 GMT

do you have any graphs for CPU usage per core? and total throughput? how many virtual systems are actually configured?

Re: Latency on Links

Kaspars_Zibarts — Thu, 17 Jan 2019 19:59:28 GMT

for 2 virtual firewalls, change to corresponding environment with vsenv and outpout

fw ctl multik stat

fw ctl pstat

fw tab -t connections -s

so theres only one interface thats 10G i assume?

why do you have so many switches?

Re: Latency on Links

Kaspars_Zibarts — Thu, 17 Jan 2019 20:03:22 GMT

if you dont have those, could you share cpview from both virtual firewalls and fwaccel stats -s

Re: Latency on Links

Terry — Thu, 17 Jan 2019 20:15:44 GMT

Sorry, I was trying to find a good view in the monitoring tool are are the other outputs:

VS10:

| CPU: |
| |
| CPU User System Idle I/O wait Interrupts |
| 0 27% 43% 29% 0% 41,511 |
| 1 22% 37% 41% 1% 41,513 |
| 2 19% 26% 55% 0% 41,514 |
| 3 23% 20% 57% 0% 83,031 |
| 4 25% 27% 48% 0% 83,033 |
| 5 20% 26% 53% 0% 41,517 |

[Expert@B:10]# fwaccel stats -s
Accelerated conns/Total conns : 755/44149 (1%)
Delayed conns/(Accelerated conns + PXL conns) : 397/42669 (0%)
Accelerated pkts/Total pkts : 1153863/110514208 (1%)
F2Fed pkts/Total pkts : 5581172/110514208 (5%)
PXL pkts/Total pkts : 103779173/110514208 (93%)
QXL pkts/Total pkts : 0/110514208 (0%)

VS15:

[Expert@B:15]# fwaccel stats -s
Accelerated conns/Total conns : 6664/7360 (90%)
Delayed conns/(Accelerated conns + PXL conns) : 1/6702 (0%)
Accelerated pkts/Total pkts : 9030680127/10905730520 (82%)
F2Fed pkts/Total pkts : 37660303/10905730520 (0%)
PXL pkts/Total pkts : 1837390090/10905730520 (16%)
QXL pkts/Total pkts : 0/10905730520 (0%)

Re: Latency on Links

Terry — Thu, 17 Jan 2019 20:19:04 GMT

VS 10:

[Expert@B:10]# fw ctl multik stat
ID | Active | CPU | Connections | Peak
----------------------------------------------
0 | Yes | 2-5 | 15185 | 33064
1 | Yes | 2-5 | 9421 | 31854
2 | Yes | 2-5 | 9476 | 30371
3 | Yes | 2-5 | 9271 | 33260

[Expert@B:10]# fw ctl pstat

Virtual System Capacity Summary:
Physical memory used: 23% (2235 MB out of 9679 MB) - below watermark
Kernel memory used: 25% (398 MB out of 1587 MB) - below watermark
Virtual memory used: 63% (1266 MB out of 1986 MB) - below watermark
Used: 722 MB by FW, 544 MB by zeco
Concurrent Connections: 42% (42096 out of 99900) - below watermark
Aggressive Aging is disabled

Hash kernel memory (hmem) statistics:
Total memory allocated: 445249932 bytes in 109023 (4084 bytes) blocks
Total memory bytes used: 333974208 unused: 111275724 (24.99%) peak: 527506424
Total memory blocks used: 108243 unused: 780 (0%) peak: 138212
Allocations: 574308181 alloc, 0 failed alloc, 569880037 free

System kernel memory (smem) statistics:
Total memory bytes used: 572596840 peak: 741113564
Total memory bytes wasted: 1662360
Blocking memory bytes used: 449015932 peak: 568585876
Non-Blocking memory bytes used: 123580908 peak: 172527688
Allocations: 5214856 alloc, 0 failed alloc, 5102620 free, 0 failed free
vmalloc bytes used: 0 expensive: no

Kernel memory (kmem) statistics:
Total memory bytes used: 457375052 peak: 1406784900
Allocations: 575986613 alloc, 0 failed alloc
571555656 free, 0 failed free

Cookies:
621606431 total, 0 alloc, 0 free,
1072073 dup, 2838224069 get, 40057397 put,
776547219 len, 1101469 cached len, 0 chain alloc,
0 chain free

Connections:
30983804 total, 20978640 TCP, 9554630 UDP, 408658 ICMP,
41876 other, 323 anticipated, 85822 recovered, 42096 concurrent,
98822 peak concurrent

Fragments:
1059899 fragments, 528442 packets, 257 expired, 0 short,
0 large, 0 duplicates, 0 failures

NAT:
30689449/0 forw, 27549777/0 bckw, 26317983 tcpudp,
522474 icmp, 16759410-56510809 alloc

Sync:
Version: new
Status: Able to Send/Receive sync packets
Sync packets sent:
total : 55006222, retransmitted : 974, retrans reqs : 849, acks : 13868
Sync packets received:
total : 248639624, were queued : 752507, dropped by net : 1350
retrans reqs : 748, received 34452 acks
retrans reqs for illegal seq : 0
dropped updates as a result of sync overload: 0
Callback statistics: handled 33092 cb, average delay : 1, max delay : 4

[Expert@B:10]# fw tab -t connections -s
HOST NAME ID #VALS #PEAK #SLINKS
localhost connections 8158 41830 128549 150211

VS15:

[Expert@B:15]# fw ctl multik stat
fw: CoreXL is disabled

[Expert@B:15]# fw ctl pstat

Virtual System Capacity Summary:
Physical memory used: 23% (2235 MB out of 9679 MB) - below watermark
Kernel memory used: 25% (398 MB out of 1587 MB) - below watermark
Virtual memory used: 34% (675 MB out of 1986 MB) - below watermark
Used: 131 MB by FW, 544 MB by zeco
Concurrent Connections: 2% (7484 out of 349900) - below watermark
Aggressive Aging is disabled

Hash kernel memory (hmem) statistics:
Total memory allocated: 27975400 bytes in 6850 (4084 bytes) blocks
Total memory bytes used: 23057976 unused: 4917424 (17.58%) peak: 40957040
Total memory blocks used: 6610 unused: 240 (3%) peak: 10393
Allocations: 1909400154 alloc, 0 failed alloc, 1909207018 free

System kernel memory (smem) statistics:
Total memory bytes used: 81773640 peak: 97879108
Total memory bytes wasted: 400312
Blocking memory bytes used: 28955788 peak: 43079544
Non-Blocking memory bytes used: 52817852 peak: 54799564
Allocations: 1230850 alloc, 0 failed alloc, 1223630 free, 0 failed free
vmalloc bytes used: 0 expensive: no

Kernel memory (kmem) statistics:
Total memory bytes used: 76612064 peak: 146028360
Allocations: 1909494711 alloc, 0 failed alloc
1909301299 free, 0 failed free

Cookies:
99580802 total, 0 alloc, 0 free,
870 dup, 523653081 get, 17350090 put,
140840608 len, 6666 cached len, 0 chain alloc,
0 chain free

Connections:
3454399 total, 2753386 TCP, 480241 UDP, 175747 ICMP,
45025 other, 5787 anticipated, 231 recovered, 7484 concurrent,
8449 peak concurrent

Fragments:
6664 fragments, 3332 packets, 0 expired, 0 short,
0 large, 0 duplicates, 0 failures

NAT:
8434507/0 forw, 16950416/0 bckw, 16931443 tcpudp,
59932 icmp, 69081-302234 alloc

Sync:
Version: new
Status: Able to Send/Receive sync packets
Sync packets sent:
total : 3948529, retransmitted : 85, retrans reqs : 40, acks : 1869
Sync packets received:
total : 17654161, were queued : 165113, dropped by net : 40
retrans reqs : 83, received 763 acks
retrans reqs for illegal seq : 0
dropped updates as a result of sync overload: 0
Callback statistics: handled 735 cb, average delay : 1, max delay : 8

[Expert@B:15]# fw tab -t connections -s
HOST NAME ID #VALS #PEAK #SLINKS
localhost connections 8158 7412 8449 22073

Re: Latency on Links

Kaspars_Zibarts — Thu, 17 Jan 2019 20:40:10 GMT

great stuff! much better idea about the setup now

VS10 - most traffic taking medium path, 4 cores allocated. System has hit the roof of 100k concurrent connections at some point.

VS15 - most traffic accelerated, corexl disabled, so runing on one core. not too busy by the looks of it - 7k concurrent connentions. i would reduce max thats set to 350k to something more reasonable as it will never handle that much with one core anyways

back to the original question - did you want to move eth1-04 to bond? how much traffic you are pushing through now and what do you want to achieve with the bond? this is 12400 box?

Re: Latency on Links

Terry — Thu, 17 Jan 2019 20:47:31 GMT

We want to bond the interface below. (currently eth1-06)

wrp965 Link encap:Ethernet HWaddr 00:12:C1:83:30:1E
inet addr:omitted Bcast:omitted Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:534457 errors:0 dropped:0 overruns:0 frame:0
TX packets:4349380 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:24415300 (23.2 MiB) TX bytes:944052433 (900.3 MiB)

All of our latency issues are on the VS15 context. We want to bond these interfaces (already setup and ready) and we may need to enable corexl. At this point I'm not sure how to do this since VS10 is already allocated the 4 available cores. What do you think?

This is a 12400 box P220, one card with 8 1 gb interfaces. 12 gb of ram.

Re: Latency on Links

Terry — Thu, 17 Jan 2019 20:48:43 GMT

Also, we have quite a few interfaces on these firewalls which is the reason for the virtual switches I believe.

Re: Latency on Links

Kaspars_Zibarts — Thu, 17 Jan 2019 20:56:52 GMT

Physical interfaces can be attached directly to virtual firewall, you dont need a switch in front of it. Feels like thats what you are doing? Can you run ifconfig | grep Link on VS0, VS10 and VS15? just to confirm topology

Re: Latency on Links

Kaspars_Zibarts — Thu, 17 Jan 2019 20:58:48 GMT

Sorry and sim affinity -l on vs0. going to bed now!

Re: Latency on Links

Kaspars_Zibarts — Thu, 17 Jan 2019 21:05:53 GMT

you will share the same cores as with VS10 since you only have 4 cores. you can enabled it smartdashboard by editing vs oject, can attach screenshots tomorrow