R80.10 VPN Performance 2200 Appliance

[Mike_Jensen]

Hello,

My environment has a Headquarters with a HA pair of 15,400's running R80.10.  We have a DR site with a 2200 appliance and 100Mbps public internet connection and the two sites communicate through a IPSEC VPN.

When the 2200 at the DR site was still on R77.30 and the Headquarters security gateways on R80.10 I was using AES-256 and SHA 384 for the encryption and everything worked fine.

After upgrading the DR site's 2200 to R80.10 I started noticing a nightly backup job that copies data from the Headquarters site to the DR site started taking several more hours than it did before.

I lowered the encryption on the VPN community to AES-128 and SHA 256 and the backup time improved by about an hour, but still is nowhere near as fast as it was when the 2200 was running R77.30.

When I look at the 2200 spec sheet it seems as if this security gateway should be able to handle this throughput.

Is it known that R80.10 will have an impact on performance such as described above?  If so does anyone have any specific details on why?  I think I need to try and convince my leadership to purchase a more powerful security gateway for our DR site.

Thank you.

[Maarten_Sjouw]

R80.10 on a older 2200 (with 2 GB of memory would indeed get you these type of results, try to upgrade the memory or find yourself a 2200B which comes with 4GB.

[Timothy_Hall]

VPN traffic using SHA-384 cannot be accelerated by SecureXL in either R77.30 or R80.10, which is probably the main reason why performance improved when you changed to SHA-256.  The 2200 does not support AES-NI so the main CPU will have to handle all encryption operations.

Depending on what blades you have enabled, it is possible that R80.10 is using more memory and the 2GB in the 2200 is not quite enough.  Please provide output from the following commands for further analysis:

free -m

fwaccel stat

fwaccel stats -s

fw ctl affinity -l -r

enabled_blades

netstat -ni

As a 2-core box, it MIGHT help to disable CoreXL but that will depend heavily on the command outputs provided.  MultiCore IPSec was introduced in R80.10 but may hurt more than help in your 2-core situation, and it is not supported to turn this feature off other than by disabling CoreXL.

[Mike_Jensen]

Hi Tim,

Below is the output from the commands you provided.

[Expert@branson-gw:0]# free -m
total used free shared buffers cached
Mem: 3973 2894 1079 0 297 1625
-/+ buffers/cache: 970 3003
Swap: 10268 0 10268

[Expert@branson-gw:0]# fwaccel stat
Accelerator Status : on
Accept Templates : enabled
Drop Templates : disabled
NAT Templates : disabled by user
NMR Templates : enabled
NMT Templates : enabled

Accelerator Features : Accounting, NAT, Cryptography, Routing,
HasClock, Templates, Synchronous, IdleDetection,
Sequencing, TcpStateDetect, AutoExpire,
DelayedNotif, TcpStateDetectV2, CPLS, McastRouting,
WireMode, DropTemplates, NatTemplates,
Streaming, MultiFW, AntiSpoofing, Nac,
ViolationStats, AsychronicNotif, ERDOS,
McastRoutingV2, NMR, NMT, NAT64, GTPAcceleration,
SCTPAcceleration
Cryptography Features : Tunnel, UDPEncapsulation, MD5, SHA1, NULL,
3DES, DES, CAST, CAST-40, AES-128, AES-256,
ESP, LinkSelection, DynamicVPN, NatTraversal,
EncRouting, AES-XCBC, SHA256

[Expert@branson-gw:0]# fwaccel stats -s
Accelerated conns/Total conns : 7/14 (50%)
Accelerated pkts/Total pkts : 131975820/133215932 (99%)
F2Fed pkts/Total pkts : 1240112/133215932 (0%)
PXL pkts/Total pkts : 0/133215932 (0%)
QXL pkts/Total pkts : 0/133215932 (0%)

[Expert@branson-gw:0]# fw ctl affinity -l -r
CPU 0: eth1
fw_1
CPU 1: eth3
fw_0
All: in.geod mpdaemon vpnd fwd lpd cpd cprid
[Expert@branson-gw:0]# enabled_blades
fw vpn vpn

[Expert@branson-gw:0]# netstat -ni
Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
eth1 1500 0 527238510 2 218 0 239480075 0 0 0 BMRU
eth3 1500 0 235821197 8 103 0 520398811 0 0 0 BMRU
lo 16436 0 639653 0 0 0 639653 0 0 0 LRU
[Expert@branson-gw:0]#

[Timothy_Hall]

Hmm that all looks fine for the moment, will need to see what the CPU utilization looks like on the two individual cores during the slow transfers, want to see if the CPU is getting topped out or if the bottleneck is somewhere else.  Also please run the commands again during a slow transfer along with a continuous 1400-byte ping using the same network path as the slow backup, do you see excessive latency, loss, or both?

Any odd messages in /var/log/messages*?

[Mike_Jensen]

The only thing that catches my eye in the message logs that is repeated over and over is below.  I will rerun the commands I did earlier during the backup job and post the results on here.

Jun 7 00:25:58 2019 branson-gw xpand[4318]: Configuration changed from localhost by user admin by the service dbset
Jun 7 00:25:58 2019 branson-gw xpand[4318]: admin localhost t +installer:packages:Check_Point_R80_10_JUMBO_HF_Bundle_T203_sk116380_FULL.tgz:tag:i

Jun 7 00:25:58 2019 branson-gw xpand[4318]: Configuration changed from localhost by user admin by the service dbset
Jun 7 00:25:58 2019 branson-gw last message repeated 3 times
Jun 7 00:25:58 2019 branson-gw xpand[4318]: admin localhost t +installer:packages::is_blink_image 1
Jun 7 00:25:58 2019 branson-gw xpand[4318]: Configuration changed from localhost by user admin by the service dbset
Jun 7 00:25:58 2019 branson-gw last message repeated 2 times
Jun 7 00:25:58 2019 branson-gw xpand[4318]: admin localhost t +installer:packages:Blink_image_1.0_Check_Point_R80.10_T462_Jumbo_T103.tgz:has_metadata 0

[Timothy_Hall]

I don't think those messages are significant and are just the Deployment Agent (CPUSE) checking for updates, unless they are happening less than 60 seconds apart or so.

[Mike_Jensen]

I was finally able to have my backup operator run the backup in quesition during the day so I can collect output from the commands.  I didn't see any drastic changes.

1400 size pings before backup job started:

Pinging 192.168.50.20 with 1400 bytes of data:
Reply from 192.168.50.20: bytes=1400 time=49ms TTL=252
Reply from 192.168.50.20: bytes=1400 time=46ms TTL=252
Reply from 192.168.50.20: bytes=1400 time=45ms TTL=252
Reply from 192.168.50.20: bytes=1400 time=47ms TTL=252

Pings with the backup job running:

Reply from 192.168.50.20: bytes=1400 time=44ms TTL=252
Reply from 192.168.50.20: bytes=1400 time=49ms TTL=252
Reply from 192.168.50.20: bytes=1400 time=46ms TTL=252
Reply from 192.168.50.20: bytes=1400 time=45ms TTL=252
Reply from 192.168.50.20: bytes=1400 time=46ms TTL=252
Reply from 192.168.50.20: bytes=1400 time=48ms TTL=252
Reply from 192.168.50.20: bytes=1400 time=45ms TTL=252
Reply from 192.168.50.20: bytes=1400 time=46ms TTL=252
Reply from 192.168.50.20: bytes=1400 time=44ms TTL=252
Reply from 192.168.50.20: bytes=1400 time=44ms TTL=252
Reply from 192.168.50.20: bytes=1400 time=47ms TTL=25

Output from these commands taken during the backup job:

Last login: Wed Jun 19 13:46:33 2019 from 165.252.92.194
[Expert@branson-gw:0]# free -m
total used free shared buffers cached
Mem: 3973 3152 821 0 305 1691
-/+ buffers/cache: 1155 2818
Swap: 10268 0 10268
[Expert@branson-gw:0]# fwaccel stat
Accelerator Status : on
Accept Templates : enabled
Drop Templates : disabled
NAT Templates : disabled by user
NMR Templates : enabled
NMT Templates : enabled

Accelerator Features : Accounting, NAT, Cryptography, Routing,
HasClock, Templates, Synchronous, IdleDetection,
Sequencing, TcpStateDetect, AutoExpire,
DelayedNotif, TcpStateDetectV2, CPLS, McastRouting,
WireMode, DropTemplates, NatTemplates,
Streaming, MultiFW, AntiSpoofing, Nac,
ViolationStats, AsychronicNotif, ERDOS,
McastRoutingV2, NMR, NMT, NAT64, GTPAcceleration,
SCTPAcceleration
Cryptography Features : Tunnel, UDPEncapsulation, MD5, SHA1, NULL,
3DES, DES, CAST, CAST-40, AES-128, AES-256,
ESP, LinkSelection, DynamicVPN, NatTraversal,
EncRouting, AES-XCBC, SHA256
[Expert@branson-gw:0]# fwaccel stats -s
Accelerated conns/Total conns : 7/18 (38%)
Accelerated pkts/Total pkts : 482899653/487635361 (99%)
F2Fed pkts/Total pkts : 4735708/487635361 (0%)
PXL pkts/Total pkts : 0/487635361 (0%)
QXL pkts/Total pkts : 0/487635361 (0%)
[Expert@branson-gw:0]# fw ctl affinity -l -r
CPU 0: eth1
fw_1
CPU 1: eth3
fw_0
All: in.geod mpdaemon vpnd fwd lpd cpd cprid
[Expert@branson-gw:0]# enabled_blades
fw vpn vpn
[Expert@branson-gw:0]# netstat -ni
Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
eth1 1500 0 789964382 2 462 0 333223305 0 0 0 BMRU
eth3 1500 0 327578418 9 103 0 780377598 0 0 0 BMRU
lo 16436 0 958723 0 0 0 958723 0 0 0 LRU
[Expert@branson-gw:0]# free -m
total used free shared buffers cached
Mem: 3973 3152 821 0 305 1692
-/+ buffers/cache: 1155 2818
Swap: 10268 0 10268
[Expert@branson-gw:0]# watch -n .5 free -m
[Expert@branson-gw:0]#
[Expert@branson-gw:0]#
[Expert@branson-gw:0]#
[Expert@branson-gw:0]#
[Expert@branson-gw:0]# free -m
total used free shared buffers cached
Mem: 3973 3153 819 0 305 1691
-/+ buffers/cache: 1156 2816
Swap: 10268 0 10268
[Expert@branson-gw:0]#

[Expert@branson-gw:0]# free -m
total used free shared buffers cached
Mem: 3973 3153 820 0 305 1692
-/+ buffers/cache: 1156 2817
Swap: 10268 0 10268
[Expert@branson-gw:0]# fwaccel stat
Accelerator Status : on
Accept Templates : enabled
Drop Templates : disabled
NAT Templates : disabled by user
NMR Templates : enabled
NMT Templates : enabled

Accelerator Features : Accounting, NAT, Cryptography, Routing,
HasClock, Templates, Synchronous, IdleDetection,
Sequencing, TcpStateDetect, AutoExpire,
DelayedNotif, TcpStateDetectV2, CPLS, McastRouting,
WireMode, DropTemplates, NatTemplates,
Streaming, MultiFW, AntiSpoofing, Nac,
ViolationStats, AsychronicNotif, ERDOS,
McastRoutingV2, NMR, NMT, NAT64, GTPAcceleration,
SCTPAcceleration
Cryptography Features : Tunnel, UDPEncapsulation, MD5, SHA1, NULL,
3DES, DES, CAST, CAST-40, AES-128, AES-256,
ESP, LinkSelection, DynamicVPN, NatTraversal,
EncRouting, AES-XCBC, SHA256
[Expert@branson-gw:0]# fwaccel stats -s
Accelerated conns/Total conns : 19/25 (76%)
Accelerated pkts/Total pkts : 484191026/488940159 (99%)
F2Fed pkts/Total pkts : 4749133/488940159 (0%)
PXL pkts/Total pkts : 0/488940159 (0%)
QXL pkts/Total pkts : 0/488940159 (0%)
[Expert@branson-gw:0]# fw ctl affinity -l -r
CPU 0: eth1
fw_1
CPU 1: eth3
fw_0
All: in.geod mpdaemon vpnd fwd lpd cpd cprid
[Expert@branson-gw:0]# netstat -ni
Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
eth1 1500 0 790937965 2 462 0 333565357 0 0 0 BMRU
eth3 1500 0 327913466 9 103 0 781343512 0 0 0 BMRU
lo 16436 0 959113 0 0 0 959113 0 0 0 LRU
[Expert@branson-gw:0]#

[Constantin_Pop]

Try installing the latest Jumbo HF version (sk116380) - there are quite a few VPN related fixes.
As for VPN performance - check sk105119

[Timothy_Hall]

All of that looks good, but you did not include the CPU utilization during that period (you can easily use cpview -t to go back and look at CPU utilization during the backup).  You have a 2/2 split on 2 CPUs, based on your command outputs the only thing on the firewall that can be holding the performance back is a fully-utilized CPU.  Given the high amount of accelerated traffic I don't think disabling CoreXL and going to a 1/1 split on the 2 CPUs will help you here.  However disabling CoreXL will also switch off multicore IPSec which was a change made in R80.10.  Disabling CoreXL might be worth a try, but I don't see how the additional overhead of multicore IPSec would cause that large of a performance drop between R77.30 and R80.10.

So after trying that if one or both CPUs are 100% utilized during the backup there isn't much you can really do to improve things, other that perhaps changing to less CPU-intensive encryption and hashing algorithms for your tunnels which is sounds like you have already done.  It would appear that you are getting all you can get out of your 2200 box.

[Mike_Jensen]

Hi,

I was running and watching cpview during the backup and I would see the CPU's briefly up to ~ 50% and then drop much lower.

With that being said do you still think this is a hardware limitation on the 2200?

[G_W_Albrecht]

I would involve TAC here to completely analyze the issue...

[Timothy_Hall]

Yeah it is not clear what is holding you back if the CPUs are not topping out given what you have provided; the cause of the slowdown may be external to the firewall.  Probably time to involve TAC as there is nothing obviously wrong with your configuration.

[Constantin_Pop]

I've seen such behavior even on virtual appliances and R80.20 version...CPU isn't fully utilized (and I have 4 cores) but the VPN throughout wouldn't go above 500Mbps for TCP but for UDP tests I could get 2 Gbps.

There could be a number of things that aren't working properly, look for MTU, MSS (which was an issue in the latest version, that was not being applied), fragmentation, etc.

Try testing using iperf3 for tcp and udp transfer.

Opening a case is a good idea.