Packet drops and CPU utilization

[RARK22]

Hi All,

Hopefully some one can set me in right direction with checkpoint issue we are having at the moment.

In our two site setup, we got 1*5900 in each site which are used for replicating site to site traffic. Please see attached image for more information on our current setup.

Problem we are having is, site to site throughput at the moment is very bad, on 10Gbps link between sites, I am getting speeds around 1.14Gb/hr which is no way workable for us with amount of data we are trying to replicate from site A to Site B and vice versa.

I'm no checkpoint expert, but one thing I've noticed is when I run command fwaccel stats, I can see below output and what captures my eye is packet drops, not sure what I've to do in order to reduce packet drops and I'm not sure either if below is cause of our throughput problems.

Name Value Name Value
---------------------------- ------------ ---------------------------- ------

Accelerated Path
--------------------------------------------------------------------------------
accel packets 781912298652 accel bytes 2410
outbound packets 236759010965 outbound bytes 2375
conns created 44313019 conns deleted
C total conns 543 C TCP conns
C non TCP conns 434 nat conns
dropped packets 545317964212 dropped bytes 1664
fragments received 1883456 fragments transmit 3
fragments dropped 0 fragments expired
IP options stripped 10 IP options restored
IP options dropped 0 corrs created
corrs deleted 0 C corrections
corrected packets 0 corrected bytes

##########

The other issue I've noticed is with CPU utilization, site A CPU are barely doing anything when we are replicating or backing up data from site A to site B (please see attached screenshot with name cpu for more information), this is a new setup and not sure if there is some sort of configuration issue.

Any help or suggestions would be appreciated.

Thanks

[PhoneBoy]

Super Seven commands output, please?
https://community.checkpoint.com/t5/General-Topics/Super-Seven-Performance-Assessment-Commands-s7pac...

[Timothy_Hall]

Will need Super Seven outputs, but what appears to be happening is that all the VPN traffic is fully accelerated and saturating one SND/IRQ core (#0/8 due to SMT) which is the bottleneck.  Hopefully you are using the AES algorithm in that VPN which is 2-10 times faster then 3DES. 

You probably need more SND/IRQ cores in your CoreXL split, and to enable Multi-Queue.  Might even need to disable SMT if >75% of your traffic is fully accelerated by SecureXL.  Please provide Super Seven outputs for further analysis before changing anything.

[RARK22]

Hi Tim/Phone boy,

Thanks for suggestions here, I have now attached super 7 commands output from both checkpoint devices (Firewall 1 is  in primary site and Firewall2 is in secondary site, bulk of the traffic flow is from primary to secondary).

We have increased our SND/IRQ cores from 2 to 5 and left rest of the CPU's as firewall worker cores.

One question which I got after increasing SND/IRQ cores is, is there any way to allocate two CPU's to one physical NIC? At the moment we just got one dedicated CPU allocated for all of our NICs.

Apart from above change we did not made any further changes, our site to site throughput per hour is around 56 GB/hr on a 10Gbps link 

Thank you.

[Timothy_Hall]

After looking at your Super Seven outputs, the situation is as I suspected initially: a high amount of accelerated traffic on both gateways is saturating individual SND/IRQ cores.  Recommend an 8/8 split on both gateways.  To allocate more than one CPU to a NIC you need to enable Multi-Queue, would recommend enabling it on the eth1-04 interface on FW1, and the eth1-01 interface on FW2 for starters.  When making these adjustments, adjust your CoreXL split first then reboot, then enable Multi-Queue on the recommended interfaces and reboot again.  Do not attempt to adjust the split and enable Multi-Queue all in one go, as that will cause further problems.

[RARK22]

Thank you very much for swift reply Tim, I've attached image with my comments in it, is my understanding correct with what you have proposed?

Thanks

[Timothy_Hall]

Yep you got it.