Low throughput from 4200 appliance

[exciteman]

We have a CheckPoint 4200 appliance running as our gateway/firewall. Our WAN speed is 1Gbps, but we can only seem to get 100Mbps throughput from the appliance.

I have connected a computer directly to our WAN-connection to confirm WAN speed, and without going through the firewall i get the correct speed (1Gbps). 

The WAN interface (eth1) says "Link Speed: 1000Mbps / Full Duplex".

I have been monitoring with CPview on the firewall, and I have not seen "Total Mbits/sec" go above 102 Mbps. To me it seems like speed is capped at 100Mbps. I am wondering what the cause of this can be, and what steps should I do to troubleshoot this issue? Appreciate any help. 

[G_W_Albrecht]

Try from different clients at the same time - and add up the throughputs...

[exciteman]

I have, and the speed still doesn't exceed 100Mbps.

[FedericoMeiners]

Hello,

• Which version are you running? 
• Which blades are you running?
• Please post output of fwaccel stats -s (or stat, don't remember right now)
• From expert (#) run ifconfig -a . Do you see errors on the interfaces? 
• Output from top
• Did you tried to connect your host directly to the firewall to perform the speed test?

Regards,

[exciteman]

Hello,

Q: Which version are you running? 

A: R77.30.

Q: Which blades are you running?

A: 

Q: Please post output of fwaccel stats -s (or stat, don't remember right now)

A:

Click to Expand

Accelerator Status : on
Accept Templates : disabled by Firewall
disabled from rule #12
Drop Templates : disabled
NAT Templates : disabled by user

Accelerator Features : Accounting, NAT, Cryptography, Routing,
HasClock, Templates, Synchronous, IdleDetection,
Sequencing, TcpStateDetect, AutoExpire,
DelayedNotif, TcpStateDetectV2, CPLS, McastRouting,
WireMode, DropTemplates, NatTemplates,
Streaming, MultiFW, AntiSpoofing, Nac,
ViolationStats, AsychronicNotif, ERDOS,
NAT64, GTPAcceleration, SCTPAcceleration,
McastRoutingV2
Cryptography Features : Tunnel, UDPEncapsulation, MD5, SHA1, NULL,
3DES, DES, CAST, CAST-40, AES-128, AES-256,
ESP, LinkSelection, DynamicVPN, NatTraversal,
EncRouting, AES-XCBC, SHA256

Q: From expert (#) run ifconfig -a . Do you see errors on the interfaces? 

A: I only see 22 errors on the trunked interface...

Q: Did you tried to connect your host directly to the firewall to perform the speed test?

A: No.

Regards.

[G_W_Albrecht]

Q: Which version are you running? 

A: R77.30.

--> having support for eleven more days, so what about the future ?

[Timothy_Hall]

The maximum speed through a 2core box like 4200 will depend on which blades are enabled (enabled_blades command), and how much traffic is being pulled into the PXL or F2F paths based on your APCL/URLF and Threat Prevention policies.  Please provide the output from the "Super Seven" commands run on your firewall for further analysis:

https://community.checkpoint.com/t5/General-Topics/Super-Seven-Performance-Assessment-Commands-s7pac...

[exciteman]

Hello!

Super Seven output:

fwaccel stat

Click to Expand

Accelerator Status : on
Accept Templates : disabled by Firewall
disabled from rule #12
Drop Templates : disabled
NAT Templates : disabled by user

Accelerator Features : Accounting, NAT, Cryptography, Routing,
HasClock, Templates, Synchronous, IdleDetection,
Sequencing, TcpStateDetect, AutoExpire,
DelayedNotif, TcpStateDetectV2, CPLS, McastRouting,
WireMode, DropTemplates, NatTemplates,
Streaming, MultiFW, AntiSpoofing, Nac,
ViolationStats, AsychronicNotif, ERDOS,
NAT64, GTPAcceleration, SCTPAcceleration,
McastRoutingV2


Cryptography Features : Tunnel, UDPEncapsulation, MD5, SHA1, NULL,
3DES, DES, CAST, CAST-40, AES-128, AES-256,
ESP, LinkSelection, DynamicVPN, NatTraversal,
EncRouting, AES-XCBC, SHA256

fwaccel stats -s

Click to Expand

Accelerated conns/Total conns : 0/607 (0%)
Accelerated pkts/Total pkts : 40/4009649 (0%)
F2Fed pkts/Total pkts : 216182/4009649 (5%)
PXL pkts/Total pkts : 3793427/4009649 (94%)
QXL pkts/Total pkts : 0/4009649 (0%)

grep -c ^processor /proc/cpuinfo

Click to Expand

2

fw ctl affinity -l -r

Click to Expand

CPU 0: eth2 eth3
fw_1
CPU 1: eth1 Mgmt
fw_0
All: usrchkd mpdaemon in.acapd vpnd lpd rad fwd in.msd fwpushd cprid cpd

netstat -ni

Click to Expand

Kernel Interface table

Iface	MTU	Met	RX-OK	RX-ERR	RX-DRP	RX-OVR	TX-OK	TX-ERR	TX-DRP	TX-OVR	Flg
Mgmt	1500	0	352341	0	0	0	315754	0	0	0	BMRU
eth1	1500	0	195704407	0	3154326	0	102409603	0	0	0	BMRU
eth2	1500	0	251686	0	1549	0	150148	0	0	0	BMRU
eth3	1500	0	104346397	23	385319	0	189718282	0	0	0	BMRU
eth3.3	1500	0	390303	0	0	0	46120	0	0	0	BMRU
eth3.5	1500	0	97345634	0	0	0	181543112	0	0	0	BMRU
eth3.6	1500	0	5467821	0	0	0	8966394	0	0	0	BMRU
eth3.7	1500	0	0	0	0	0	0	0	0	0	BMRU
eth3.9	1500	0	79155	0	0	0	1099	0	0	0	BMRU
eth3.10	1500	0	473634	0	0	0	260472	0	0	0	BMRU
eth3.15	1500	0	81049	0	0	0	8108	0	0	0	BMRU
eth3.20	1500	0	508709	0	0	0	229251	0	0	0	BMRU
lo	16436	0	1101289	0	0	0	1101289	0	0	0	LRU

fw ctl multik stat

Click to Expand

ID | Active | CPU | Connections | Peak
----------------------------------------------
0 | Yes | 1 | 271 | 3596
1 | Yes | 0 | 368 | 3657

cpstat os -f multi_cpu -o 1

Click to Expand

Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1| 1| 79| 20| 80| ?| 2183|
| 2| 4| 62| 34| 66| ?| 2183|
---------------------------------------------------------------------------------

Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1| 1| 79| 20| 80| ?| 2183|
| 2| 4| 62| 34| 66| ?| 2183|
---------------------------------------------------------------------------------

Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1| 2| 85| 13| 87| ?| 2272|
| 2| 17| 51| 32| 68| ?| 2272|
---------------------------------------------------------------------------------

Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1| 2| 85| 13| 87| ?| 2272|
| 2| 17| 51| 32| 68| ?| 2272|
---------------------------------------------------------------------------------

Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1| 0| 83| 16| 84| ?| 2235|
| 2| 11| 43| 47| 53| ?| 2235|
---------------------------------------------------------------------------------

Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1| 0| 83| 16| 84| ?| 2235|
| 2| 11| 43| 47| 53| ?| 2235|
---------------------------------------------------------------------------------

Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1| 0| 85| 14| 86| ?| 2254|
| 2| 1| 37| 63| 37| ?| 2254|
---------------------------------------------------------------------------------

Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1| 0| 85| 14| 86| ?| 2254|
| 2| 1| 37| 63| 37| ?| 2254|
---------------------------------------------------------------------------------

Cheers.

[Timothy_Hall]

You are getting frame loss (RX-DRP) rates of between 0.3% and 1.6% on your interfaces due to buffering misses which is probably the main thing slowing you down.  This is almost certainly due to high CPU load on your 2 cores, given the large number blades you have enabled on an old 2-core box like that 4200, 100Mbps top throughput doesn't seem that unreasonable to me.  Currently you have a 2/2 CoreXL split on your box, in some cases disabling CoreXL and going to a 1/1 split helps on a 2-core box but given your high PXL% I don't think doing that will help in this case.

The 4200 only has 4GB of RAM which may not be enough for all you are trying to do.  Please provide output of the free -m command to see if a memory upgrade will help.

You can probably pick up some more speed by tuning your policies, the two major areas in your case are Threat Prevention and APCL/URLF. In order to figure out where to focus your efforts, try this and report back what you see:

1) Run Internet speed test and note throughput

2) On the gateway from expert mode run commands ips off and fw amw unload

3) Wait 60 seconds

4) From a completely new browser instance run an Internet speed test and note throughput.  If throughput has substantially increased you need to tune your IPS & Threat Prevention configuration.

5) Run commands ips on and fw amw fetch local

6) Wait 60 seconds

7) From a completely new browser instance run an Internet speed test and note throughput. (should be about the same as #1)

😎 On gateway object in SmartConsole, uncheck the APCL and URLF blades and reinstall policy to the gateway.

9) Wait 60 seconds

9) From a completely new browser instance run an Internet speed test and note throughput. If throughput has substantially increased you need to tune your APCL/URLF policy, typically this will involve removing the "Any Any Any Accept" rule at the bottom of your APCL/URLF policy (which is not necessary except for logging purposes), and making sure you are using object "Internet" in the Destination column of all APCL/URLF rules and NOT "Any".

10) Recheck the APCL and URLF checkboxes and reinstall policy to the gateway.

11) From a completely new browser instance run an Internet speed test and note throughput. (should be about the same as #1)

Let us know what you find out.

[exciteman]

Thanks for the input!

I’ve done the steps you suggested, and I found this:

free -m command:

	total	used	free	shared	buffers	cached
Mem:	3973	3289	684	0	34	834
-/+ buffers/cache:		2420	1553
Swap:	10268	0	10268

Throughput tests (peaks - CPview):

Without any changes: 74 Mbps

ips off & fw amw unload: 234 Mbps

Reverted (ips on & fw amw fetch local) 93 Mbps

APCL & URLF blades disabled: 115 Mbps

Reverted (APCL & URLF enabled) 95 Mbps

So it seems like the IPS & Threat Prevention needs tuning. Do you have any suggestions for that?

I will do your suggested tuning for APCL/URLF also.

[Timothy_Hall]

Looks like your box is not hitting swap at all which is good, no memory upgrade needed.

We'll need to do a few more tests to determine whether it is IPS specifically (more likely) or the rest of Threat Prevention (less likely) that is causing the bulk of the slowdown:

1) Run Internet speed test and note throughput

2) On the gateway from expert mode run commands ips off

3) Wait 60 seconds

4) From a completely new browser instance run an Internet speed test and note throughput.  If throughput has substantially increased you need to tune your IPS configuration.

5) Run command ips on

6) Run command fw amw unload

7) Wait 60 seconds

😎 From a completely new browser instance run an Internet speed test and note throughput.  If throughput has substantially increased you need to tune your TP (AV/ABOT) configuration.

9) Run command fw amw fetch local

You may well see a performance improvement at both steps #4 & #8, I'd suggest focusing on where you get the biggest increase for tuning.  If turning off IPS provides most of the gain, determine which IPS profile is in use by your 4200 gateway and open it for editing.  Sort the IPS protections by the "Performance Impact" rating and disable all IPS Protections with a "Critical" or "High" rating.  That should help a lot.

If turning off Threat Prevention (amw) provided most of the gain, my guess is that Anti-virus is causing most of the overhead as Anti-bot tends to be pretty low impact.  I'll need to see the AV & ABOT settings in the relevant TP profile applied to your gateway to make specific recommendations.