finetune acceleration on internet fw

[Amir_Arama]

Hi there,

i have pretty low acceleration rate on my internet gw (r80.30 ha cluster). tac and professional services didn't solve this issue and said it's appropriate, but i wonder if there is more that i can do to make it better.

here are some outputs:

[Expert@]# fwaccel stat
+-----------------------------------------------------------------------------+
|Id|Name |Status |Interfaces |Features |
+-----------------------------------------------------------------------------+
|0 |SND |enabled |eth4,eth5,eth0,eth2,eth3 |Acceleration,Cryptography |
| | | | |Crypto: Tunnel,UDPEncap,MD5, |
| | | | |SHA1,NULL,3DES,DES,CAST, |
| | | | |CAST-40,AES-128,AES-256,ESP, |
| | | | |LinkSelection,DynamicVPN, |
| | | | |NatTraversal,AES-XCBC,SHA256 |
+-----------------------------------------------------------------------------+

Accept Templates : enabled
Drop Templates : enabled
NAT Templates : enabled

since the system marked this post as a spam i guess because i posted lots of output, i will try to add other outputs in separate comments.

[Amir_Arama]

[Expert@]# fwaccel stats -s
Accelerated conns/Total conns : 138/8296 (1%)
Accelerated pkts/Total pkts : 18393539360/79399018144 (23%)
F2Fed pkts/Total pkts : 51288084805/79399018144 (64%)
F2V pkts/Total pkts : 198043963/79399018144 (0%)
CPASXL pkts/Total pkts : 13960870/79399018144 (0%)
PSLXL pkts/Total pkts : 9703433109/79399018144 (12%)
QOS inbound pkts/Total pkts : 0/79399018144 (0%)
QOS outbound pkts/Total pkts : 0/79399018144 (0%)
Corrected pkts/Total pkts : 0/79399018144 (0%)

[_Val_]

please share output from "enabled_blades" command

[Amir_Arama]

already shared

[Amir_Arama]

[Expert@]# fwaccel stats -d
Reason Value Reason Value
-------------------- --------------- -------------------- ---------------
General 3 CPASXL Decision 10
PSLXL Decision 1350662 Clear Packet on VPN 0
Encryption Failed 1 Drop Template 0
Decryption Failed 4 Interface Down 0
Cluster Error 0 XMT Error 0
Anti-Spoofing 10523280 Local Spoofing 418
Sanity Error 1189 Monitored Spoofed 0
QXL Decision 0 C2S Violation 0
S2C Violation 0 Loop Prevention 0
DOS Fragments 0 DOS IP Options 0
DOS Blacklists 0 DOS Penalty Box 0
DOS Rate Limiting 0 Syn Attack 0
Reorder 0 Virt Defrag Timeout 9
Invalid Interface 0 Null Routing info 0
Unable to get out ifn 0 Resource exhausted 0
Conn not found 0 Failed to del corr 0
Corr instead of conn 0 Del zombie conn fail 0
FW UUID no match 3 Offload mismatch 0
SIM init failed 0 Null stream init info 0
Unable to get CGNAT 0 Null stream app info 0
Failed get init info 0 SIM add stream failed 0
Collid conn not found 0 Del collid conn fail 0
Add conn after collid 0 SEQ valid 0
Enqueue QoS failed 0 AUX CI null 0
Link dead 0 VPN packet too big 0
NAT64 failed 0 NAT46 failed 0
Packet > MTU 0 NAC validation 0
TCP state violation 916 Enforce packet 0
GTP check packet 0 Bridge route error 0
Route ifn changed 0 IP forwarding 0
Copy MACS failed 0 Fragments Drops 0
Send Notification 0 Conn not found RST 0
Forward to PPAK fail 0 Cluster forward fail 0
F2F before encrypt 0 Forward dst encrypt 0
Correction I/S fail 0 Do inbound F2F 0
Packet UDP failed 0 F2F not allowed 0
Do routing 0 Fanout won't F2F 0
SCTP validation fail 0 SCTP not data 0
Invalid TCP option 0 Invalid MSS option 0
Invalid MSS value 0 Invalid window scale 0

[Amir_Arama]

[Expert@:0]# fw ctl multik stat
ID | Active | CPU | Connections | Peak
----------------------------------------------
0 | Yes | 11 | 3792 | 6840
1 | Yes | 10 | 3682 | 6516
2 | Yes | 9 | 3943 | 6580
3 | Yes | 8 | 3805 | 6590
4 | Yes | 7 | 3777 | 6482
5 | Yes | 6 | 3774 | 6540
6 | Yes | 5 | 3736 | 6548
7 | Yes | 4 | 3501 | 6316
8 | Yes | 3 | 3997 | 6786
9 | Yes | 2 | 3767 | 6599
10 | Yes | 1 | 4052 | 6607

[Expert@]# enabled_blades
fw vpn urlf av appi ips identityServer SSL_INSPECT anti_bot ThreatEmulation mon vpn

[_Val_]

HTTPSi could be the main reason. See if you can tune the policy to avoid over-inspecting. How does your inspection policy look like?

[Amir_Arama]

src_group>internet:https = bypass
src_group>internet:logmein = bypass
all_lans>dst_group:https = bypass
src_group>internet:https:url_group = bypass
all_lans>internet:https:url_group = bypass
all_lans>internet:https = inspect
internet>published services:https = inspect
any>any:https&8080 = bypass

[_Val_]

Ok, no issues here.

[PhoneBoy]

A lot of F2F traffic: are you running in explicit proxy mode by chance and/or you have Remote Access users using Visitor Mode?
It could also just be HTTPS Inspection causing this.

[Amir_Arama]

no proxy, no visitor mode.

only ssl inspection

[PhoneBoy]

I don’t remember if HTTPS Inspection traffic goes F2F or not (may be CPAS).
A PPPoE interface is another possible reason.
See also: https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...

[Amir_Arama]

no pppoe interface

[PhoneBoy]

The SK I pointed you at lists various reasons for traffic going F2F (section 1 primarily but a few others in other sections).
Any of those apply?

[Amir_Arama]

i followed the sk, i didn't found anything special.

besides i do have small pmtu & syn attack protections enabled. and yes i do have critical performance ips protections.

[PhoneBoy]

Small PMTU and SYN Attack will disable Accept Templates, which means initial connections will go F2F, but the actual connections should be accelerated (assuming nothing else pulls it into F2F).
Critical IPS Protections should only trigger F2F in circumstances where traffic might trigger a given protection.
That said, might be worth checking which IPS signatures are having the highest CPU impact: https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...

[Amir_Arama]

thanks ! checked.

[_Val_]

Those are most probably the root causes then. Personally, I would be very surprised if PS missed anything here.

What is the usual situation, FWKs are all running high CPU, or jsut some of them?

[Amir_Arama]

some of them most of the time, but it's dynamic. and something cores get too 100%+ so i don't want to lose any connections that are elephant flow etc

[Timothy_Hall]

HTTPS-Inspected traffic should be in the CPASXL path.

There are a variety of things that can cause high F2F, usually legacy features or signatures that are enabled.  Looking at your enabled blades, I'd say you almost certainly have an IPS signature enabled causing the high F2F.   Suggestions:

1) SYN Attack used to cause large amounts of traffic to go F2F, but that was resolved in R80.20.  Please post output of fwaccel synatk config so we can see if it is properly being handled in SecureXL.  Also provide output of fwaccel stats -p.

2) Do you have any of these IPS signatures enabled (these are quoted from my Max Power book):

- IP ID Masking/Fingerprint Scrambling
- Time to Live (TTL) Masking/Fingerprint Scrambling
- ASCII Only Response Headers
- Network Quota (check out the “Rate Limiting” feature in Chapter 12 for a much
more efficient way to enforce quotas)
- ClusterXL Load Sharing Sticky Decision Function (SDF), which only applies to
Load Sharing Multicast ClusterXL deployments; note that enabling the Mobile
Access Blade forces the use of SDF on a Load Sharing Multicast cluster.

3) Try disabling the IPS checkbox on your gateway and reinstalling policy.  Then run fwaccel stats -r, wait 10 minutes, and run fwaccel stats -s.  Did the F2F % drop a lot?  If so we need to focus on your IPS config. Note that doing this will expose your organization to attacks while IPS is disabled.

4) The next step is labor intensive, and involves running fwaccel conns and fw ctl multik gconn.  Starting in R80.30 connections handled in F2F are no longer listed in the output of fwaccel conns but all connections appear in the output of fw ctl multik gconn.  You should be able to do some crunching and figure out what kind of connections are listed by the latter command but not the former; the attributes of these F2F connections (internal/external IP, port numbers, etc.) should give you some hints about why F2F is necessary.

5) Bit of a long shot, but make sure you do not have wire mode enabled on any of your VPN Communities.  Also do you have a large percentage of protocols traversing the firewall that are not TCP or UDP-based?  All those protocols cannot be accelerated.

[Amir_Arama]

Thank you very much, it seems that ips make a difference, for the 15min i checked anyway.
here are all the outputs you asked for:

[Expert@-M1:0]# fwaccel synatk config
enabled 1
enforce 1
global_high_threshold 10000
periodic_updates 1
cookie_resolution_shift 6
min_frag_sz 80
high_threshold 5000
low_threshold 1000
score_alpha 100
monitor_log_interval (msec) 60000
grace_timeout (msec) 30000
min_time_in_active (msec) 60000

[Expert@-M1:0]# fwaccel stats -p
F2F packets:
--------------
Violation Packets Violation Packets
-------------------- --------------- -------------------- ---------------
pkt has IP options 6613905 ICMP miss conn 89398634
TCP-SYN miss conn 848873200 TCP-other miss conn 52026188649
UDP miss conn 1553600199 other miss conn 9012671
VPN returned F2F 357 uni-directional viol 0
possible spoof viol 39405 TCP state viol 0
SCTP state affecting 0 out if not def/accl 0
bridge, src=dst 0 routing decision err 0
sanity checks failed 0 fwd to non-pivot 0
broadcast/multicast 0 cluster message 501441851
cluster forward 0 chain forwarding 0
F2V conn match pkts 3024697 general reason 0
route changes 0

IP ID Masking/Fingerprint Scrambling = inactive
- Time to Live (TTL) Masking/Fingerprint Scrambling = inactive
- ASCII Only Response Headers = inactive
- Network Quota = inactive
- ClusterXL Load Sharing Sticky Decision Function (SDF) = cluster is in ha mode

#before turning off ips
[Expert@-M1:0]# fwaccel stats -s
Accelerated conns/Total conns : 134/8871 (1%)
Accelerated pkts/Total pkts : 19916797015/84742855952 (23%)
F2Fed pkts/Total pkts : 54179237080/84742855952 (63%)
F2V pkts/Total pkts : 216625665/84742855952 (0%)
CPASXL pkts/Total pkts : 15385824/84742855952 (0%)
PSLXL pkts/Total pkts : 10631436033/84742855952 (12%)
QOS inbound pkts/Total pkts : 0/84742855952 (0%)
QOS outbound pkts/Total pkts : 0/84742855952 (0%)
Corrected pkts/Total pkts : 0/84742855952 (0%)

ips off -n
IPS is disabled
Deleting templates
fwaccel stats -r

#after 15min

[expert@-M1:0]# fwaccel stats -s
Accelerated conns/Total conns : 0/18446744073709551242 (0%)
Accelerated pkts/Total pkts : 18956875/49664511 (38%)
F2Fed pkts/Total pkts : 16862300/49664511 (33%)
F2V pkts/Total pkts : 132737/49664511 (0%)
CPASXL pkts/Total pkts : 9524/49664511 (0%)
PSLXL pkts/Total pkts : 13835812/49664511 (27%)
QOS inbound pkts/Total pkts : 0/49664511 (0%)
QOS outbound pkts/Total pkts : 0/49664511 (0%)
Corrected pkts/Total pkts : 0/49664511 (0%)

EDIT: i checked the status two hours after running ips on (verified it's on) and it was very surprising (f2f still low, and lots of accelerated conns. i wonder if it's reliable to test statistics in short time):

[Expert@-M1:0]# fwaccel stats -s
Accelerated conns/Total conns : 18446744073709551483/18446744073709551536 (166%)
Accelerated pkts/Total pkts : 263188262/769370683 (34%)
F2Fed pkts/Total pkts : 287010534/769370683 (37%)
F2V pkts/Total pkts : 1988718/769370683 (0%)
CPASXL pkts/Total pkts : 176607/769370683 (0%)
PSLXL pkts/Total pkts : 218995280/769370683 (28%)
QOS inbound pkts/Total pkts : 0/769370683 (0%)
QOS outbound pkts/Total pkts : 0/769370683 (0%)
Corrected pkts/Total pkts : 0/769370683 (0%)
[Expert@-M1:0]# ips stat
IPS Status: Enabled
IPS Update Version: 635214613
Global Detect: Off
Bypass Under Load: On
[Expert@-M1:0]#  

for #3 - will do.
for #4 - negative

[Timothy_Hall · Accepted solution]

Bypass Under Load: On

The F2F percentage remains low because IPS is still off, but was disabled by the Bypass under Load feature.  This feature does not work correctly in today's world of multi-core firewalls, as it will disable IPS on all cores even if only one of them is above the high watermark for CPU use due to an elephant flow.  Disable the Bypass under Load feature and the high F2F should return.

SecureXL is properly handling your SYN Flood protection.

You probably need to disable the Small PMTU signature to start with and see how that affects F2F.  Although Phoneboy said it only disables SecureXL templating and not throughput acceleration, I'm not sure if that is correct.

If still high F2F, next you need to examine your IPS protections and sort them by performance impact rating.  Try disabling any IPS protections with a performance impact of Critical.  Do the same thing with Inspection Settings, sort them by performance impact and disable any that are Critical unless you really need them.  This procedure is covered on pages 359-363 of the third edition of my book and should make a big difference.

[Amir_Arama]

Thank you so much