topic Re: finetune acceleration on internet fw in Firewall and Security Management

finetune acceleration on internet fw

Amir_Arama — Sun, 11 Jul 2021 17:34:40 GMT

Hi there,

i have pretty low acceleration rate on my internet gw (r80.30 ha cluster). tac and professional services didn't solve this issue and said it's appropriate, but i wonder if there is more that i can do to make it better.

here are some outputs:

Accept Templates : enabled
Drop Templates : enabled
NAT Templates : enabled

since the system marked this post as a spam i guess because i posted lots of output, i will try to add other outputs in separate comments.

Re: finetune acceleration on internet fw

Amir_Arama — Sun, 11 Jul 2021 13:04:30 GMT

[Expert@]# fwaccel stats -s
Accelerated conns/Total conns : 138/8296 (1%)
Accelerated pkts/Total pkts : 18393539360/79399018144 (23%)
F2Fed pkts/Total pkts : 51288084805/79399018144 (64%)
F2V pkts/Total pkts : 198043963/79399018144 (0%)
CPASXL pkts/Total pkts : 13960870/79399018144 (0%)
PSLXL pkts/Total pkts : 9703433109/79399018144 (12%)
QOS inbound pkts/Total pkts : 0/79399018144 (0%)
QOS outbound pkts/Total pkts : 0/79399018144 (0%)
Corrected pkts/Total pkts : 0/79399018144 (0%)

Re: finetune acceleration on internet fw

Amir_Arama — Sun, 11 Jul 2021 13:04:53 GMT

[Expert@]# fwaccel stats -d
Reason Value Reason Value
-------------------- --------------- -------------------- ---------------
General 3 CPASXL Decision 10
PSLXL Decision 1350662 Clear Packet on VPN 0
Encryption Failed 1 Drop Template 0
Decryption Failed 4 Interface Down 0
Cluster Error 0 XMT Error 0
Anti-Spoofing 10523280 Local Spoofing 418
Sanity Error 1189 Monitored Spoofed 0
QXL Decision 0 C2S Violation 0
S2C Violation 0 Loop Prevention 0
DOS Fragments 0 DOS IP Options 0
DOS Blacklists 0 DOS Penalty Box 0
DOS Rate Limiting 0 Syn Attack 0
Reorder 0 Virt Defrag Timeout 9
Invalid Interface 0 Null Routing info 0
Unable to get out ifn 0 Resource exhausted 0
Conn not found 0 Failed to del corr 0
Corr instead of conn 0 Del zombie conn fail 0
FW UUID no match 3 Offload mismatch 0
SIM init failed 0 Null stream init info 0
Unable to get CGNAT 0 Null stream app info 0
Failed get init info 0 SIM add stream failed 0
Collid conn not found 0 Del collid conn fail 0
Add conn after collid 0 SEQ valid 0
Enqueue QoS failed 0 AUX CI null 0
Link dead 0 VPN packet too big 0
NAT64 failed 0 NAT46 failed 0
Packet > MTU 0 NAC validation 0
TCP state violation 916 Enforce packet 0
GTP check packet 0 Bridge route error 0
Route ifn changed 0 IP forwarding 0
Copy MACS failed 0 Fragments Drops 0
Send Notification 0 Conn not found RST 0
Forward to PPAK fail 0 Cluster forward fail 0
F2F before encrypt 0 Forward dst encrypt 0
Correction I/S fail 0 Do inbound F2F 0
Packet UDP failed 0 F2F not allowed 0
Do routing 0 Fanout won't F2F 0
SCTP validation fail 0 SCTP not data 0
Invalid TCP option 0 Invalid MSS option 0
Invalid MSS value 0 Invalid window scale 0

Re: finetune acceleration on internet fw

Amir_Arama — Sun, 11 Jul 2021 13:05:15 GMT

[Expert@:0]# fw ctl multik stat
ID | Active | CPU | Connections | Peak
----------------------------------------------
0 | Yes | 11 | 3792 | 6840
1 | Yes | 10 | 3682 | 6516
2 | Yes | 9 | 3943 | 6580
3 | Yes | 8 | 3805 | 6590
4 | Yes | 7 | 3777 | 6482
5 | Yes | 6 | 3774 | 6540
6 | Yes | 5 | 3736 | 6548
7 | Yes | 4 | 3501 | 6316
8 | Yes | 3 | 3997 | 6786
9 | Yes | 2 | 3767 | 6599
10 | Yes | 1 | 4052 | 6607

[Expert@]# enabled_blades
fw vpn urlf av appi ips identityServer SSL_INSPECT anti_bot ThreatEmulation mon vpn

Re: finetune acceleration on internet fw

PhoneBoy — Sun, 11 Jul 2021 17:32:02 GMT

A lot of F2F traffic: are you running in explicit proxy mode by chance and/or you have Remote Access users using Visitor Mode?
It could also just be HTTPS Inspection causing this.

Re: finetune acceleration on internet fw

Amir_Arama — Sun, 11 Jul 2021 18:44:17 GMT

no proxy, no visitor mode.

only ssl inspection

Re: finetune acceleration on internet fw

PhoneBoy — Sun, 11 Jul 2021 19:38:34 GMT

I don’t remember if HTTPS Inspection traffic goes F2F or not (may be CPAS).
A PPPoE interface is another possible reason.
See also: https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk32578

Re: finetune acceleration on internet fw

Amir_Arama — Sun, 11 Jul 2021 21:01:17 GMT

no pppoe interface

Re: finetune acceleration on internet fw

PhoneBoy — Mon, 12 Jul 2021 00:21:53 GMT

The SK I pointed you at lists various reasons for traffic going F2F (section 1 primarily but a few others in other sections).
Any of those apply?

Re: finetune acceleration on internet fw

_Val_ — Mon, 12 Jul 2021 05:51:24 GMT

please share output from "enabled_blades" command

Re: finetune acceleration on internet fw

_Val_ — Mon, 12 Jul 2021 05:57:10 GMT

HTTPSi could be the main reason. See if you can tune the policy to avoid over-inspecting. How does your inspection policy look like?

Re: finetune acceleration on internet fw

Amir_Arama — Mon, 12 Jul 2021 06:36:27 GMT

already shared

Re: finetune acceleration on internet fw

Amir_Arama — Mon, 12 Jul 2021 07:05:46 GMT

i followed the sk, i didn't found anything special.

besides i do have small pmtu & syn attack protections enabled. and yes i do have critical performance ips protections.

Re: finetune acceleration on internet fw

Amir_Arama — Mon, 12 Jul 2021 07:18:39 GMT

src_group>internet:https = bypass
src_group>internet:logmein = bypass
all_lans>dst_group:https = bypass
src_group>internet:https:url_group = bypass
all_lans>internet:https:url_group = bypass
all_lans>internet:https = inspect
internet>published services:https = inspect
any>any:https&8080 = bypass

Re: finetune acceleration on internet fw

_Val_ — Mon, 12 Jul 2021 07:24:20 GMT

Ok, no issues here.

Re: finetune acceleration on internet fw

PhoneBoy — Mon, 12 Jul 2021 07:25:29 GMT

Small PMTU and SYN Attack will disable Accept Templates, which means initial connections will go F2F, but the actual connections should be accelerated (assuming nothing else pulls it into F2F).
Critical IPS Protections should only trigger F2F in circumstances where traffic might trigger a given protection.
That said, might be worth checking which IPS signatures are having the highest CPU impact: https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk110737&partition=Basic&product=IPS

Re: finetune acceleration on internet fw

_Val_ — Mon, 12 Jul 2021 07:26:01 GMT

Those are most probably the root causes then. Personally, I would be very surprised if PS missed anything here.

What is the usual situation, FWKs are all running high CPU, or jsut some of them?

Re: finetune acceleration on internet fw

Timothy_Hall — Mon, 12 Jul 2021 19:08:57 GMT

HTTPS-Inspected traffic should be in the CPASXL path.

There are a variety of things that can cause high F2F, usually legacy features or signatures that are enabled. Looking at your enabled blades, I'd say you almost certainly have an IPS signature enabled causing the high F2F. Suggestions:

1) SYN Attack used to cause large amounts of traffic to go F2F, but that was resolved in R80.20. Please post output of fwaccel synatk config so we can see if it is properly being handled in SecureXL. Also provide output of fwaccel stats -p.

2) Do you have any of these IPS signatures enabled (these are quoted from my Max Power book):

- IP ID Masking/Fingerprint Scrambling
- Time to Live (TTL) Masking/Fingerprint Scrambling
- ASCII Only Response Headers
- Network Quota (check out the “Rate Limiting” feature in Chapter 12 for a much
more efficient way to enforce quotas)
- ClusterXL Load Sharing Sticky Decision Function (SDF), which only applies to
Load Sharing Multicast ClusterXL deployments; note that enabling the Mobile
Access Blade forces the use of SDF on a Load Sharing Multicast cluster.

3) Try disabling the IPS checkbox on your gateway and reinstalling policy. Then run fwaccel stats -r, wait 10 minutes, and run fwaccel stats -s. Did the F2F % drop a lot? If so we need to focus on your IPS config. Note that doing this will expose your organization to attacks while IPS is disabled.

4) The next step is labor intensive, and involves running fwaccel conns and fw ctl multik gconn. Starting in R80.30 connections handled in F2F are no longer listed in the output of fwaccel conns but all connections appear in the output of fw ctl multik gconn. You should be able to do some crunching and figure out what kind of connections are listed by the latter command but not the former; the attributes of these F2F connections (internal/external IP, port numbers, etc.) should give you some hints about why F2F is necessary.

5) Bit of a long shot, but make sure you do not have wire mode enabled on any of your VPN Communities. Also do you have a large percentage of protocols traversing the firewall that are not TCP or UDP-based? All those protocols cannot be accelerated.

Re: finetune acceleration on internet fw

Amir_Arama — Mon, 12 Jul 2021 16:12:42 GMT

some of them most of the time, but it's dynamic. and something cores get too 100%+ so i don't want to lose any connections that are elephant flow etc

Re: finetune acceleration on internet fw

Amir_Arama — Mon, 12 Jul 2021 16:13:15 GMT

thanks ! checked.