Finding root cause for all the F2F traffic

[David_Chau]

69% of my traffic is F2F.  I am not running IPS or HTTPS inspection.  The majority of the traffic is VPN traffic and currently IP compression is not enabled in the community.  I am currently running VSX.  Any ideas?  Below is some output.

[Expert@fwoc2:3]# fwaccel stats -s
Accelerated conns/Total conns : 93/302 (30%)
Accelerated pkts/Total pkts : 2709202/8804447 (30%)
F2Fed pkts/Total pkts : 6095245/8804447 (69%)
PXL pkts/Total pkts : 0/8804447 (0%)
QXL pkts/Total pkts : 0/8804447 (0%)

[Expert@fwoc2:3]# fw ver
This is Check Point's software version R77.30 - Build 161
[Expert@fwoc2:3]# free -m
total used free shared buffers cached
Mem: 64206 6783 57423 0 1053 2323
-/+ buffers/cache: 3407 60799
Swap: 18449 0 18449
[Expert@fwoc2:3]# netstat -ni
Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
lo3 16436 0 3528958 0 0 0 3528958 0 0 0 LRU
wrp192 1500 0 9080691 0 0 0 237149 0 0 0 BMRU
wrp193 1500 0 826888 0 0 0 1641614 0 0 0 BMRU
wrp194 1500 0 649853 0 0 0 2264704 0 0 0 BMRU
wrp195 1500 0 19642 0 0 0 568489 0 0 0 BMRU
wrp196 1500 0 47075 0 0 0 580195 0 0 0 BMRU
[Expert@fwoc2:3]# fwaccel stat
Accelerator Status : on
Accept Templates : enabled
Drop Templates : disabled
NAT Templates : disabled by user

Accelerator Features : Accounting, NAT, Cryptography, Routing,
HasClock, Templates, Synchronous, IdleDetection,
Sequencing, TcpStateDetect, AutoExpire,
DelayedNotif, TcpStateDetectV2, CPLS, McastRouting,
WireMode, DropTemplates, NatTemplates,
Streaming, MultiFW, AntiSpoofing, Nac,
ViolationStats, AsychronicNotif, ERDOS,
NAT64, GTPAcceleration, SCTPAcceleration,
McastRoutingV2
Cryptography Features : Tunnel, UDPEncapsulation, MD5, SHA1, NULL,
3DES, DES, CAST, CAST-40, AES-128, AES-256,
ESP, LinkSelection, DynamicVPN, NatTraversal,
EncRouting, AES-XCBC, SHA256
[Expert@fwoc2:3]# fw ctl multik stat
fw: CoreXL is disabled

[Expert@fwoc2:0]# fw ctl affinity -l -r
CPU 0: Mgmt Sync
CPU 1: eth3-01 eth3-02
CPU 2:
CPU 3:
CPU 4:
CPU 5:
CPU 6:
CPU 7:
CPU 8:
CPU 9:
CPU 10:
CPU 11:
CPU 12:
CPU 13:
CPU 14:
CPU 15:
CPU 16:
CPU 17:
CPU 18:
CPU 19:
All:

[Daniel_Taney]

Can we see the output of: enabled_blades

Also, do you know why CoreXL is disabled on a Gateway with 20 cores?

[David_Chau]

I don't think we ever enabled CoreXL. Not sure if we needed that at the moment since our utilization is pretty low.
Below is output for enabled_blades and top
[Expert@fwoc2:3]# enabled_blades
fw vpn

top - 11:27:39 up 1 day, 12:43, 1 user, load average: 0.56, 0.31, 0.29
Tasks: 439 total, 1 running, 438 sleeping, 0 stopped, 0 zombie
Cpu0 : 0.5%us, 0.8%sy, 0.0%ni, 97.9%id, 0.6%wa, 0.0%hi, 0.2%si, 0.0%st
Cpu1 : 1.3%us, 1.3%sy, 0.0%ni, 96.6%id, 0.0%wa, 0.0%hi, 0.7%si, 0.0%st
Cpu2 : 0.8%us, 0.9%sy, 0.0%ni, 98.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu3 : 0.6%us, 0.6%sy, 0.0%ni, 98.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu4 : 0.5%us, 0.5%sy, 0.0%ni, 99.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu5 : 0.6%us, 0.4%sy, 0.0%ni, 99.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu6 : 0.6%us, 0.4%sy, 0.0%ni, 98.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu7 : 0.6%us, 0.4%sy, 0.0%ni, 99.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu8 : 0.5%us, 0.4%sy, 0.0%ni, 99.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu9 : 0.4%us, 0.3%sy, 0.0%ni, 99.2%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu10 : 0.7%us, 0.6%sy, 0.0%ni, 98.6%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu11 : 1.6%us, 1.5%sy, 0.0%ni, 96.9%id, 0.0%wa, 0.0%hi, 0.1%si, 0.0%st
Cpu12 : 0.9%us, 1.0%sy, 0.0%ni, 98.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu13 : 0.7%us, 0.6%sy, 0.0%ni, 98.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu14 : 0.6%us, 0.4%sy, 0.0%ni, 99.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu15 : 0.4%us, 0.3%sy, 0.0%ni, 99.2%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu16 : 0.5%us, 0.3%sy, 0.0%ni, 99.2%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu17 : 0.5%us, 0.2%sy, 0.0%ni, 99.2%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu18 : 0.5%us, 0.2%sy, 0.0%ni, 99.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu19 : 0.3%us, 0.2%sy, 0.0%ni, 99.4%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 65747876k total, 6954900k used, 58792976k free, 1080828k buffers
Swap: 18892344k total, 0k used, 18892344k free, 2382616k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6872 admin 15 0 242m 64m 25m S 14 0.1 113:57.89 pdpd
3151 admin 0 -20 627m 88m 22m S 5 0.1 32:39.64 fwk4_dev
23237 nobody 16 0 7508 2204 1764 S 4 0.0 2:51.30 wmic
4186 admin 16 0 173m 36m 25m S 2 0.1 0:34.54 cpd
27283 admin 15 0 2372 1304 832 R 2 0.0 0:00.01 top
1 admin 15 0 1972 720 624 S 0 0.0 0:03.15 init
2 admin RT -5 0 0 0 S 0 0.0 0:02.65 migration/0
3 admin 15 0 0 0 0 S 0 0.0 0:00.00 ksoftirqd/0
4 admin RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/0
5 admin RT -5 0 0 0 S 0 0.0 0:00.91 migration/1

[HeikoAnkenbrand · Accepted solution]

The medium path is not available without CoreXL. Thus traffic is forwarded to the F2F path, because CoreXL is deactivated.

My recommendations:

- First step enable CoreXL on a 20 core system.

- Second step optimize your rules for accept templates. 

- Afterwards eneble drop and nat templates

- Optimize affinity on VSX

- Update your firewall to R80.10+ for VPN multicore support

Some more information to your questions:

Accept Template - Feature that accelerates the speed, at which a connection is established by matching a new connection to a set of attributes. When a new connection matches the Accept Template, subsequent connections are established without performing a rule match and therefore are accelerated. Accept Templates are generated from active connections according to policy rules. Currently, Accept Template acceleration is performed only on connections with the same destination port (using wildcards for source ports).

Accept Tamplate is enabled by default if SecureXL is used.

Throughput Acceleration - The first packets of a new TCP connection require more processing when processed by the firewall module. If the connection is eligible for acceleration, after minimal security processing the packet is offloaded to the SecureXL device associated with the proper egress interface. Subsequent packets of the connection can be processed on the accelerated path and directly sent from the inbound to the outbound interface via the SecureXL device.

Connection Rate Acceleration  SecureXL also improves the rate of new connections (connections per second) and the connection setup/teardown rate (sessions per second). To accelerate the rate of new connections, connections that do not match a specified 5 tuple are still processed by SecureXL. For example, if the source port is masked and only the other 4 tuple attributes require a match. When a connection is processed on the accelerated path, SecureXL creates a template of that connection that does not include the source port tuple. A new connection that matches the other 4 tuples is processed on the accelerated path because it matches the template. The firewall module does not inspect the new connection, increasing firewall connection rates.

[HeikoAnkenbrand]

Has nothing to do with your problem, but you can also enable drop and nat templates.

Drop Template - Feature that accelerates the speed, at which a connection is dropped by matching a new connection to a set of attributes. When a new connection matches the Drop Template, subsequent connections are dropped without performing a rule match and therefore are accelerated. Currently, Drop Template acceleration is performed only on connections with the same destination port (does not use wildcards for source ports).
Drop Template is disabled by default if SecureXL is used. It can be activated via smart Dashboard and does not require a reboot of the firewall.

NAT Templates - Using SecureXL Templates for NAT traffic is critical to achieve high session rate for NAT. SecureXL Templates are supported for Static NAT and Hide NAT using the existing SecureXL Templates mechanism. Normally the first packet would use the F2F path. However, if SecureXL is used, the first packet will not be forwarded to the F2F path if Accept Tamplate and NAT Template match. Enabling or disabling of NAT Templates requires a firewall reboot.

[HeikoAnkenbrand]

More you can found here in my articles:

R80.x Security Gateway Architecture (Logical Packet Flow)
R80.x Security Gateway Architecture (Content Inspection) 

Or in this Check Point SK's:

SecureXL Mechanism
SecureKnowledge: SecureXL SecureKnowledge: NAT Templates 
SecureKnowledge: VPN Core 
SecureKnowledge: CoreXL 
SecureKnowledge: CoreXL Dynamic Dispatcher in R77.30 / R80.10 and above 
SecureKnowledge: Best Practices - Security Gateway Performance

If you want to update you can use these documents:
Performance Tuning R80.10 Administratio Guide
Performance Tuning R80.20 Administration Guide

[Wolfgang]

David,

your shown utilization is realy poor. Maybe all of your traffic can‘t be accelerated. Have a look at SecureXL Mechanism chapter (1) Acceleration of packets . There is a good description which traffic isn‘t accelerated.

And too I recommend to enable CoreXL on a 20 core system with VSX. At the moment you don‘t need the cores but later with more traffic I‘m sure you need 😉

best regards

Wolfgang Becher

[Daniel_Taney]

I'll be honest... I don't think I've seen a Check Point deployment where CoreXL would be off but SecureXL on. I think the best place to start would be to get CoreXL fired up and set correctly. The default configuration for a 20-core Gateway would be 2 SND's and 18 FWK's. It seems to me the first step is to analyzing performance would be to get this enabled. 

Can you also run fwaccel stats -p to display the F2F Packets / SecureXL violations?

[Daniel_Taney]

I also missed that this was VSX in your OP. How many VS's run on this GW or cluster? 

It looks like the SecureXL stats you provided were from vs3. Have you looked at those performance numbers for each VS on the system? Are you seeing mostly F2F on everything? 

[David_Chau]

I am running two active VSs and 2 VSs in standby. Here is the output for the other active VS:

[Expert@fwoc2:4]# fwaccel stats -s
Accelerated conns/Total conns : 215/267 (80%)
Accelerated pkts/Total pkts : 342488/546064 (62%)
F2Fed pkts/Total pkts : 196019/546064 (35%)
PXL pkts/Total pkts : 7557/546064 (1%)
QXL pkts/Total pkts : 0/546064 (0%)
[Expert@fwoc2:4]# vsenv 3
Context is set to Virtual Device fwoc2_atm-branch (ID 3).

[Daniel_Taney]

Ok, so this looks like it is really just the one VS that is affected. This one has a much lower rate of F2F connections!

[David_Chau]

here you go:

[Expert@fwoc2:3]# fwaccel stats -p
F2F packets:
--------------
Violation Packets Violation Packets
-------------------- --------------- -------------------- ---------------
pkt is a fragment 1476 pkt has IP options 22
ICMP miss conn 1070 TCP-SYN miss conn 4877
TCP-other miss conn 5181 UDP miss conn 697374
other miss conn 953 VPN returned F2F 131
ICMP conn is F2Fed 916 TCP conn is F2Fed 172108
UDP conn is F2Fed 1726 other conn is F2Fed 0
uni-directional viol 0 possible spoof viol 0
TCP state viol 65 out if not def/accl 0
bridge, src=dst 0 routing decision err 0
sanity checks failed 0 temp conn expired 0
fwd to non-pivot 0 broadcast/multicast 0
cluster message 0 partial conn 0
PXL returned F2F 17 cluster forward 296
chain forwarding 0 general reason 0

[Daniel_Taney]

From sk98722 This is the list of violation counters and their descriptions:

• pkt is a fragment - Fragmented packets are always F2F
• pkt has IP options - Packets with IP options are always F2F
• ICMP miss conn - ICMP packets, for which no ICMP connection was found in SecureXL tables
• TCP-SYN miss conn - TCP-SYN packets, for which no TCP connection was found in SecureXL tables
• TCP-other miss conn - TCP packets (other than TCP-SYN), for which no TCP connection was found in SecureXL tables
• UDP miss conn - UDP packets, for which no UDP connection was found in SecureXL tables
• other miss conn - Other packets (non-TCP/non-UDP/non-ICMP), for which no connection was found in SecureXL tables
• VPN returned F2F - Post VPN encrypt / Post VPN decrypt decision was to F2F these packets
• ICMP conn is F2Fed - These ICMP packets were marked explicitly by FireWall as F2F
• TCP conn is F2Fed - These TCP packets were marked explicitly by FireWall as F2F
• UDP conn is F2Fed - These UDP packets were marked explicitly by FireWall as F2F
• other conn is F2Fed - These packets (non-TCP/non-UDP/non-ICMP) were marked explicitly by FireWall as F2F
• uni-directional viol - These packets "seen" in opposite direction (as recorded in the kernel tables) to their corresponding connections that were offloaded by FireWall as uni-directional
• possible spoof viol - These packets failed Anti-Spoofing checks
• TCP state viol - These packets failed TCP state validation checks (includes Sequence validations)
• out if not def/accl - These packets passed through outgoing interface that is marked as non-accelerated (see sk79880)
• bridge, src=dst - These packets passed the same interface (in Bridge mode, inbound / outbound interfaces are the same)
• routing decision err - ARP was not resolved for these packets
• sanity checks failed - VPN post-decryption sanity checks failed for these packets
• temp conn expired - These packets arrives on Temp connection, which has lived past TTL, or packet count
• partial conn - These were inbound packets matched to partial connections
• PXL returned F2F - These inbound packets were F2F based on PXL decision
• cluster forward - These packets were Forwarded to this cluster member over Sync interface from peer members (see "Packet Forwarding"

It looks like TCP conn is F2Fed has the highest number of violations. This would indicate that the Firewall is explicitly determining the traffic doesn't qualify for acceleration. 

Maybe as a next step, use fwaccel conns command and look for connections marked with an "F" as these are the connections being sent to F2F. See if there's any pattern in terms of the type of traffic. In sk98348, the following debug is also given to log reasons for F2F connections. If you go this route, it would only apply to new connections.

[Expert@HostName]# fw ctl debug 0
[Expert@HostName]# fwaccel dbg resetall
[Expert@HostName]# sim dbg resetall
[Expert@HostName]# fw ctl debug -buf 32000
[Expert@HostName]# fwaccel dbg + offload
[Expert@HostName]# sim dbg + f2f
[Expert@HostName]# fwaccel dbg list
[Expert@HostName]# sim dbg list
[Expert@HostName]# fw ctl kdebug -T -f > /var/log/debug.txt

[Mike_Jensen]

In chapter 1 what exactly is meant by "packets that match a service that uses a Resource"?

[Timothy_Hall]

Resources are legacy objects used to integrate third-party servers with a Check Point Firewall, such as URL Filtering (Websense) or Anti-Virus (Trend Micro).  Traffic matching these resource objects must always go F2F:

[Timothy_Hall]

Actually it is quite simple, the Medium Path (PXL) where traffic tends to congregate on most firewalls is not available unless CoreXL is enabled.  So any traffic that cannot be fully handled in SXL is going straight to F2F because CoreXL is disabled.  Get CoreXL enabled (unless you are using Traditional Mode VPNs or route-based VPNs on your R77.30 gateway) and post the statistics again, hopefully most of what is now in F2F will be PXL which is just fine.

[CheckPointerXL]

Hello Tim,

what is best practice to find traffic handled by F2F?

i'm trying to take a look to flows marked with some text at column "Not offloaded reason" into the output of fw tab -t connections -z.

Is this a good practice? any other idea?

Accelerated conns/Total conns : 168/19965 (0%)
LightSpeed conns/Total conns : 0/19965 (0%)
Accelerated pkts/Total pkts : 1958573561/3033151000 (64%)
LightSpeed pkts/Total pkts : 0/3033151000 (0%)
F2Fed pkts/Total pkts : 1074577439/3033151000 (35%)
F2V pkts/Total pkts : 82332855/3033151000 (2%)
CPASXL pkts/Total pkts : 88457791/3033151000 (2%)
PSLXL pkts/Total pkts : 1840341823/3033151000 (60%)
CPAS pipeline pkts/Total pkts : 0/3033151000 (0%)
PSL pipeline pkts/Total pkts : 0/3033151000 (0%)
CPAS inline pkts/Total pkts : 0/3033151000 (0%)
PSL inline pkts/Total pkts : 0/3033151000 (0%)
QOS inbound pkts/Total pkts : 0/3033151000 (0%)
QOS outbound pkts/Total pkts : 0/3033151000 (0%)
Corrected pkts/Total pkts : 0/3033151000 (0%)

F2F packets:
--------------
Violation Packets Violation Packets
-------------------- --------------- -------------------- ---------------
Pkt has IP options 0 ICMP miss conn 1915274
TCP-SYN miss conn 132398284 TCP-other miss conn 443687737
UDP miss conn 49754462 Other miss conn 231648
VPN returned F2F 5383 Uni-directional viol 0
Possible spoof viol 291 TCP state viol 47405
SCTP state affecting 0 Out if not def/accl 0
Bridge src=dst 0 Routing decision err 0
Sanity checks failed 0 Fwd to non-pivot 0
Broadcast/multicast 0 Cluster message 186027350
Cluster forward 12555187 Chain forwarding 0
F2V conn match pkts 191290 General reason 0
Route changes 0 VPN multicast traffic 0
GTP non-accelerated 0 Unresolved nexthop 368

[Timothy_Hall]

I've never found the F2F violation statistics to be of much help when trying to identify why certain connections end up F2F/slowpath.  I think this may be because a violation like this is a connection being kicked out of the fastpath/medium path that was originally selected to be partially or fully accelerated, but does not show why it ended up F2F/slowpath right from the beginning.  Here are the relevant pages from my Gateway Performance Optimization Class about how to determine why certain connections ended up F2F/slowpath:

[Daniel_Taney]

I also hate to be "that guy" but it is probably worth pointing out that in R77.30, individual VS's only run in 32-bit mode; despite the GAIA Operating System itself being 64-bit. I'm assuming this box must be pretty beefy if it has 20-cores. You would definitely get some benefits by upgrading to R80.20. 

When you do upgrade to R80.20, you have to manually set the VS's to run in 64-bit mode:
[hostname@expert:0]# vs_bits [-stat | 32 | 64 ] 

Be aware it does issue a cpstop and cpstart on the vsx gateway.

There were also huge improvements made to SecureXL. Most things can be accelerated & templated now and the list of exceptions shrinks dramatically.