R80.x Performance Tuning and Debug Tips – fw monitor

[HeikoAnkenbrand]

R80.20 - fw monitor

Tip 1

SecureXL has been significantly revised in R80.20. It now works in user space. This has also led to some changes in "fw monitor".

Since R80.20  "fw monitor" is able to show the traffic accelerated with SecureXL. Thus it is possible to see SecureXL (provide more performance) modules in fw monitor chain. For more informations revert to "SecureXL offloading chain modules" in this article. Now you can see that SecureXL is used, which increases the performance of the firewall.

SecureXL "fwaccel off" does not have to be disabled on R80.20 to run "fw monitor". This is good for performance, so "fw monitor" does not affect performance any more.

# fwaccel off                                      > no longer necessary in R80.20 and above

# fw monitor -e "accept(...);"

R77.30 and R80.10 - fw monitor

On R77.30 and R80.10 only disabling SecureXL allows to see the complete connection in fw monitor, which may be required for troubleshooting purposes or revert to "How to disable SecureXL for specific IP addresses".

# fwaccel off                            

# fw monitor -e "accept(...);"

Chapter

More interesting articles:

- R80.x Architecture and Performance Tuning - Link Collection
- Article list (Heiko Ankenbrand)

New fw monitor inspection points in R80.20

Tip 2

Furthermore there are new fw monitor inspection points available:

Inspection point	Name of fw monitor inspection point	Relation to firewall VM	Available since version
i	Pre-Inbound	Before the inbound FireWall VM                            (for example, eth1:i)	always
I	Post-Inbound	After the inbound FireWall VM                               (for example, eth1:I)	always
id	Pre-Inbound VPN	Inbound before decrypt                                          (for example, eth1:id)	R80.20
iD	Post-Inbound VPN	Inbound after decrypt                                             (for example, eth1:ID)	R80.20
iq	Pre-Inbound QoS	Inbound before QoS                                               (for example, eth1:iq)	R80.20
iQ	Post-Inbound QoS	Inbound after QoS                                                  (for example, eth1:IQ)	R80.20
o	Pre-Outbound	Before the outbound FireWall VM                           (for example, eth1:o)	always
O	Post-Outbound	After the outbound FireWall VM                              (for example, eth1:O)	always
e oe	Pre-Outbound VPN*	Outbound before encrypt                                        (for example, eth1:e)    in R80.10                                                                                 (for example, eth1:oe)  in R80.20	R80.10 R80.20
E OE	Post-Outbound VPN*	Outbound after encrypt                                           (for example, eth1:E)    in R80.10                                                                                 (for example, eth1:OE)  in R80.20	R80.10 R80.20
oq	Pre-Outbound QoS	Outbound before QoS                                             (for example, eth1:oq)	R80.20
oQ	Post-Outbound QoS	Outbound after QoS                                                (for example, eth1:OQ)	R80.20

* The fw monitor inspection point is different in R80.10 ("e" or "E") and R80.20 ("oe" and "OE")

For more information, see sk30583, fw monitor or How to use FW Monitor.

SecureXL offloading chain modules

Tip 3

Like I said SecureXL has been significantly revised in R80.20. It now works in user space. This has also led to some changes in "fw monitor"

There are new fw monitor chain (SecureXL) objects that do not run in the virtual machine.

# fw ctl chain  
The new fw monitor chain modules (SecureXL) do not run in the virtual machine (vm).
SecureXL inbound (sxl_in)                 > Packet received in SecureXL from network
SecureXL inbound CT (sxl_ct)           > Accelerated packets moved from inbound to outbound processing (post routing)
SecureXL outbound (sxl_out)            > Accelerated packet starts outbound processing
SecureXL deliver (sxl_deliver)          > SecureXL transmits accelerated packet

New VM chain modules in R80.20

Tip 4

There are more new chain modules in R80.20

vpn before offload (vpn_in)                  > FW inbound preparing the tunnel for offloading the packet (along with the connection)
fw offload inbound (offload_in)            > FW inbound that perform the offload
fw post VM inbound  (post_vm)            > Packet was not offloaded (slow path) - continue processing in FW inbound

# fw ctl chain 

New fw monitor chain key (00000000)

Tip 5

In Firewall kernel (now also SecureXL), each kernel is associated with a key (red) witch specifies the type of traffic applicable to the chain modul.

 # fw ctl chain 

Key	Function
ffffffff	IP Option Stip/Restore
00000001	new processed flows
00000002	wire mode
00000003	will applied to all ciphered traffic (VPN)
00000000	SecureXL offloading (new in R80.20+)

References

R&D meeting Israel

[HeikoAnkenbrand]

I can see in the lab that many SecureXL connections are visible in fw monitor without disabling SecureXL. Does this mean that under R80.20 we don't have to deactivate SexureXL with "fwaccel off" to run "fw monitor"?

Could you confirm that from Check Point?

Regards

Heiko

[Reihard_Westle]

Nice information.

Thanks

Reinhard

[PhoneBoy]

That’s correct and one of the benefits of moving most of SecureXL into user space.

[HeikoAnkenbrand]

Hi Dameon,

thanks for this information.

Regards

Heiko

[Roger_Bachstein]

[Martin_Raska]

Can someone explain performance benefit of moving SecureXL into user space? My understanding was everything in user space can create a bottleneck and old implementation of SecureXL layer under kernel at low level should be faster.

[PhoneBoy]

The SecureXL driver takes a certain amount of kernel memory per core and that was adding up to more kernel memory than Intel/Linux was allowing.

On the 23900 in particular, we could not leverage all the processor cores due to this limitation. 

By moving all (or most) of SecureXL to user space, it's possible to leverage more processor cores as the firewall can entirely run in user space.

(Note it still doesn't by default in R80.20 in non-VSX mode, but it can be enabled.)

It also means certain kinds of low-level packet processing that could not easily be done in SecureXL because it was being done in the kernel now can.

For VSX in particular, it means you can now configure the penalty box features on a per-VS basis.

It also improves session establishment rates on the higher-end appliances.

[Tim_Maurer]

Check Point should include this in the SK.

[Sven_Glock]

Am I right that this will influence tcpdump usage as well?

[PhoneBoy]

Yes, namely that you don't need to turn of SecureXL to use it

[Rob_Bush]

Hello.  I am dealing with an issue where I have found that when SecureXL is turned on for our R80.20 gateway, traffic is not flowing across our VPN correctly.  I just stumbled on this article and I'm very excited to try and use the new features in FW Monitor to isolate the issue.  That being said, I noticed your comment here that SecureXL does not run in the user space on non-VSX mode, so I'm presuming the ability to use the SecureXL features of FW Monitor then would not work.  I have searched the Checkpoint KB to try and find out how to enable SecureXL to run in the user space on non-VSX, but I'm striking out.  Could you point me in the right direction on how to enable that?

[Rob_Bush]

Might have found it.  Is it the "kiss_usermode_enabled" parameter of the SecureXL Kernel Parameters?

[Martin_Raska]

Please explain why its not enabled by default or how to enable SecureXL in user space or relevant SK which decribe it. Thanks

[HeikoAnkenbrand · Accepted solution]

Hi Martin Raska,

With R80.20 SecureXL works automatically in user space.

You can switch SecureXL on and off as usual in R80.20.

Permanent:

# cpconfig

Until the next reboot:

# fwaccel off

or

# fwaccel on

Regards

Heiko

[Martin_Raska]

Thank you, I was confused by Daemon comment "(Note it still doesn't by default in R80.20 in non-VSX mode, but it can be enabled.)"

[PhoneBoy]

To be clear, a LOT of SecureXL is already in userspace (as Heiko said).

Some firewalling is still in the kernel.

There is a separate "user mode" switch specifically for the 23900 on R80.20 (or anything above 40 cores, really) that moves all of the firewall into userspace (same as VSX, where we've already done that).

We haven't published the command yet and I need to check with R&D before I post it here

[PhoneBoy]

I want to correct a few inaccuracies that I helped to propagate above thanks to a gentle email from R&D.

SecureXL is actually not in userspace in R80.20, it still mostly happens in the kernel.

That said a lot of other things have moved to userspace that lead to some of the performance/functionality improvements in R80.20 even in Security Gateway mode.

VSX instances have run in userspace since R75.40VS.

Starting from R80.20, it is possible to run regular Security Gateway (non-VSX) mode with usermode firewall enabled.

It is NOT the default in R80.20 and is ONLY required on platforms that have more than 40 cores (eg 23900).

On platforms with 40 cores or less, you can enable it if you wish, but don't expect a performance improvement.

If you wish to enable usermode firewall in R80.20:

• cpprod_util FwSetUsermode 1
• reboot

In R80.30, we plan to make this the default for the 23900.

[Rene_Rosenkrant]

On VSX R80.20 with Jumbo 47. I see that the flow is sometimes limited to "id", for trafic related to VPN.

Stating that disabling secureXL, is not longer necessary must be in very specific situations?

And disabling secureXL can make VPN unstable.

Is it backed by Checkpoint and will Checkpoint troubleshoot with fw monitor without disabling secureXL?

Best regards

René Rosenkrantz 

[Rene_Rosenkrant]

On VSX R80.20 with Jumbo 47. I see that the flow is sometimes limited to "id", for trafic related to VPN.

Stating that disabling secureXL, is not longer necessary must be in very specific situations?

And disabling secureXL can make VPN unstable.

Is it backed by Checkpoint and will Checkpoint troubleshoot with fw monitor without disabling secureXL?

Best regards

René Rosenkrantz 

[Anurag_Lilha]

Awesome article! very informative.

The fw monitor syntax is changed in R80.20 Take_73 and above. The SK30583 usage is shown as:

• For IPv4:

  [Expert@HostName]# fw monitor [-h] [-u|-s] [-i] [-d] [-D] [-t] [{-e <expr>}+ | -f <filter_file_name>|-] [-l length] [-m i|I|o|O] [-x offset[,length]] [-o <output_file_name>] <[-pi position] [-pI position] [-po position] [-pO position] | -p all> [-a] [-ci count] [-co count] [-v VSID] 
  Note: Relevant for R80.20 from Jumbo Hotfix Accumulator for R80.20 (Take 73)
  [-w whole packet] [-F simple filter "{src IP},{src port},{dst IP},{dst port},{protocol num}"] [-U clear]

Just wanted to clarify this new syntax:

• For source IP filter only:

  [Expert@HostName]# fw monitor -F "{src=x.x.x.x}"

• For destination IP filter only:

  [Expert@HostName]# fw monitor -F "{dst=x.x.x.x}"

Is that correct?

[Timothy_Hall · Accepted solution]

Just wanted to clarify this new syntax:

• For source IP filter only:

  [Expert@HostName]# fw monitor -F "{src=x.x.x.x}"

• For destination IP filter only:

  [Expert@HostName]# fw monitor -F "{dst=x.x.x.x}"

Is that correct?

This syntax is not correct, and specifying it in the way shown will give you a COMPLETELY UNFILTERED capture.  If you make even the slightest mistake in the fw monitor -F syntax it will blast you with every accepted packet four times, and almost certainly cause amplification of your own SSH session packets as they will get caught in it as well.   Note that wildcards such as "*" and "?" are NOT supported in the filtering expression either, and will be treated as a syntax error. So if you wanted to see everything from a srcIP the correct syntax is:

fw monitor -F srcIP,0,0,0,0

Note this will only show you packets sourced from srcIP but not the replies coming back.  If you want to see replies coming back (i.e. the equivalent filtering of "host srcIP"), the syntax is:

fw monitor -F srcIP,0,0,0,0 -F 0,0,srcIP,0,0

[HeikoAnkenbrand]

Small comment:
The FW Monitor utility captures network packets at multiple capture points along the FireWall inspection chains:
- The -F option already shows packets on SecureXL level (Picture red).
- The -e option already shows packets on inspection kernel level (Picture blue).