TCP Out of State Drops - Hidden Monitor Mode

[Timothy_Hall]

This post is based on my initial comments here:

https://community.checkpoint.com/t5/Security-Gateways/Find-the-TCP-out-of-state-packets/m-p/111302/h...

Situation: The "Drop out of state TCP packets" checkbox on the Stateful Inspection screen of the Global Properties is unchecked.  This is not the default setting and is not considered security best practices, as traffic that is "out of state" or does not otherwise exist in the firewall's connections state table will not be blocked if the checkbox is not set.

Rechecking this box is always a bit nerve-racking, especially if no one knows or can remember why it was unchecked in the first place.  Leaving this box unchecked will permit various situations such as asymmetric routing and faulty TCP implementations to continue working, and checking the box could suddenly break these things.  It is possible to quickly toggle the checkbox on a live security gateway as described here if things do suddenly break:

sk117374: How to enable/disable Out of State inspection on a Security Gateway without performing a p...

However the nagging question still remains: regardless of how we toggle the dropping of out of state TCP packets, when should we actually do it?  If we enable it after regular business hours, things may well be broken all night and we get a really lousy start to our morning.  If we do it in the middle of production during the day, the sudden breaking of things could cause even more career-limiting problems to occur even if we manage to toggle it back quickly from the gateway CLI.

But there is another way: there is a hidden "monitor mode" functionality for dropping out of state TCP packets that will allow us to enable this function, see logs for it, but it doesn't actually drop the traffic.  This will allow us to assess the impact of enabling this checkbox without actually breaking things.  So here we see a typical "out of state" drop where the checkbox is set:

However then we toggle this kernel value:

[Expert@A-GW-01:0]# fw ctl get int fw_tcp_out_of_state_monitor
fw_tcp_out_of_state_monitor = 0
[Expert@A-GW-01:0]# fw ctl set int fw_tcp_out_of_state_monitor 1

[Expert@A-GW-01:0]# fw ctl get int fw_tcp_out_of_state_monitor
fw_tcp_out_of_state_monitor = 1

And now:

So toggling the fw_tcp_out_of_state_monitor kernel value to 1, checking the "Drop out of state TCP packets" box and reinstalling the policy will allow us to observe in the logs what would happen if the box was to be rechecked.  Once we are confident that things won't get broken, we toggle fw_tcp_out_of_state_monitor back to its default of 0.

[Wolfgang]

@Timothy_Hall , this will save us from a lot of trouble monday  morning like you wrote.

With the monitoring feature we have time to analyze and debug the traffic without any disruption.

Any information which version support this ? Is it new in R81 or available too in older releases?

Wolfgang

[Timothy_Hall]

Just checked in my lab, the variable fw_tcp_out_of_state_monitor exists in R81 and also at least back into gateway version R80.10.  Can't see why it wouldn't exist in R77.30 and earlier as well, although I have no easy way to verify that at the moment.

[JozkoMrkvicka]

Do you see some real life situation why "TCP packet out of state" should be allowed ?

Isnt it functionality of firewall to inspect the TCP packet and drop packet which is not correct (wrong implementation of TCP RFC or routing) ?

Instead of correcting real TCP packer order, the admins are allowing workaround ?

[Timothy_Hall]

The most common situation is allowing asymmetric routing which is not the end of the world, but is certainly not desirable as it can cause some strange effects especially in the realm of network performance.  Another scenario is the checkbox gets unset as part of troubleshooting and never gets rechecked; I've also seen situations where an organization's firewalls were outsourced to a MSP, then insourced back to the internal IT team, and the box had been left unchecked by the MSP.  Hard to know what to do in that case. 

One other situation would be to uncheck this box prior to a gateway cluster version upgrade to blunt the effect of a non-stateful failover, and then forgetting to recheck it.  However unless you are jumping a very very long way as far as gateway version during the cluster upgrade, Check Point has gotten very good about maintaining "statefulness" through all of the upgrade steps assuming the proper procedures are followed.  So unchecking this box as part of a cluster upgrade is becoming much less common now.

[Vladimir]

Hi @Timothy_Hall . Thank you for a great addition to the toolbox.

Question though: when we enable bypass, is out of state traffic being processed by the rulebase at all?

Also, to add to your statement about the original cause (for unchecked box), being asymmetric routing: in these cases, you pretty much guaranteed to see the anti-spoofing configured in "Detect", disabled outright or including exemptions.  

[Timothy_Hall]

> Question though: when we enable bypass, is out of state traffic being processed by the rulebase at all?

For purposes of stateful inspection and the Firewall/Network policy layer, it is not being inspected as the Bypass is specific to the Firewall blade/feature.  I would assume the out of state traffic is still able to be inspected by other blades such as APCL/URLF and Threat Prevention.

> Also, to add to your statement about the original cause (for unchecked box), being asymmetric routing: in these cases, you pretty much guaranteed to see the anti-spoofing configured in "Detect", disabled outright or including exemptions.  

True but it is possible to configure the same network/subnet on more than one interface in the topology configuration, so antispoofing can still be utilized properly in an asymmetric scenario.  However as you mention in most cases administrators get confused about what is really going on with traffic getting dropped, and either set antispoofing to Detect or just disable it completely.

[HristoGrigorov]

Option is available on R77.20 SMB gateways:

[Expert@myfw]# fw ctl get int fw_tcp_out_of_state_monitor
fw_tcp_out_of_state_monitor = 0

[Equipe_reseau]

@Timothy_Hall, thank you for your useful post.

When the traffic is bypassed, the firewall still allow it, right ?

[G_W_Albrecht]

Yes.

[Lakes]

Hi Timothy_Hall,

Can you please explain the inner mechanics of Hidden Monitor mode? Does this feature send RST packets (to client and server) when out of state packet arrives at the gateway to invite new connection formed after the timed out connection is removed from the connection table? I 'll appreciate if you can shed some light on the inner workings of the Hidden Monitor Mode enabled feature.

[Timothy_Hall]

Be aware there were some fixes for the out of state monitor mode introduced in R81.10 Jumbo HFA 93+:

PRJ-29668, PRHF-18663

SecureXL

When the "fw_tcp_out_of_state_monitor" mode is enabled with the "fw_allow_out_of_state_tcp" flag, some connections may be dropped, although they should go through and be monitored.

[OdWa]

Thanks

[Wolfgang]

Have a look at How to enable/disable Out of State inspection on a Security Gateway without performing a policy inst...

"The above procedure is only temporary and will not survive a reboot, restart of Check Point services (cpstop;cpstart, or cprestart), or policy installation."