Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
Timothy_Hall
Champion
Champion

TCP Out of State Drops - Hidden Monitor Mode

This post is based on my initial comments here:

https://community.checkpoint.com/t5/Security-Gateways/Find-the-TCP-out-of-state-packets/m-p/111302/h...

Situation: The "Drop out of state TCP packets" checkbox on the Stateful Inspection screen of the Global Properties is unchecked.  This is not the default setting and is not considered security best practices, as traffic that is "out of state" or does not otherwise exist in the firewall's connections state table will not be blocked if the checkbox is not set.

Rechecking this box is always a bit nerve-racking, especially if no one knows or can remember why it was unchecked in the first place.  Leaving this box unchecked will permit various situations such as asymmetric routing and faulty TCP implementations to continue working, and checking the box could suddenly break these things.  It is possible to quickly toggle the checkbox on a live security gateway as described here if things do suddenly break:

sk117374: How to enable/disable Out of State inspection on a Security Gateway without performing a p...

However the nagging question still remains: regardless of how we toggle the dropping of out of state TCP packets, when should we actually do it?  If we enable it after regular business hours, things may well be broken all night and we get a really lousy start to our morning.  If we do it in the middle of production during the day, the sudden breaking of things could cause even more career-limiting problems to occur even if we manage to toggle it back quickly from the gateway CLI.

But there is another way: there is a hidden "monitor mode" functionality for dropping out of state TCP packets that will allow us to enable this function, see logs for it, but it doesn't actually drop the traffic.  This will allow us to assess the impact of enabling this checkbox without actually breaking things.  So here we see a typical "out of state" drop where the checkbox is set:

state_drop.png

However then we toggle this kernel value:

[Expert@A-GW-01:0]# fw ctl get int fw_tcp_out_of_state_monitor
fw_tcp_out_of_state_monitor = 0
[Expert@A-GW-01:0]# fw ctl set int fw_tcp_out_of_state_monitor 1

[Expert@A-GW-01:0]# fw ctl get int fw_tcp_out_of_state_monitor
fw_tcp_out_of_state_monitor = 1

And now:

bypass.png

So toggling the fw_tcp_out_of_state_monitor kernel value to 1, checking the "Drop out of state TCP packets" box and reinstalling the policy will allow us to observe in the logs what would happen if the box was to be rechecked.  Once we are confident that things won't get broken, we toggle fw_tcp_out_of_state_monitor back to its default of 0.

Gateway Performance Optimization R81.20 Course
now available at maxpowerfirewalls.com
13 Replies
Wolfgang
Authority
Authority

@Timothy_Hall , this will save us from a lot of trouble monday  morning like you wrote.

With the monitoring feature we have time to analyze and debug the traffic without any disruption.

Any information which version support this ? Is it new in R81 or available too in older releases?

Wolfgang

 

 

0 Kudos
Timothy_Hall
Champion
Champion

Just checked in my lab, the variable fw_tcp_out_of_state_monitor exists in R81 and also at least back into gateway version R80.10.  Can't see why it wouldn't exist in R77.30 and earlier as well, although I have no easy way to verify that at the moment.

Gateway Performance Optimization R81.20 Course
now available at maxpowerfirewalls.com
0 Kudos
JozkoMrkvicka
Mentor
Mentor

Do you see some real life situation why "TCP packet out of state" should be allowed ?

Isnt it functionality of firewall to inspect the TCP packet and drop packet which is not correct (wrong implementation of TCP RFC or routing) ?

Instead of correcting real TCP packer order, the admins are allowing workaround ?

Kind regards,
Jozko Mrkvicka
0 Kudos
Timothy_Hall
Champion
Champion

The most common situation is allowing asymmetric routing which is not the end of the world, but is certainly not desirable as it can cause some strange effects especially in the realm of network performance.  Another scenario is the checkbox gets unset as part of troubleshooting and never gets rechecked; I've also seen situations where an organization's firewalls were outsourced to a MSP, then insourced back to the internal IT team, and the box had been left unchecked by the MSP.  Hard to know what to do in that case. 

One other situation would be to uncheck this box prior to a gateway cluster version upgrade to blunt the effect of a non-stateful failover, and then forgetting to recheck it.  However unless you are jumping a very very long way as far as gateway version during the cluster upgrade, Check Point has gotten very good about maintaining "statefulness" through all of the upgrade steps assuming the proper procedures are followed.  So unchecking this box as part of a cluster upgrade is becoming much less common now.

Gateway Performance Optimization R81.20 Course
now available at maxpowerfirewalls.com
Vladimir
Champion
Champion

Hi @Timothy_Hall . Thank you for a great addition to the toolbox.

Question though: when we enable bypass, is out of state traffic being processed by the rulebase at all?

Also, to add to your statement about the original cause (for unchecked box), being asymmetric routing: in these cases, you pretty much guaranteed to see the anti-spoofing configured in "Detect", disabled outright or including exemptions.  

0 Kudos
Timothy_Hall
Champion
Champion

> Question though: when we enable bypass, is out of state traffic being processed by the rulebase at all?

For purposes of stateful inspection and the Firewall/Network policy layer, it is not being inspected as the Bypass is specific to the Firewall blade/feature.  I would assume the out of state traffic is still able to be inspected by other blades such as APCL/URLF and Threat Prevention.

> Also, to add to your statement about the original cause (for unchecked box), being asymmetric routing: in these cases, you pretty much guaranteed to see the anti-spoofing configured in "Detect", disabled outright or including exemptions.  

True but it is possible to configure the same network/subnet on more than one interface in the topology configuration, so antispoofing can still be utilized properly in an asymmetric scenario.  However as you mention in most cases administrators get confused about what is really going on with traffic getting dropped, and either set antispoofing to Detect or just disable it completely.

Gateway Performance Optimization R81.20 Course
now available at maxpowerfirewalls.com
0 Kudos
HristoGrigorov

Option is available on R77.20 SMB gateways:

[Expert@myfw]# fw ctl get int fw_tcp_out_of_state_monitor
fw_tcp_out_of_state_monitor = 0

Equipe_reseau
Participant

@Timothy_Hall, thank you for your useful post.

When the traffic is bypassed, the firewall still allow it, right ?

0 Kudos
G_W_Albrecht
Legend
Legend

Yes.

CCSE CCTE CCSM SMB Specialist
0 Kudos
Lakes
Explorer

Hi Timothy_Hall,

Can you please explain the inner mechanics of Hidden Monitor mode? Does this feature send RST packets (to client and server) when out of state packet arrives at the gateway to invite new connection formed after the timed out connection is removed from the connection table? I 'll appreciate if you can shed some light on the inner workings of the Hidden Monitor Mode enabled feature.

0 Kudos
Timothy_Hall
Champion
Champion

Be aware there were some fixes for the out of state monitor mode introduced in R81.10 Jumbo HFA 93+:

 

PRJ-29668,
PRHF-18663

SecureXL

When the "fw_tcp_out_of_state_monitor" mode is enabled with the "fw_allow_out_of_state_tcp" flag, some connections may be dropped, although they should go through and be monitored.

 

Gateway Performance Optimization R81.20 Course
now available at maxpowerfirewalls.com
0 Kudos
OdWa
Employee
Employee

Thanks

0 Kudos
Wolfgang
Authority
Authority

Have a look at How to enable/disable Out of State inspection on a Security Gateway without performing a policy inst...

"The above procedure is only temporary and will not survive a reboot, restart of Check Point services (cpstop;cpstart, or cprestart), or policy installation."

0 Kudos

Leaderboard

Epsum factorial non deposit quid pro quo hic escorol.

Upcoming Events

    CheckMates Events