How to Disable Out of State TCP/ICMP without a Policy Install

[PhoneBoy]

Generally speaking, disabling "out of state" checks is not recommended.

However, it is sometimes useful in certain situations.

In the distant past, it required a policy installation to change this configuration, but it's possible to enable/disable these checks "on the fly," at least temporarily. 

Refer to the following SK: How to enable/disable Out of State inspection on a Security Gateway without performing a policy inst... 

[Kris_Meldrum]

How would the firewall behave if we disabled state checking for a period of time, say for an upgrade, and then re-enabled?

To be more specific, my belief is that we would allow the existing connections to continue to flow, but don't connections only get registered on the SYN? So if we have long lived connections, and a new SYN is never sent between the time we disabled and when we re-enable state checking, would a connection be registered for that connection? Or would it drop as out-of-state as soon as state checking was re-enabled?

[the_rock]

That is my understanding as well, but maybe someone else can confirm.

[PhoneBoy]

I'm pretty sure I've suggested exactly this for upgrades where people are concerned about losing active connections. 🙂
My understanding is as long as some traffic on the connection is seen, and the connection would be allowed by the Access Policy, there should be an entry for it in the connections table.

[Kris_Meldrum]

I've heard this recommended by Tim as well which is why I'm looking at it. Let me enrich this story and see if it makes a difference in your answer. 

Specifically, we're seeing an issue when we're expanding security groups on a chassis. The basic process is down the standby, insert blades, add them to the security group with autoclone enabled, bring the standby back up. Here's where the problem occurs. When we bring the standby back up there is a mismatch of blades, and we believe that it messes with SyncXL. Where let's say the backup of the connection is on 2_4 (chassis 2 was the standby and where we've expanded the security group). When we bring chassis 2 back up, the SyncXL distribution is different from chassis1 to chassis2, and normally the flow would've gone to 2_04, but instead is sent to 2_11 (new blade) which has no knowledge of the connection and is dropped as out of state.  R&D has said this shouldn't happen, but it was proven in solution center. So my question is, if there is no connection registered on 2_11 and we turn off state check, would 2_11 accept the packet and add it to the connection table? 

Adding to the connection table is the big question since these are long lived connections and we don't expect to see a new SYN.

[PhoneBoy]

Doesn't make a difference in my answer, though this being Scalable Platforms does add a new wrinkle 🙂

[Timothy_Hall]

I'm not familiar with the intricacies of how SyncXL differs from "standard" state sync, but disabling the dropping of out of state TCP packets should help blunt the effect of the issue you are seeing.  Just remember that while out of state checks are disabled, try to "exercise" as many of those long-running connections as possible before re-enabling it to resurrect as many of those long-running connections back into the new module's state table as possible. 

Note that there is also a hidden "monitor mode" for out of state checks that may be helpful in assessing whether it is safe to re-enable the out of state check setting again by watching the logs:

TCP Out of State Drops - Hidden Monitor Mode

[Kris_Meldrum]

Interesting...I think that should help in testing.

The biggest difference between chassis and standard appliance is the connection is synchronized to a specific blade on the backup chassis, not every blade gets a copy of the connection, and this is determined by the distribution algorithm. However, when we change the blade counts it changes the algorithm so the traffic get's given to blades that have no knowledge of the connection, hence why I'm curious if a blade would register a connection based on an ACK if it has no prior knowledge of the connection.

[Kris_Meldrum]

Just to bring everything full circle we found while testing in the lab that the connections were not registered based on the ACK. As soon as we re-enabled TCP state checking we saw multiple connections start dropping as out of state (that shouldn't be).  It did fix the problem regarding the first failover with uneven number of SGM's, but once re-enabled the connections hadn't been registered (or properly registered) and drops start occurring again. This is at least the case for the chassis with R76sp.50, not sure about other platforms.

Also, the TCP state monitor is unsupported on R76sp.50 as well just as an FYI.

[Timothy_Hall]

OK thanks for the update, interesting.