Firewall behavior when handling terminated TCP connections

[Vanness_Chen]

Hi Check Point experts,

We have a VSX environment and recently noticed an interesting behavior that we’d like to get some feedback on.

From SmartView, the customer observed a large number of connections coming from Internet IPs with source port 443 trying to access an internal OA subnet. After checking tcpdump captures, we believe this is related to TCP sessions that are not being closed gracefully when OA clients access Internet services over port 443.

In our setup, there is an F5 load balancer in front of the Check Point firewall. When a connection is abnormally terminated, F5 starts its TCP idle timeout counter (300 seconds). Once the timer expires, F5 sends a RST-ACK packet toward the firewall on behalf of the original Internet IP. These RST-ACK packets are then dropped by the firewall and show up in the logs.

We already tried enabling the workaround mentioned in SK19746:

fw ctl set int fw_rst_expired_conn 1

but this does not seem to change the behavior.

At this point, we’re trying to better understand whether this is expected behavior in such a topology, or if there are other tuning options or best practices (on either Check Point or F5 side) that people have successfully used in similar scenarios.

Any shared experience or suggestions would be greatly appreciated.

[Lesley]

I think this needs to be changed on the F5. In capture you can see that both sides (client and server) wants to close the session with RESETS and FIN packets. 

I think you have to look into and change the scenario 

https://my.f5.com/manage/s/article/K13004262

[Vanness_Chen]

Hi Lesley:

Thanks for your reply.

I previously worked with the F5 engineer to run TCPdump at the same time, and on the F5 side we also observed both the client and the server sending RST packets to each other.

If the workaround described in SK19746 cannot notify F5 to stop waiting for the connection to terminate, then I’ll follow up with the F5 engineer to discuss possible configuration changes on the F5 side instead.

[Lesley · Accepted solution]

I think there is no point to allow the reset packet from the F5 on the check point firewall (the last packet in the capture). Both client and server already have closed it. I suspect traffic is no dropped on firewall with out of state message (please confirm)

Even if you somehow allow the traffic on the firewall it will end up on the client with no purpose because session is already gone

[Vanness_Chen]

Hi Lesley:

The RST-ACK packets coming from the F5 are simply being blocked by the firewall’s cleanup rule, and there are no any out of state messages.

[Lesley]

As a test you can allow the traffic with a rule and maybe then the out of state message will pop-up

[the_rock]

That would make sense, unless you allow them explicitly.

[the_rock]

I would agree with @Lesley . From my experience with this, it usually turns out to be something on F5 side.

[PhoneBoy]

What are the exact errors showing in the logs?
In any case, I suspect we are a bit more aggressive in our dropping of ended connections than the F% is: https://support.checkpoint.com/results/sk/sk39272 

[Vanness_Chen]

Hi PhoenBoy:

Thanks for your reply.

The firewall is simply performing a policy check on the packet. Since there is no matching accept rule, the packet is eventually dropped by the cleanup rule.

[PhoneBoy · Accepted solution]

We remove the entries from the state table shortly after the connection gracefully closes.
Which would make these log messages expected.
If you want to eliminate them, you could create a "no log" rule for the relevant traffic. 

You could create a service like the following:

This service should match source port 443 and a destination port of TCP High Ports (above 1024).

[Vanness_Chen]

Hi PhoneBoy:

I tried the approach you suggested today and it worked very well. I’m glad I learned how to specify the source port (s_port) in a policy rule.
Thank you

[the_rock]

Excellent, thanks for updating us.