Incorrect NAT IP on Interface when failover

[lullejd]

Hi all,

We have this weird situation. Our firewall has a BGP peering with a peer from which certain routes are being learned. The BGP PEER is 172.17.0.10 as per diagram below. One of the routes learned from this peer is 1.1.1.0/24.

BGP works fine and as per below screenshot, it can be seen that route towards 1.1.1.0/24 should pass through eth2.777.

So far so good. Traffic passes fine. Source 172.16.168.34 is able to talk to 1.1.1.0/24 through eth2.777. The source 172.16.168.34 is NATted behind the gateway ip address which is 172.17.0.1.

The issue arises when BGP peer is down. Obviously the learned routes will not be available so since 1.1.1.0/24 is a public IP address, traffic goes then through eth5 which is through the default route.

The only thing is that through fw monitor, we can see that traffic although is going out through the correct interface (eth5), it is still being NATted behind the IP address of interface eth2.777.

In this case originally I was pinging 1.1.1.1 and upon route failover, it stopped working since NATting is not being changed accordingly. However if I try to ping another IP address which was not in use before for example 1.1.1.2, NATting is performed correctly and hidden behind the public ip of eth5. This is causing severe outages on our environment because we expect that traffic is NATted correctly when route failover occurs. It seems that NAT problem occurs only on machines which already have a connection established. So somewhere in the NAT table the values are not being updated.

Effectively this is a lab environment I have created since originally this is a production critical system.

I have also tried to disable secureXL however to no avail. Our production firewalls are with R80.30 Jumbo Hotfix 217, however in the lab I managed even to replicate it on R81 with latest jumbo hotfix, so it seems something common on Checkpoint.

Has anyone encountered such behaviour and how did you overcome it? Basically traffic going out of the correct interface but NATting with the wrong IP address.

For traffic that is ICMP its solved with this command - fw_allow_simultaneous_ping 1 , however we have multiple tunnels which are being established behind our firewall using UDP/443, which when primary route (BGP) is down, they need to still be reachable via the default route.

Thanks in advance

[Yair_Shahar]

Hi,

Same NAT IP will be applied to packets of an already opened connection, main reason is that the server will not recognize this is the same connection and will probably drop the connection.

Once new connection is established new NAT IP will be applied.

Yair

[lullejd]

Hi Yair,

Thanks for the reply.

But what do you mean a new connection? The originating traffic constantly sends tunneled traffic on UDP/443 and UDP if fire and forget. How will CheckPoint be aware that the connection is terminated since there is no three way handshake?

Thanks

[Yair_Shahar]

Do these UDP/443 packets has same source port?

[lullejd]

Hi Yair,

Yes, Source port is 443 and destination port is 443 too.

[Yair_Shahar]

I assume the firewall holds this connection (same 5-tuple) in the firewall connection table and every packet that match this tuple counts as same connection and refresh the timeout so connection not getting deleted (timed out) from the table.

what is the gap between each udp packet?

maybe you can change service timeouts (in service properties) to have the connection deleted from the table on each packet or so?

Yair

[lullejd]

Hi,

The time delta between each packet is negligible as its constantly sending these packets. In wireshark there is a filter with src and dst ip addresses. The screenshot below shows the time delta between each packet.

what do you mean by ' change service timeouts to have the connection deleted form the table on each packet?'

thanks

[Yair_Shahar]

In such delta time I assume we would not be able to cause the connection to get deleted from the connection table on each packet.

BTW - can you verify it is indeed always same connection in 'fw tab -t connections -f -u' | grep <server ip> - you will probably see connection timeout always refreshed

what is the option to make those connections to use different source port?

[lullejd]

option to make these connections use different source port i think its impossible since this is an outpost of AWS always communicating with AWS. So its a black box for us.

with regards to running fw tab -t connections -f -u i think will hang the firewall as it uses a lot of RAM

[Yair_Shahar]

Thanks

I would suggest to contact TAC to see if there any other solution I might not be aware of.

Yair