topic Re: Incorrect NAT IP on Interface when failover in Firewall and Security Management

Incorrect NAT IP on Interface when failover

lullejd — Thu, 30 Sep 2021 06:18:42 GMT

Hi all,

We have this weird situation. Our firewall has a BGP peering with a peer from which certain routes are being learned. The BGP PEER is 172.17.0.10 as per diagram below. One of the routes learned from this peer is 1.1.1.0/24.

BGP works fine and as per below screenshot, it can be seen that route towards 1.1.1.0/24 should pass through eth2.777.

So far so good. Traffic passes fine. Source 172.16.168.34 is able to talk to 1.1.1.0/24 through eth2.777. The source 172.16.168.34 is NATted behind the gateway ip address which is 172.17.0.1.

The issue arises when BGP peer is down. Obviously the learned routes will not be available so since 1.1.1.0/24 is a public IP address, traffic goes then through eth5 which is through the default route.

The only thing is that through fw monitor, we can see that traffic although is going out through the correct interface (eth5), it is still being NATted behind the IP address of interface eth2.777.

In this case originally I was pinging 1.1.1.1 and upon route failover, it stopped working since NATting is not being changed accordingly. However if I try to ping another IP address which was not in use before for example 1.1.1.2, NATting is performed correctly and hidden behind the public ip of eth5. This is causing severe outages on our environment because we expect that traffic is NATted correctly when route failover occurs. It seems that NAT problem occurs only on machines which already have a connection established. So somewhere in the NAT table the values are not being updated.

Effectively this is a lab environment I have created since originally this is a production critical system.

I have also tried to disable secureXL however to no avail. Our production firewalls are with R80.30 Jumbo Hotfix 217, however in the lab I managed even to replicate it on R81 with latest jumbo hotfix, so it seems something common on Checkpoint.

Has anyone encountered such behaviour and how did you overcome it? Basically traffic going out of the correct interface but NATting with the wrong IP address.

For traffic that is ICMP its solved with this command - fw_allow_simultaneous_ping 1 , however we have multiple tunnels which are being established behind our firewall using UDP/443, which when primary route (BGP) is down, they need to still be reachable via the default route.

Thanks in advance

Re: Incorrect NAT IP on Interface when failover

Yair_Shahar — Thu, 30 Sep 2021 08:55:39 GMT

Hi,

Same NAT IP will be applied to packets of an already opened connection, main reason is that the server will not recognize this is the same connection and will probably drop the connection.

Once new connection is established new NAT IP will be applied.

Yair

Re: Incorrect NAT IP on Interface when failover

lullejd — Thu, 30 Sep 2021 09:00:22 GMT

Hi Yair,

Thanks for the reply.

But what do you mean a new connection? The originating traffic constantly sends tunneled traffic on UDP/443 and UDP if fire and forget. How will CheckPoint be aware that the connection is terminated since there is no three way handshake?

Thanks

Re: Incorrect NAT IP on Interface when failover

Yair_Shahar — Thu, 30 Sep 2021 10:00:57 GMT

Do these UDP/443 packets has same source port?

Re: Incorrect NAT IP on Interface when failover

lullejd — Thu, 30 Sep 2021 12:03:42 GMT

Hi Yair,

Yes, Source port is 443 and destination port is 443 too.

Re: Incorrect NAT IP on Interface when failover

Yair_Shahar — Thu, 30 Sep 2021 12:26:14 GMT

I assume the firewall holds this connection (same 5-tuple) in the firewall connection table and every packet that match this tuple counts as same connection and refresh the timeout so connection not getting deleted (timed out) from the table.

what is the gap between each udp packet?

maybe you can change service timeouts (in service properties) to have the connection deleted from the table on each packet or so?

Yair

Re: Incorrect NAT IP on Interface when failover

lullejd — Thu, 30 Sep 2021 13:31:45 GMT

Hi,

The time delta between each packet is negligible as its constantly sending these packets. In wireshark there is a filter with src and dst ip addresses. The screenshot below shows the time delta between each packet.

what do you mean by ' change service timeouts to have the connection deleted form the table on each packet?'

thanks

Re: Incorrect NAT IP on Interface when failover

Yair_Shahar — Thu, 30 Sep 2021 13:46:38 GMT

In such delta time I assume we would not be able to cause the connection to get deleted from the connection table on each packet.

BTW - can you verify it is indeed always same connection in 'fw tab -t connections -f -u' | grep <server ip> - you will probably see connection timeout always refreshed

what is the option to make those connections to use different source port?

Re: Incorrect NAT IP on Interface when failover

lullejd — Thu, 30 Sep 2021 14:08:45 GMT

option to make these connections use different source port i think its impossible since this is an outpost of AWS always communicating with AWS. So its a black box for us.

with regards to running fw tab -t connections -f -u i think will hang the firewall as it uses a lot of RAM

Re: Incorrect NAT IP on Interface when failover

Yair_Shahar — Thu, 30 Sep 2021 14:25:53 GMT

Thanks

I would suggest to contact TAC to see if there any other solution I might not be aware of.

Yair