topic Re: Hide NAT: Simultaneous Connections to the same destination in General Topics

Hide NAT: Simultaneous Connections to the same destination

Di_Junior — Mon, 15 Oct 2018 21:29:31 GMT

Dear Mates

I need a help with the regards to clarification of Hide NAT. According to sk27396, Hide NAT has a limitation of 50.000 simultaneous connections to the same destination. And one of the recomendation provided in the same sk27396 is to Hide behind a range of addresses instead of Hiding behind a single address. Therefore, I need clarification on the following questions that I have:

1. When hiding behind a range of addresses, how is the range IP allocated? Does it use the first available IP in the range, and when it reaches the maximum number of allowed connections (50.000), it automatically starts using the next available IP in the range, or we as administrator can influence on how the IPs in the range for Hide NAT are allocated?

2. Is there any command to verify the number of simultaneous connection to the same destination on the Firewall?

The reason for my questions is because I am hiding a network of many users (Ex: 10.10.0.0/18) behind a single IP address, and they all establish simultaneous connections to the DNS server which is on the Internet leaving my internal network with the single public IP address assigned for Hide NAT. The users of this network use it in order to perform multiple sales transactions.

Your help will be really appreciated.

Re: Hide NAT: Simultaneous Connections to the same destination

Timothy_Hall — Tue, 16 Oct 2018 00:01:23 GMT

In regards to your first question it appears to be some kind of static hash function that you can't modify, please see my response here:

https://community.checkpoint.com/message/6516-r8010-hide-behind-many-question#comment-8238

For the second question, please look at my "favorite" command #2 in this post:

https://community.checkpoint.com/message/6843-my-top-3-check-point-cli-commands#comment-6946

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

Re: Hide NAT: Simultaneous Connections to the same destination

Di_Junior — Tue, 16 Oct 2018 00:41:56 GMT

Hi Tim, thank you very much for always giving a hand when needed. I have been reading about this all issue and I found out that the number of availbale ports is further devided to the number of CoreXL fw instances.

My other question is whether there is any command that can show how many ports are assigned to each coreXL fw instances.

If we have to configure Manual NAT for a network as big as 10.10.0.0/18 with a total of 16,382 available hosts, what would recommend as a range to use in Manual NAT in order to try and overcome the 50k limitation.

If there is another solution other than Manual NAT to overcome the 50K limitation on a network with 16,382 devices going to the same DNS, I would appreciate the recommendation.

Note: even though the network can grow up to 16,382 hosts, we only have up to 7000 active hosts actively issuing the DNS request.

Once again thank you.

Re: Hide NAT: Simultaneous Connections to the same destination

Timothy_Hall — Tue, 16 Oct 2018 01:17:49 GMT

> Hi Tim, thank you very much for always giving a hand when needed. I have been reading about this all issue and I found out that the number of availbale ports is further devided to the number of CoreXL fw instances. My other question is whether there is any command that can show how many ports are assigned to each coreXL fw instances.

Hide NAT ports are dynamically shared and allocated among the CoreXL instances starting in R77.30. The static allocation of Hide NAT ports was a big limitation in R77.20 and earlier and was covered in the first edition of my book. See Dynamic NAT port allocation feature. So there is really no need to track how many Hide NAT ports are in use by a CoreXL instance (Firewall Worker) since it can always allocate one from the pool assuming there is one available.

I believe the setup in the R80.10 - Hide behind many question post is appropriate for your situation. Since it sounds like there will be a crapload of DNS traffic going through the firewall, if it will be a cluster I'd strongly recommend disabling cluster synchronization on the Advanced properties for service domain-udp, as your cluster sync network is likely to get overloaded by all the rapid-fire recursive DNS lookups.

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

Re: Hide NAT: Simultaneous Connections to the same destination

Di_Junior — Tue, 16 Oct 2018 01:31:08 GMT

hi Tim,

yes we are using a cluster in load sharing unicast mode.

Thanks once again

Re: Hide NAT: Simultaneous Connections to the same destination

Di_Junior — Tue, 16 Oct 2018 10:54:47 GMT

Hi Again Tim, I have been looking for a command to show whether the Hide NAT connection is being dropped dute to port Exhaustion.

Any hints?

Thanks

Re: Hide NAT: Simultaneous Connections to the same destination

Timothy_Hall — Tue, 16 Oct 2018 11:46:31 GMT

I don't think there is any kind of counter to examine, you just need to watch for this error message in your logs: sk69480: 'NAT Hide failure - there are currently no available ports for hide operation' log appears repeatedly in SmartView Tracker.

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

Re: Hide NAT: Simultaneous Connections to the same destination

Di_Junior — Tue, 16 Oct 2018 13:55:17 GMT

Hi Tim

As I have mentioned, we have a clustering solution in load sharing unicast mode, we I run the suggest command on both cluster members, I get different number of concurrent connections I get different counters.

Member 1:

#fw tab -u -t connections | grep -ci 294E11EE
57475

Member 2:

# fw tab -u -t connections | grep -ci 294E11EE
53514

1. Are these counters isolated from each member? in the sense that each member is showing how many connections it has towards the limit which is 50k separately.

2. Does it represents the counter for the entire cluster (combining the connections from both cluster members)?

3. Why is it showing different counters (one is 53514, and the other is 53514 )?

Thanks once again

Re: Hide NAT: Simultaneous Connections to the same destination

Timothy_Hall — Tue, 16 Oct 2018 21:02:46 GMT

Would have to diff the raw un-grepped output to figure out the differences between the two members, but they don't seem far enough off to me to indicate a problem is present.

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

Re: Hide NAT: Simultaneous Connections to the same destination

Di_Junior — Tue, 16 Oct 2018 21:10:28 GMT

Hi Tim, thank you very much.

Using the commands you provided helped me explain how the problem was not in the firewall. For a second I forgot to devide by 2 and I got scared lol.

We are about to implement the Manual NAT today in order to assign additional range for the public IPs, because our DNSs have rule to block a certain number of concurrent traffic from a specific IP.

I am again following the instructions you provided about the Manual NAT (many-to-few), so far it worked so well in my Lab. Now I have to implement in the production environment.

Just as an additional question is, since I have a cluster in load sharing, I have to create ARP entry on both members right?

By the way, I just ordered your book from Amazon.

Once again thank you a lot.

Re: Hide NAT: Simultaneous Connections to the same destination

Timothy_Hall — Tue, 16 Oct 2018 21:43:11 GMT

Look here: sk30197: Configuring Proxy ARP for Manual NAT as the exact steps for adding proxy ARPs vary widely depending on OS, cluster mode, and version.

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

Re: Hide NAT: Simultaneous Connections to the same destination

Di_Junior — Wed, 17 Oct 2018 00:35:30 GMT

Hi Tim,

Everything worked as expected.

Thank you very much

Re: Hide NAT: Simultaneous Connections to the same destination

Di_Junior — Tue, 06 Nov 2018 03:10:52 GMT

Hi Tim,

Everything seems to be working now, but the access to the corporate resources is very slow specially when accessing through a browser.

Any hints on how this issue could be overcome? Or some suggestions to improve the response time?

Thanks in advance

Re: Hide NAT: Simultaneous Connections to the same destination

Timothy_Hall — Tue, 06 Nov 2018 13:56:47 GMT

Can you please characterize the bad performance? Does the web browser:

Spin for a long time then the entire page loads relatively quickly
Page starts loading immediately but elements load slowly one by one
Most of page loads quickly but last few elements hang

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

Re: Hide NAT: Simultaneous Connections to the same destination

Di_Junior — Tue, 06 Nov 2018 14:07:41 GMT

Spin for a long time then page loads quickly but few elements load slowly one by one.

Re: Hide NAT: Simultaneous Connections to the same destination

Timothy_Hall — Tue, 06 Nov 2018 14:35:57 GMT

Hmm, is all the content being rendered for the loading webpage located on the one server using the Hide NAT? In other words are there external references to other web servers for part of the loading content? That might give you a clue about what is holding you up.

Use of Hide NAT should not cause this slow loading behavior on its own, since you are seeing a couple of different elements here I'd say taking a full tcpdump of the page load process and pulling it into Wireshark is probably the only definitive way to determine what is going on. When looking at the capture, be on the lookout for long gaps of time between network activity while the page is still trying to load (Wireshark can be configured to sort packet times by the largest inter-packet delays instead of absolute timestamp which is great for situations like these). This should start giving you hints about whether the long delays are mainly caused by the client waiting for the server or the server waiting for the client, which should give you an idea of where to start looking.

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

Re: Hide NAT: Simultaneous Connections to the same destination

Di_Junior — Mon, 12 Nov 2018 16:04:30 GMT

Hi @Timothy Hall

I am struggling to undertand what has been said in sk27396 (see extract in picture bellow).

Does it mean that I can have more 50K concurrent connections using a single Hide NAT IP if it is to a different destination?

For example I have used the command you showed me to check the number of concurrent connections. I get 146232 coonections using a single Hide NAT IP. How do I know if it is to the same destination? if it is to different destination, do connections dont get dropped? because if I devide this number by 2, I get more than 50K limit. How can tell if any connection was dropped if I dont see any logs on smartView Tracker.

Thanks in advance

Re: Hide NAT: Simultaneous Connections to the same destination

Timothy_Hall — Mon, 12 Nov 2018 19:58:38 GMT

Unfortunately there are a lot of different ways to get to the "NAT Hide failure - there are currently no available ports for hide operation" message , so I've never really been able to conclusively determine if Check Point can support more than 50k hide NATs behind the same source IP address as long as the destinations are unique. It is alluded to in several SKs that this is possible but I've gotten that error message enough times over the years to be wary.

We will probably need a definitive answer from R&D, so I'm tagging Dameon Welch-Abernathy‌ here and hopefully he can find out for us if it is possible, and when it became possible (version-wise) because I'm pretty sure this has not been possible since the very beginning. Probably changed around version R75 or so which is also when "many to fewer" Hide NATs became possible:

https://community.checkpoint.com/message/6516-r8010-hide-behind-many-question#comment-8238

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

Re: Hide NAT: Simultaneous Connections to the same destination

PhoneBoy — Mon, 12 Nov 2018 20:37:47 GMT

Theoretically, yes.

In reality, there are some caveats to getting to 50,000 connections to the same destination IP.

Of course, there are only so many ports connections connections can be multiplexed through.

This is why things like Carrier-Grade NAT exist as well.

I'll see if I find the right resource to comment on this.

Re: Hide NAT: Simultaneous Connections to the same destination

PhoneBoy — Tue, 20 Nov 2018 23:37:54 GMT

Does it mean that I can have more 50K concurrent connections using a single Hide NAT IP if it is to a different destination?

The same port can be used again, if the connection is to a different destination, or using a different NAT hide IP address, or is a different IP protocol (e.g. TCP and UDP).

A single hide behind a single hide IP will be enough, unless you have more than 50K simultaneous connections to the same destination.
In that case you will need to hide behind a range of hide addresses.

The outbound connections are split among the HIDE range using X mod N.

That means that the same source IP will always get the same hide IP.
So if we have a lot of connections from the same source to the same destination, we will be out of ports, and hiding behind a range will not solve the problem.

For example I have used the command you showed me to check the number of concurrent connections. I get 146232 connections using a single Hide NAT IP. How do I know if it is to the same destination?

Right now, there isn't a good way to do this.

If you suspect this is an issue, we have a way to determine this with an internal script that parses the fwx_alloc table.

You can send the output of fw tab -t fwx_alloc -u during your peak connection time and we can run it for you.

This can be captured from your standby node.

Note that we do plan to expose these HIDE NAT statistics in later releases.