Hide NAT: Simultaneous Connections to the same destination

[Di_Junior]

Dear Mates

I need a help with the regards to clarification of Hide NAT. According to sk27396, Hide NAT has a limitation of 50.000 simultaneous connections to the same destination. And one of the recomendation provided in the same sk27396 is to Hide behind a range of addresses instead of Hiding behind a single address. Therefore, I need clarification on the following questions that I have:

1. When hiding behind a range of addresses, how is the range IP allocated? Does it use the first available IP in the range, and when it reaches the maximum number of allowed connections (50.000), it automatically starts using the next available IP in the range, or we as administrator can influence on how the IPs in the range for Hide NAT  are allocated? 

2. Is there any command to verify the number of simultaneous connection to the same destination on the Firewall?

The reason for my questions is because I am hiding a network of many users (Ex: 10.10.0.0/18) behind a single IP address, and they all establish simultaneous connections to the DNS server which is on the Internet leaving my internal network with the single public IP address assigned for Hide NAT. The users of this network use it in order to perform multiple sales transactions.

Your help will be really appreciated.

[Timothy_Hall · Accepted solution]

In regards to your first question it appears to be some kind of static hash function that you can't modify, please see my response here:

https://community.checkpoint.com/message/6516-r8010-hide-behind-many-question#comment-8238 

For the second question, please look at my "favorite" command #2 in this post:

https://community.checkpoint.com/message/6843-my-top-3-check-point-cli-commands#comment-6946 

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

[Di_Junior]

Hi Tim, thank you very much for always giving a hand when needed. I have been reading about this all issue and I found out that the number of availbale ports is further devided to the number of CoreXL fw instances. 

My other question is whether there is any command that can show how many ports are assigned to each coreXL fw instances. 

If we have to configure Manual NAT for a network as big as 10.10.0.0/18 with a total of 16,382 available hosts, what would recommend as a range to use in Manual NAT in order to try and overcome the 50k limitation.

If there is another solution other than Manual NAT to overcome the 50K limitation on a network with 16,382 devices going to the same DNS, I would appreciate the recommendation.

Note: even though the network can grow up to 16,382 hosts, we only have up to 7000 active hosts actively issuing the DNS request.

Once again thank you.

[Timothy_Hall]

> Hi Tim, thank you very much for always giving a hand when needed. I have been reading about this all issue and I found out that the number of availbale ports is further devided to the number of CoreXL fw instances.  My other question is whether there is any command that can show how many ports are assigned to each coreXL fw instances. 

Hide NAT ports are dynamically shared and allocated among the CoreXL instances starting in R77.30.  The static allocation of Hide NAT ports was a big limitation in R77.20 and earlier and was covered in the first edition of my book.  See Dynamic NAT port allocation feature.  So there is really no need to track how many Hide NAT ports are in use by a CoreXL instance (Firewall Worker) since it can always allocate one from the pool assuming there is one available.

I believe the setup in the R80.10 - Hide behind many question post is appropriate for your situation.  Since it sounds like there will be a crapload of DNS traffic going through the firewall, if it will be a cluster I'd strongly recommend disabling cluster synchronization on the Advanced properties for service domain-udp, as your cluster sync network is likely to get overloaded by all the rapid-fire recursive DNS lookups.

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

[Di_Junior]

hi Tim,

yes we are using a cluster in load sharing unicast mode. 

Thanks once again 

[Di_Junior]

Hi Again Tim, I have been looking for a command to show whether the Hide NAT connection is being dropped dute to port Exhaustion.

Any hints?

Thanks

[Timothy_Hall]

I don't think there is any kind of counter to examine, you just need to watch for this error message in your logs: sk69480: 'NAT Hide failure - there are currently no available ports for hide operation' log appears .... 

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

[Di_Junior]

Hi Tim,

I have finnaly got my copy.

[Di_Junior]

Hi Tim

As I have mentioned, we have a clustering solution in load sharing unicast mode, we I run the suggest command on both cluster members, I get different number of concurrent connections  I get different counters.

Member 1:

#fw tab -u -t connections | grep -ci 294E11EE
57475

Member 2:

# fw tab -u -t connections | grep -ci 294E11EE
53514

1. Are these counters isolated from each member? in the sense that each member is showing how many connections it has towards the limit which is 50k separately.

2. Does it represents the counter for the entire cluster (combining the connections from both cluster members)? 

3. Why is it showing different counters (one is 53514, and the other is 53514 )?

Thanks once again

[Timothy_Hall]

Would have to diff the raw un-grepped output to figure out the differences between the two members, but they don't seem far enough off to me to indicate a problem is present.

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

[Di_Junior]

Hi Tim, thank you very much.

Using the commands you provided helped me explain how the problem was not in the firewall. For a second I forgot to devide by 2 and I got scared lol.

We are about to implement the Manual NAT today in order to assign additional range for the public IPs, because our DNSs have rule to block a certain number of concurrent traffic from a specific IP.

I am again following the instructions you provided about the Manual NAT (many-to-few), so far it worked so well in my Lab. Now I have to implement in the production environment.

Just as an additional question is, since I have a cluster in load sharing, I have to create ARP entry on both members right?

By the way, I just ordered your book from Amazon.

Once again thank you a lot.

[Timothy_Hall]

Look here: sk30197: Configuring Proxy ARP for Manual NAT as the exact steps for adding proxy ARPs vary widely depending on OS, cluster mode, and version.

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

[Di_Junior]

Hi Tim,

Everything worked as expected.

Thank you very much

[Di_Junior]

Hi Tim,

Everything seems to be working now, but the access to the corporate resources is very slow specially when accessing through a browser.

Any hints on how this issue could be overcome? Or some suggestions to improve the response time?

Thanks in advance

[Timothy_Hall]

Can you please characterize the bad performance?  Does the web browser:

• Spin for a long time then the entire page loads relatively quickly
• Page starts loading immediately but elements load slowly one by one
• Most of page loads quickly but last few elements hang

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

[Di_Junior]

Spin for a long time then  page loads quickly but few elements load slowly one by one.

[Timothy_Hall]

Hmm, is all the content being rendered for the loading webpage located on the one server using the Hide NAT?  In other words are there external references to other web servers for part of the loading content?  That might give you a clue about what is holding you up.

Use of Hide NAT should not cause this slow loading behavior on its own, since you are seeing a couple of different elements here I'd say taking a full tcpdump of the page load process and pulling it into Wireshark is probably the only definitive way to determine what is going on.  When looking at the capture, be on the lookout for long gaps of time between network activity while the page is still trying to load (Wireshark can be configured to sort packet times by the largest inter-packet delays instead of absolute timestamp which is great for situations like these).  This should start giving you hints about whether the long delays are mainly caused by the client waiting for the server or the server waiting for the client, which should give you an idea of where to start looking.

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

[Di_Junior]

Hi  @Timothy Hall 

I am struggling to undertand what has been said in sk27396 (see extract in picture bellow).

Does it mean that I can have more 50K concurrent connections using a single Hide NAT IP if it is to a different destination?

For example I have used the command you showed me to check the number of concurrent connections. I get 146232 coonections using a single Hide NAT IP. How do I know if it is to the same destination? if it is to different destination, do connections dont get dropped? because if I devide this number by 2, I get more than 50K limit. How can tell if any connection was dropped if I dont see any logs on smartView Tracker.

Thanks in advance

[Timothy_Hall]

Unfortunately there are a lot of different ways to get to the "NAT Hide failure - there are currently no available ports for hide operation" message , so I've never really been able to conclusively determine if Check Point can support more than 50k hide NATs behind the same source IP address as long as the destinations are unique.  It is alluded to in several SKs that this is possible but I've gotten that error message enough times over the years to be wary. 

We will probably need a definitive answer from R&D, so I'm tagging Dameon Welch-Abernathy‌ here and hopefully he can find out for us if it is possible, and when it became possible (version-wise) because I'm pretty sure this has not been possible since the very beginning.  Probably changed around version R75 or so which is also when "many to fewer" Hide NATs became possible:

https://community.checkpoint.com/message/6516-r8010-hide-behind-many-question#comment-8238

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

[PhoneBoy]

Theoretically, yes.

In reality, there are some caveats to getting to 50,000 connections to the same destination IP.

Of course, there are only so many ports connections connections can be multiplexed through. 

This is why things like Carrier-Grade NAT exist as well. 

I'll see if I find the right resource to comment on this.

[PhoneBoy]

Does it mean that I can have more 50K concurrent connections using a single Hide NAT IP if it is to a different destination?

The same port can be used again, if the connection is to a different destination, or using a different NAT hide IP address, or is a different IP protocol (e.g. TCP and UDP).

A single hide behind a single hide IP will be enough, unless you have more than 50K simultaneous connections to the same destination.
In that case you will need to hide behind a range of hide addresses.

The outbound connections are split among the HIDE range using X mod N.

That means that the same source IP will always get the same hide IP.
So if we have a lot of connections from the same source to the same destination, we will be out of ports, and hiding behind a range will not solve the problem.

For example I have used the command you showed me to check the number of concurrent connections. I get 146232 connections using a single Hide NAT IP. How do I know if it is to the same destination?

Right now, there isn't a good way to do this.

If you suspect this is an issue, we have a way to determine this with an internal script that parses the fwx_alloc table.

You can send the output of fw tab -t fwx_alloc -u during your peak connection time and we can run it for you.

This can be captured from your standby node.

Note that we do plan to expose these HIDE NAT statistics in later releases.

[Di_Junior]

Hi there  Dameon Welch-Abernathy

could you please elaborate more on the quote below:

"The outbound connections are split among the HIDE range using X mod N".

Could you perhaps give an example? lets say we are HIDING the network 192.168.0.0/24 behind the range of 4 IPs (10.10.0.1 - 10.10.0.4). I have been asked this question before about the allocation of IPs when hiding behind a range, and I was not able to give a proper answer.

Thanks in advance

[PhoneBoy]

X mod N is the math equation used to determine this.

In the context of the question you asked:

• The IP address is X (converted to an integer number)
  • x.y.z.w = (x * 256^3) + (y * 256^2) + (z * 256^1) + w
• N is the number of IPs in the hide range
• The result of X mod N (the remainder of N divided by X) will determine which IP in the range to use for hiding

So using your example:

• 192.168.0.1 would hide behind IP 10.10.0.2 (3232235521 mod 4 = 1)
• 192.168.0.2 would hide behind IP 10.10.0.3 (3232235522 mod 4 = 2)
• 192.168.0.3 would hide behind IP 10.10.0.4 (3232235523 mod 4 = 3)
• 192.168.0.4 would hide behind IP 10.10.0.1 (3232235524 mod 4 = 0)

[Di_Junior]

Hi Dameon Welch-Abernathy 

The output file contains alot of information, is there any way I can get the output of the command you gave (fw tab -t fwx_alloc -u) so that you can run it, or I can just paste the output here.

Otherwise, I would also like to know if there is any tool that could help me read the output since its in Hexadecimal.

Thanks 

[Timothy_Hall]

Try passing -f (format) to the command, should display IP addresses in decimal:

fw tab -t fwx_alloc -u -f

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

[Di_Junior]

Hi Timothy Hall 

Just to add an additional question to this thread.

Is the 50k limitation is applied when doing "Static NAT"? if not, why?

Thanks in advance.

I am taking my book to CPX360 in Vienna so that you can sign it.

[Timothy_Hall]

The 50k per limitation only applies to Hide NAT, due to the limited number of ports available for source port remapping by the firewall so it can determine where return traffic is supposed to go.  Static NAT is 1-1 so there is no need for the ports to be changed by the firewall, as such the 50k limit does not apply.  Also keep in mind my update in this thread:

https://community.checkpoint.com/message/8238-re-r8010-hide-behind-many-question?commentID=8238#comm... 

I am be happy to sign your hardcopy at CPX Vienna Dialungana, and will generally be hanging out at the CheckMates Scavenger Hunt booth in the Expo with Dameon Welch-Abernathy‌ and Valeri Loukine‌.  If I'm not not there try the CheckMates Use Cases speaking track room.

--
"IPS Immersion Training" Self-paced Video Class
Now Available at http://www.maxpowerfirewalls.com

[PhoneBoy]

Actually, there is no "CheckMates" booth in Vienna (last minute change).

Valeri Loukine‌ and will be wandering around the expo area this evening (Monday).

On Tuesday and Wednesday, the "CheckMates lounge" will be in our breakout sessions in Strauss 3.