Please help me understand this NAT behavior

[HansKazan]

Dear CheckMates

I am writing this post in the hopes that someone can help me understand this behavior. All appliances involved are running R82 JHF 118.

Situation: Policy Migration for migration to Maestro SGs with big bang

Old Policy has about 800 object NAT (automatic rules) with installation target on "oldfwcluster"

Actions Taken:

Cloned policy and changed installation target to "newfwcluster" and observed all automatic NAT rules of "oldfwcluster" staying in the policy package (Expectation would have been for them to no longer be part of the policy package).

I recreated all these object NAT rules manually and put them above the Automatically Created NAT rules (new rules have highest priority) as the policy had to be prepared before the migration as downtime could only be short.

Observation Post Migration:

In the logs we can verify that we are hitting the newly created manual NAT rules, yet the client is furious that the some expected traffic is not working. When changing the objects to be automatically NAT by changing the installation target from "oldfwcluster" to "newfwcluster" it appears to be working. 

When verifying the logs, we observe that the manual NAT rules are still being hit, but the client is adamant that the problem has been resolved.

Is there anyone that can make sense of this, is the object being fetched incorrectly at one point in the establishment of the connection that is post-logging? Is this a known limitation I am not aware of? Or does OBJECT NAT have absolute PRIORITY without being reflected in logging? Any input is welcome!

[Chris_Atkinson]

Was the type of NAT set incorrectly static vs hide or proxy-arp issues perhaps?

Is the issue fixed or has it moved / subsided temporarily i.e. Hide port failure / exhaustion depending on distribution mode?

[HansKazan]

Rules were correctly recreated with both Static and Hide NAT variants. The Manual rule is still being matched according to the logs. Proxy-ARP is not used in the environment.

When changing the object NAT to install on the new installation target the issue appears to be resolved. The rulebase now has an identical rule above the automatic one, with the manual rule being hit according to the logs. With this setup, the NAT issue is resolved. 

The manual rule is now redundant and can be removed. However, I prefer to get rid of the Object NAT. Ongoing discussion at the moment.

[emmap]

Without a deep dive into the setup I can't really say why the object NAT is hitting when the manual one isn't, but in future the more reliable way to do this might have been to use the 'replace' option in 'where used' to change the NAT gateway on the objects in the change window and push policy to the new SG. If the policy isn't installed to the old gateway then it will still have its copy of the policy with itself doing the NATs so rollback is still fast if that would be a concern here.

[HansKazan]

I agree, but the rollback window was too short to manually change 800 object NATs without an API script (I understand with the old policy installed we would have more time, but part of the requirements). Given the information we had at the time we did not expect our method to result into issues.

[Hugo_vd_Kooij · Accepted solution]

In my experience there are a few things automatic NAT object do for you that you have to do additionaly if you use manual NAT rules. 

Fixing ARP is the most prominent one.

Also there are certain protocols that will ignore manual NAT rules. This was something I found out in a deep-dive debug with the TAC/R&D VOIP team in the past. The exact behavious differs per version but it still is something I am aware of.

So there are multiple options that might aply to your issue here. 

In your caes preparing a script to change the policy through the API is propably the fastest way to do your next conversion attempt.

[HansKazan]

Thank you for your feedback. I was indeed not aware that Object NAT did more for you than advertised in the SmartConsole window.

The protocol in question is unknown to me and using the high port of TCP/20001.

I must stress that according to the logging, we were correctly hitting the manually created NAT rule. If Proxy-ARP is a free gift from Object NAT it could potentially explain it. However, the environment does not appear to need Proxy-ARP to function for these traffic flows.2

The client does a mix and match of Object NAT and manual NAT depending on which administrator is creating NAT rules. There is no Proxy-ARP configured on any of the appliances, leading me to believe this may then have been a hidden requirement for the traffic flow that not even the administrators would have been aware of.

Viewing sk30197 it does mention the need to consolidate manual ARP entries for Manual NAT. I will investigate if ARP entries are required for these traffic flows. 

Many thanks for the lead!

[HansKazan]

I should have just actually read the documentation properly.

https://sc1.checkpoint.com/documents/R82/WebAdminGuides/EN/CP_R82_SecurityManagement_AdminGuide/Cont...

While it is stated that static-ARP MUST be configured for manual static entries. I have been getting away without the need to do so until now. Issue been confirmed to have been the lack of Proxy ARP.

In hindsight I should have known this, thank you for the lesson