topic Re: Bi-directional NAT is not working post VMSS deployment in Cloud Firewall

Bi-directional NAT is not working post VMSS deployment

Abhishek_Singh1 — Tue, 20 Oct 2020 08:54:59 GMT

Hi All,

For some reason the bi-directional NAT is not working for one of our Destination Natted traffic in VMSS deployment (2 instances) . Return traffic from destination is not able to connect to the original src.

R80.40 - Build 105

Am I missing something ??

Rule -

Original Src- 10.22.x.x-23 , 10.22.y.y-23 ( Network Group - consisting of Network )

Original Dst - 10.22.8.40

Original Port - 443

Xlated Src - Original

Xlated Dst - 10.22.133.10 (Type Static)

Xlated Port- Original

Log -

Cann't see - Additonal Nat Rule -1 , which generally comes in traffic .

@PhoneBoy --- Any help , will be highly appreciated.

Re: Bi-directional NAT is not working post VMSS deployment

Rohit_Raut — Tue, 20 Oct 2020 10:42:42 GMT

Do you see nating in fw monitor?

Re: Bi-directional NAT is not working post VMSS deployment

G_W_Albrecht — Tue, 20 Oct 2020 11:33:19 GMT

R80.40 - Build 105 and which Jumbo HFA ?

Re: Bi-directional NAT is not working post VMSS deployment

Abhishek_Singh1 — Tue, 20 Oct 2020 11:35:05 GMT

HOTFIX_R80_40_JUMBO_HF_MAIN Take: 78

Re: Bi-directional NAT is not working post VMSS deployment

PhoneBoy — Wed, 21 Oct 2020 00:56:23 GMT

Did you read the help with bi-directional NAT?
This option is only relevant for automatic NAT rules.

In any case, not sure I understand what is happening here.
Can you be far more explicit about what you expect, what is actually happening, etc?

Re: Bi-directional NAT is not working post VMSS deployment

Matthias_Haas — Thu, 22 Oct 2020 06:49:55 GMT

do you see a drop/out of state for the return packet (Source: 10.22.133.10, Destination: 10.22.x.x-23 , 10.22.y.y-23) ?

Re: Bi-directional NAT is not working post VMSS deployment

Abhishek_Singh1 — Thu, 22 Oct 2020 12:43:38 GMT

This is observed for EAST-to-WEST return traffic .

Basically , the incoming packet from CP internal LB is coming to 1 instance of GW (eth1) and the return traffic is coming on the other GW (eth1) via the CP internal LB .

I did added the LocalGatewayInternal ( Xlated Source - type Hide NAT) but still no luck .

Note - the interesting traffic over here is FTP - Passive .

Re: Bi-directional NAT is not working post VMSS deployment

Abhishek_Singh1 — Thu, 22 Oct 2020 12:45:23 GMT

Yup , exactly.

Re: Bi-directional NAT is not working post VMSS deployment

Matthias_Haas — Fri, 23 Oct 2020 07:43:06 GMT

does this Destination NAT work for other connections?

With two firewall instances I would expect that there is a chance of 50:50 that it will work as the internal LB is selecting the firewall instance based on the IPs of the packet and as NAT is modifying one of these IPs, the loadbalancer could selecting a different instance for the return packet (as in your case).

But if so, I would expect more problems, not just for FTP -Passive.

Doing a Xlated Source NAT with LocalGatewayInternal should solve the problem.

Did you check the log to see that the Source IP is modified correctly ?

A "dynamic_objects -l" on the instance should show you the IP attached to LocalGatewayInternal

If the Source NAT is correct, is a UDR used for the subnet in which your destination 10.22.133.10 is deployed ?

If so, please make sure, that the subnet of the internal Inferface of the VMSS is routed directly (next hop type Virtual network) and not forwared to the internal LB otherwise we would have the same problem

Re: Bi-directional NAT is not working post VMSS deployment

Abhishek_Singh1 — Fri, 23 Oct 2020 11:28:47 GMT

Hi @Matthias_Haas ,

Please find my response inline to your comments (in italic font)--

does this Destination NAT work for other connections? --- we just have this single application traffic utilizing DNAT , no other used case so far.

But if so, I would expect more problems, not just for FTP -Passive.

Doing a Xlated Source NAT with LocalGatewayInternal should solve the problem.

Did you check the log to see that the Source IP is modified correctly ?

A "dynamic_objects -l" on the instance should show you the IP attached to LocalGatewayInternal

--- Yes, the source NAT is happening properly , have validated the translated Src to be fw eth1 IP and Dst to be 10.22.133.10 using tcpdump & fw mon

If the Source NAT is correct, is a UDR used for the subnet in which your destination 10.22.133.10 is deployed ? -- Yes UDR is added on the FW internal interface eth1 subnet for destination 10.22.133.10 via Virtual network (next hop)

Re: Bi-directional NAT is not working post VMSS deployment

Abhishek_Singh1 — Fri, 23 Oct 2020 11:32:10 GMT

also we enable one service on a random port 2222 , it works fine ... have observed issue with just Passive FTP connection ( 21 , data-port (2000 - 4000)) ...

Already allowed in access rule .

No issue observed for this FTP service in single instance (VMSS solution), or the earlier deployed cluster gateway 😞

Re: Bi-directional NAT is not working post VMSS deployment

Matthias_Haas — Fri, 23 Oct 2020 11:42:12 GMT

Abhishek,

<Yes UDR is added on the FW internal interface eth1 subnet for destination 10.22.133.10 via Virtual network (next hop)

I mean the UDR attached to the subnet of the destination 10.22.133.10 (relevant for the return packet)?

Update: make sure, that the eth1 subnet is not forwarded to the internal LB

Re: Bi-directional NAT is not working post VMSS deployment

Abhishek_Singh1 — Fri, 23 Oct 2020 11:42:46 GMT

Ohk , thats UDR is pointing towards VMSS internal LB . ( Hence , we thought of adding Source Hide NAT to overcome asymmetric of return traffic).

Basically for all the subnets , as per design we have default route pointing towards VMSS internal LB so that checkpoint can inspect the traffic.

If in this case , I change the route for CP subnet directly vis Azure fabric , that would lead to CP missing the return traffic , isn't it ??

Re: Bi-directional NAT is not working post VMSS deployment

Matthias_Haas — Fri, 23 Oct 2020 11:48:27 GMT

<If in this case , I change the route for CP subnet directly vis Azure fabric , that would lead to CP missing the return traffic , isn't it ??

exactly (just for the internal/eth1 segment of the VMSS)

Re: Bi-directional NAT is not working post VMSS deployment

Matthias_Haas — Fri, 23 Oct 2020 15:24:35 GMT

that‘s the route you need to add

Re: Bi-directional NAT is not working post VMSS deployment

Abhishek_Singh1 — Wed, 28 Oct 2020 11:35:48 GMT

Yes , thats done as per the doc. Routing wise we are sorted , no split,asymmetric scenerio .

Finally with end-to-end packet captures , realized Checkpoint is specifically dropping 227 PASV response towards client whenever we enable SNAT .

Zdebug -

@;266774109;[cpu_2];[fw4_1];fw_log_drop_ex: Packet proto=6 x.x.x.x:21 -> y.y.y.y:17024 dropped by fw_post_vm_chain_handler Reason: Handler 'ftp_code' drop;

kernel debug -

@;237812014;26Oct2020 7:53:08.451195;[cpu_1];[fw4_2];fw_xlate_scan_ftp_cmd: 227 command;

@;237812014;26Oct2020 7:53:08.451196;[cpu_1];[fw4_2];fw_xlate_anticipate_cookie: changing packet to <y.y.y.y, 9dd>;

@;237812014;26Oct2020 7:53:08.451197;[cpu_1];[fw4_2];fw_xlate_update_packet: new field (len=16, delta=-1) is 'y,y,y,y,9,221';

@;237812014;26Oct2020 7:53:08.451199;[cpu_1];[fw4_2];fw_xlate_update_length: Got -3 from fwseqvalid_reg_offset_deltas;

@;237812014;26Oct2020 7:53:08.451200;[cpu_1];[fw4_2];fw_post_vm_chain_handler: handler function returned action DROP;

@;237812014;26Oct2020 7:53:08.451202;[cpu_1];[fw4_2];fw_log_drop_ex: Packet proto=6 y.y.y.y:21 -> x.x.x.x:61627 dropped by fw_post_vm_chain_handler Reason: Handler 'ftp_code' drop;

@;237812014;26Oct2020 7:53:08.451204;[cpu_1];[fw4_2];After POST VM: <dir 1, y.y.y.y:21 -> x.x.x.x:61627 IPP 6> (len=87) TCP flags=0x18 (PUSH-ACK), seq=3417305397, ack=951427193, data end=3417305444 ;

@;237812014;26Oct2020 7:53:08.451205;[cpu_1];[fw4_2];POST VM Final action=DROP;

@;237812014;26Oct2020 7:53:08.451205;[cpu_1];[fw4_2]; ----- Stateful POST VM outbound Completed -----

@PhoneBoy --- We have already opened a TAC case - SR#6-0002342606 , but not getting proper attention . Can you please suggest and highlight this to appropriate Checkpoint resources . Thanks in Advance 🙂

Re: Bi-directional NAT is not working post VMSS deployment

Abhishek_Singh1 — Wed, 28 Oct 2020 11:41:49 GMT

Just to add on --- Have tried all the permutation & combination of Global Nat settings , Custom TCP service (with protocol as None) , sks available on internet search with this ftp_code drop , everything with no luck...

Re: Bi-directional NAT is not working post VMSS deployment

PhoneBoy — Wed, 28 Oct 2020 14:22:50 GMT

Have you tried: https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk112001

Re: Bi-directional NAT is not working post VMSS deployment

Abhishek_Singh1 — Thu, 29 Oct 2020 09:28:01 GMT

Yes , gone through that SK ... we aren't using FTP over TLS 😞 . Plus , the client in our case is never receiving an response from Server . ( 227 PASV response from server is not relayed back to client , CP is dropping it)

Have already tried with custom TCP service , allowing port-21 , >1024 (with None as protocol) --- with no luck.

Is there a way I can force Checkpoint to bypass the standard inspection for this traffic ??

Re: Bi-directional NAT is not working post VMSS deployment

Abhishek_Singh1 — Sat, 31 Oct 2020 11:57:47 GMT

@Timothy_Hall -- do you have any insights on this??