NAT rulematch required drop

[YshayM]

hey,

we have one firewall that every ssh connection that goes through it has a delay of 5 seconds, and it happens only on the first connection, after the first connection the next connections are immediate for few minutes, and after some time with no activity it happens again.

after some troubleshooting i saw this log 3 times when it happened using fw ctl zdebug + drop

fw_log_drop_ex: Packet proto=6 10.0.0.1:6489 -> 10.20.20.20:22 dropped by fw_first_packet_xlation Reason: NAT rulematch required HOLD - vanishing packet;

the gateway in question is R81.20 take 41

currently i am out of options why that happens, i would be grateful if someone could help me. 

[Gaurav_Pandya]

Reason for this error is due to gateway couldn't determine the correct NAT translation for the first packet, it is often due to NAT port exhaustion, misconfigured Hide NAT rules, or hidden protocol issues (like GRE/SIP).

Check logs for "NAT port is not enough": Confirms port exhaustion.

Verify NAT Rules: Ensure the manual or automatic NAT rule is active, correct, and matches the packet source/destination.

Check for Hide NAT Limits: Review if many devices are sharing one IP. You may need to add more IPs to the NAT pool.

Debug Traffic: Run fw ctl zdebug + drop | grep xlate or fw ctl zdebug -m fw + drop xlate to see exact failure reason.

[YshayM]

in debug i see the correct NAT rule is matched, and we dont have any issues with it.

also we dont have NAT exhaustion or something like that.

[Timothy_Hall]

1) Do you have domain objects used in your Access Control policy, especially non-FQDN ones?  If so, and the firewall has URL filtering enabled, you really should replace them with Custom Application/Site objects if they only need to match web-based traffic.

sk184096: First packet delays for around 10 seconds due to pending WSDNSD DNS lookup over TCP

sk182103: Initial packet (SYN or 1st UDP) to a specific subnet is delayed for 6 seconds

2) Also, check your DNS settings in the gateway's Gaia OS, and ensure all configured DNS servers are correct and responding quickly with nslookup/dig.  Removing any slow/nonexistent DNS servers will help a lot.

3) Less likely, but it could be Hide NAT port exhaustion. Search your logs for "NAT Hide Failure" or "exhausted".  You can also view the top 2 NAT pools with the highest utilization live on the cpview screen Advanced...NAT...Pool-IPv4.  You can increase the number of top NAT pools displayed from 2 to up to 200 by tweaking the fwx_alloc_top_pools_num kernel variable.

[YshayM]

we dont have NAT port exhaustion.

we do have some non-FQDN domains, i will take a look on those sk you mentioned, and try to clean as much non-FQDN domains from our policy, it is on production so i believe it will take me few days

i'll give an update after i check it

[Gaurav_Pandya]

You can validate this by placing an SSH connection rule above the existing FQDN rules.
As a best practice, when FQDN‑based rules are in use, all critical connections or high‑priority rules should always be positioned above the FQDN rules in the rule hierarchy.

[YshayM]

i tried it but the issue still persist

[the_rock]

You may wish to open TAC case to check why this happens via remote session.

[the_rock]

I was just about to send you 2nd sk Tim referenced. To me, seems most relevant in your case.