IKE Failure on Site to site IPSec VPN with Zyxel USG - open for suggestions

[Daniel_Cimpeanu]

Hi,

I have recently encountered a somewhat annoying behavior on an IPSec VPN tunnel between a 3100 appliance running Gaia R80.30 and a Zyxel USG 60

Topology is somewhat simple: Checkpoint with internal LAN behind it, attempting communication through IPSec VPN with a NAT-ed LAN behing a Zyxel USG at customer side (the NAT comes due to address restrictions in my Checkpoint environment).

LAN3 cannot be migrated to LAN2 range due to limited onsite resources. Checkpoint doesn't know anything of LAN3 range (and shouldn't, as that is what the customer is NATing to on their side).

 

Here comes the fun part:

- LAN3 hosts accessing LAN1 servers works fine for permitted services.

- LAN1  hosts accessing LAN2 IP addresses (which are NATed to LAN3 range) are rejected with IKE Failure - "encryption failure: Error occurred" message

NAT definitions and access policies on USG have been checked, rechecked, defined and redefined - what am I missing? 

I'm running IKEv1 and the higher encryption - tried lowering it as well but without success (shouldn't be the one causing this anyway)

I'll take any suggestions I can get, I'm running out of ideas on my own 🙂 

Thank you!  

[Timothy_Hall]

You should be getting another error log separate from "encryption failure" which just indicates that the packet couldn't be encrypted because there was not a working tunnel to put it into; that message is just a symptom of the problem and not the cause.  Can pretty much guarantee you are getting a Quick Mode Phase 2 failure due to the Check Point trying to propose overly-large subnets/Proxy-IDs; Zyxel is very picky about what subnet proposals it will accept from peers, whereas the Check Point is much more tolerant about what it will match.  That is why tunnels initiated from the Zyxel side work but the other direction does not.  See this article:

https://community.checkpoint.com/t5/SMB-Appliances-and-SMP/VPN-problem-Phase-2-Quick-Mode-Received-N...

[Daniel_Cimpeanu]

Thank you Tim! Indeed the Zyxel peer replies to the key install with "No proposal chosen"

I tried going through the links presented in the thread you have referenced.  Encryption algorithms match on both ends. 

One thing I have noticed in the Key install step send by the Checkpoint is that the 3100 is including in the IKE IDs the local Checkpoint LAN and the local Zyxel LAN range (LAN3), not the one that Checkpoint should use (LAN2 that should further be NATed to the Zyxel local LAN3 range) - if that makes sense.

[Timothy_Hall]

Unless you are using R80.40+ which allows per-community VPN domains, you'll need to hand edit the appropriate user.def* config file on the SMS to ensure the Check Point proposes exactly what the Zyxel is expecting for subnets/Proxy-IDs in Phase 2.  By default Check Point will "roll up" or aggregate adjacent subnets and propose the largest possible subnets it can, which is not what the Zyxel is expecting.  See Scenario 1 of this SK:

sk108600: VPN Site-to-Site with 3rd party