topic Encryption of IKE (ESP) over VPN in Firewall and Security Management

Encryption of IKE (ESP) over VPN

bu007 — Wed, 16 Apr 2025 20:14:59 GMT

Hello

did anyone ran into an issue where 3rd part VPN tunnel traffic is dropped between two CP gateways that have a VTI based VPN between them. This describes my situation https://support.checkpoint.com/results/sk/sk177715.

So the two CP gateways that have VTI tunnel between them are serving just as "routers" between 3rd party GWs that are trying to establish their own VPN tunnel. We ran into this issue before as I will describe. Mind you this is a VTI tunnel that connects thousands of host with no issues for years now.

We used to get error "Encryption Failure: Failed to enforce VPN Policy (11)" up till R81.10 (and solved the issue with "set int encrypt_non_gw_rdp_ike 1" parameter), but when we upgraded to R81.20 this error changed to "Failure preparing tunnel creation, internal error" Routing checks out and ICMP traffic betwen 3rd party gateways is routed via VTI tunnel, so its not the crypto map. Tried domain based VPN routeing, exclude ESP services, host routes, nothing helps. Both CPs are R81.20 manged by the same SMS,

CP detect the traffic as ESP. Did anyone had similar issues with IPsec over IPsec on R81.20?

Re: Encryption of IKE (ESP) over VPN

PhoneBoy — Wed, 16 Apr 2025 21:29:34 GMT

Since this broke as a result of an upgrade, your best bet is to engage the TAC if you haven't already.

Re: Encryption of IKE (ESP) over VPN

bu007 — Thu, 17 Apr 2025 07:56:44 GMT

Yeah I did that already. I have a case open, just wanted to check if maybe someone already had this issue and was able to solve it.

Re: Encryption of IKE (ESP) over VPN

G_W_Albrecht — Thu, 17 Apr 2025 09:31:53 GMT

I found sk170141: Site to Site VPN traffic is being dropped for "Failure preparing tunnel creation, internal error"

Re: Encryption of IKE (ESP) over VPN

bu007 — Thu, 17 Apr 2025 10:18:12 GMT

Hello Albrecht, I've seen that SK, routing is OK. I tried adding a host route and domain based routing, It did not help. Thanks for replaying.

Re: Encryption of IKE (ESP) over VPN

Timothy_Hall — Thu, 17 Apr 2025 13:43:06 GMT

Firewall appliance model? Is UPPAK in use? (fwaccel stat). Have personally seen issues with VPN traffic attempting to traverse but not terminating on a gateway with UPPAK enabled.

sk182775: Packet loss (fwconn_key_init_links failed) for ESP packets when using User-Mode SecureXL

No ICMP traffic trough VPN after migration

Re: Encryption of IKE (ESP) over VPN

bu007 — Thu, 17 Apr 2025 20:15:55 GMT

Hello Timothy,

Thank you for your reply. I thought of that to, I did have the "vpn accel off x.x.x.x" command ready but need a maintenance window, which I didn't have, yet. I would like to get as much info as possible, and it is on my "to do" list. ICMP traffic actually works its the ESP packets that are the problem. JHF is 92

6900 appliance

To sum up:

packet is being droped at "iD":

vpn_is_it_encrypted_packet: dir 0, x.x.x.x:0 -> y.y.y.y:0 IPP 50 IPsec packet, but not ours ;
...
vpnk_multik_forward (in): multicore VPN enabled;
...
chain_ipsec_methods_ok: ******************* Illegal interfaces group 0 get_interfaces_group = -8 ifnum = 34 ;
...
Illegal interfaces group 0 get_interfaces_group = -8 ifnum = 34 ; -> no idea atm
...
vpnk_get_mspi_from_opaque: retuned mspi = [fail]
get_msa_by_mspi: mspi [fail] ... returning;
...
vpn_enc_scheme_to_schemname: illegal scheme -1;
...
fw_log_drop_ex: Packet proto=50 ... dropped by vpn_drop_and_log
Reason: Failure preparing tunnel creation, internal error;

So no mspi, no SA.

I had some different issues on R81.10 but is it possible that packet is being handled by IKED instead of VPND?

I've uploaded this logs to TAC just now. I'll post what the problem/solution was.

Re: Encryption of IKE (ESP) over VPN

Timothy_Hall — Thu, 17 Apr 2025 23:50:50 GMT

There were a couple of initial issues with the new iked in R81.20 which assumed all of the IKE-related roles from the much older vpnd, but as long as you are running the latest recommended Jumbo for R81.20 it shouldn't be your problem. You could try temporarily disabling iked with the vpn iked disable command as documented below and see if it clears the problem; you can also disable iked permanently:

sk180252: Route Injection Mechanism (RIM) in R81.10 does not work as expected with LSM satellite gateways

Re: Encryption of IKE (ESP) over VPN

Chris_Wilson — Tue, 27 Jan 2026 23:48:43 GMT

so what was the solution?