IPsec connection between Windows servers fails behind Checkpoint Maestro

[fourcly]

Hello everyone,

We currently have the problem that we want to establish an IPsec connection between two Windows servers in order to encrypt the traffic between them (the IPsec connection is not on the checkpoint, but directly between the two servers). However, no connection is established between the servers. We have a Checkpoint Maestro cluster between the two networks. (R81.20) If we hang the DC (on our side) in front of the checkpoint as a test, then the connection works, if we put it behind the checkpoint again, it doesn't work again. Could this be a MTU/MSS problem?
We have no NAT and no VPN tunnels active on the checkpoint.

In the Wireshark trace I see a lot of ISAKMP Identity Protection (Main Mode) packets and a lot of Unknown (243,244,246) packets.

Any ideas?

Thanks!

[AkosBakos]

What is you MTU setting? the default 1500 on the participating interfaces?

[fourcly]

correct, the MTU on the interfaces is 1500

[AkosBakos]

Here is a good tool to determinate the neccessary MTU size

If you calculate, what is the result?

[fourcly]

when I calculate it there, I get Header size (overhead): 58 bytes
MTU: 1442 bytes, but don't we still have to calculate the IPsec ESP overhead here?

[AkosBakos]

Hi @fourcly 

Yes, but I don't know what is the exact number for ESP

And one more thing: we had issues with RA VPN on MAESTRO. Logn story short: if only one SGM was in use, the VPN issue disappared.

You have the ability to do this kind of test? This would be a good step to narrow down the issue. Maybe this belongs to maestro and not MTU.

And the  layer 4 distribution more is enabled on the Security Group?

Akos

[fourcly]

Hi,

yes, I'll give it a try and leave only one SGM active.

the output of "show distribution configuration" is Distribution Mode: N/A.

Paul 

[fourcly]

Hi @AkosBakos 

I have just tested with only one active SGM, but with the same error pattern. The layer 4 distribution is set to auto-topology (per-port).
Should I now try to adjust the MTU on the incoming and outgoing interface?

Paul

[AkosBakos]

Hi @fourcly 

The MTU is a curious thing. There is no exact suggestion in the guides.

Akos

[AkosBakos]

Here is better calculator: https://travelping.github.io/encapcalc/

[Timothy_Hall]

MTU issues usually only come in to play once all IKE negotiations are complete and IPSec starts, but it doesn't sound like you are getting that far.  A few things:

1) Are you sure no NAT is configured in Maestro?  If there is the two IKE peers will shift from UDP/500 to UDP/4500 at IKEv1 Main Mode packet 5 or at IKEv2 packet 3 (not 100% sure on the exact packet where the NAT-T switch happens for IKEv2).  Are you seeing port 4500 at any point?

2) If there is no NAT, it may be a distribution issue of some kind.  To confirm try forcing UDP ports 500/4500 traffic along with ESP/50 to always be handled by the SMO via the asg_excp_conf command as detailed below, although it sounds like you tried it with only one SGM active and the problem persisted so that is not a distribution issue.

https://sc1.checkpoint.com/documents/R81.20/WebAdminGuides/EN/CP_R81.20_Chassis_AdminGuide/Content/T...

3) Another option is to Fast Forward UDP ports 500/4500 and IP Proto 50 (ESP) directly through the Orchestrator, since the VPN traffic is encrypted and can't be inspected anyway:

https://sc1.checkpoint.com/documents/R81.20/WebAdminGuides/EN/CP_R81.20_Chassis_AdminGuide/Content/T...

Beyond that we'll need to see a packet capture of the IKE packets to figure out what is going on.