Checkpoint Site to Site VPN, Tunnel is UP, but traffic doesn't arrive to destination

[cyberluke365]

Hello,

I have a site-to-site VPN between Check Point R81.20 Take 53 and Cisco Meraki. It is a policy-based VPN (IKEv1).

Let's consider this rudimental schema:

1. If I ping the PC (192.168.1.1) from Cisco Meraki (ETH1 192.168.2.254), (by using tcpdump on Check Point) I can see the traffic reaching the Check Point; the PC responds, but I don’t see these replies on Cisco Meraki.

2. If I ping Cisco Meraki (ETH1 192.168.2.254) from the PC (192.168.1.1), I see in the Smart Log that the traffic is encrypted, but, again, it doesn’t reach Cisco Meraki.

Any thought ?

Thank you.

[PhoneBoy]

What does fw monitor say?
Note that fw monitor will show the traffic as it is encrypted/decrypted (though the source/destination IP will change to the tunnel endpoints).

[cyberluke365]

Hello,

I made two tests while running fw monitor -F "192.168.1.1,0,192.168.2.254,0,0" -F "192.168.2.254,0,192.168.1.1,0,0".

1. PING 192.168.2.254 (from PC1 - 192.168.1.1) 

[vs_0][ppak_0] bond1:i[44]: 192.168.1.1 -> 192.168.2.254 (ICMP) len=60 id=62652
ICMP: type=8 code=0 echo request id=1 seq=814
[vs_0][fw_3] bond1:i[44]: 192.168.1.1 -> 192.168.2.254 (ICMP) len=60 id=62652
ICMP: type=8 code=0 echo request id=1 seq=814
[vs_0][fw_3] bond1:I[44]: 192.168.1.1 -> 192.168.2.254 (ICMP) len=60 id=62652
ICMP: type=8 code=0 echo request id=1 seq=814
[vs_0][fw_3] bond1.9:o[44]: 192.168.1.1 -> 192.168.2.254 (ICMP) len=60 id=62652
ICMP: type=8 code=0 echo request id=1 seq=814

ICMP packets never reach Cisco Meraki.

2. PING PC1 - 192.168.1.1 (from 192.168.2.254)

[vs_0][ppak_0] bond1.9:iD[44]: 192.168.2.254 -> 192.168.1.1 (ICMP) len=84 id=0
ICMP: type=8 code=0 echo request id=10670 seq=0
[vs_0][ppak_0] bond1.9:i[44]: 192.168.2.254 -> 192.168.1.1 (ICMP) len=84 id=0
ICMP: type=8 code=0 echo request id=10670 seq=0
[vs_0][fw_0] bond1.9:i[44]: 192.168.2.254 -> 192.168.1.1 (ICMP) len=84 id=0
ICMP: type=8 code=0 echo request id=10670 seq=0
[vs_0][fw_0] bond1.9:I[44]: 192.168.2.254 -> 192.168.1.1 (ICMP) len=84 id=0
ICMP: type=8 code=0 echo request id=10670 seq=0
[vs_0][fw_0] bond1:o[44]: 192.168.2.254 -> 192.168.1.1 (ICMP) len=84 id=0
ICMP: type=8 code=0 echo request id=10670 seq=0
[vs_0][fw_0] bond1:O[44]: 192.168.2.254 -> 192.168.1.1 (ICMP) len=84 id=0
ICMP: type=8 code=0 echo request id=10670 seq=0
[vs_0][ppak_0] bond1:i[44]: 192.168.1.1 -> 192.168.2.254 (ICMP) len=84 id=62732
ICMP: type=0 code=0 echo reply id=10670 seq=0
[vs_0][fw_0] bond1:i[44]: 192.168.1.1 -> 192.168.2.254 (ICMP) len=84 id=62732
ICMP: type=0 code=0 echo reply id=10670 seq=0
[vs_0][fw_0] bond1:I[44]: 192.168.1.1 -> 192.168.2.254 (ICMP) len=84 id=62732
ICMP: type=0 code=0 echo reply id=10670 seq=0
[vs_0][fw_0] bond1.9:o[44]: 192.168.1.1 -> 192.168.2.254 (ICMP) len=84 id=62732
ICMP: type=0 code=0 echo reply id=10670 seq=0
[vs_0][fw_0] bond1.9:O[44]: 192.168.1.1 -> 192.168.2.254 (ICMP) len=84 id=62732
ICMP: type=0 code=0 echo reply id=10670 seq=0
[vs_0][fw_0] bond1.9:Oe[44]: 192.168.1.1 -> 192.168.2.254 (ICMP) len=84 id=62732
ICMP: type=0 code=0 echo reply id=10670 seq=0

ICMP packets reach PC1, but replies never come back Cisco Meraki.

[the_rock]

So, just for the context, in fw monitor, though Im sure you may know this, but in case you did not...D means decrypted and E means encrypted, but if you say packet never comes back, did you do basic zdebug on CP to see if anything possibly gets dropped?

What do logs show on Meraki end?

On a side note, is this domain or route based? How are enc domains configured? I ask, because IF you have combination of hosts/subnet, then you can set tunnel mgmt tab inside vpn community in smart console as "per gateway". Happy to do remote and assist, if you are allowed to.

Andy

[cyberluke365]

Hello @the_rock ,
thank you for your useful information. Well, I'll provide more details.

1. Let's start with fw monitor. This is the complete trace for 1 ICMP packet - PING from PC to Cisco Meraki

[vs_0][ppak_0] bond1:i[44]: 192.168.1.1 -> 192.168.2.254 (ICMP) len=60 id=62769
ICMP: type=8 code=0 echo request id=1 seq=872
[vs_0][fw_1] bond1:i[44]: 192.168.1.1 -> 192.168.2.254 (ICMP) len=60 id=62769
ICMP: type=8 code=0 echo request id=1 seq=872
[vs_0][fw_1] bond1:I[44]: 192.168.1.1 -> 192.168.2.254 (ICMP) len=60 id=62769
ICMP: type=8 code=0 echo request id=1 seq=872
[vs_0][fw_1] bond1.9:o[44]: 192.168.1.1 -> 192.168.2.254 (ICMP) len=60 id=62769
ICMP: type=8 code=0 echo request id=1 seq=872
[vs_0][fw_1] bond1.9:O[44]: 192.168.1.1 -> 192.168.2.254 (ICMP) len=60 id=62769
ICMP: type=8 code=0 echo request id=1 seq=872
[vs_0][fw_0] bond1.9:Oe[44]: 192.168.1.1 -> 192.168.2.254 (ICMP) len=60 id=62769
ICMP: type=8 code=0 echo request id=1 seq=872
[vs_0][ppak_0] bond1.9:Oe[44]: 192.168.1.1 -> 192.168.2.254 (ICMP) len=60 id=62769
ICMP: type=8 code=0 echo request id=1 seq=872

2. This is the complete trace for 1 ICMP packet - PING from Cisco Meraki to Check Point

2.A. It arrives to Check Point:

[vs_0][ppak_0] bond1.9:iD[44]: 192.168.2.254 -> 192.168.1.1 (ICMP) len=84 id=0
ICMP: type=8 code=0 echo request id=10926 seq=0
[vs_0][ppak_0] bond1.9:i[44]: 192.168.2.254 -> 192.168.1.1 (ICMP) len=84 id=0
ICMP: type=8 code=0 echo request id=10926 seq=0
[vs_0][fw_0] bond1.9:i[44]: 192.168.2.254 -> 192.168.1.1 (ICMP) len=84 id=0
ICMP: type=8 code=0 echo request id=10926 seq=0
[vs_0][fw_0] bond1.9:I[44]: 192.168.2.254 -> 192.168.1.1 (ICMP) len=84 id=0
ICMP: type=8 code=0 echo request id=10926 seq=0
[vs_0][fw_0] bond1:o[44]: 192.168.2.254 -> 192.168.1.1 (ICMP) len=84 id=0
ICMP: type=8 code=0 echo request id=10926 seq=0
[vs_0][fw_0] bond1:O[44]: 192.168.2.254 -> 192.168.1.1 (ICMP) len=84 id=0
ICMP: type=8 code=0 echo request id=10926 seq=0

2.B. PC replies:

[vs_0][ppak_0] bond1:i[44]: 192.168.1.1 -> 192.168.2.254 (ICMP) len=84 id=62773
ICMP: type=0 code=0 echo reply id=10926 seq=0
[vs_0][fw_0] bond1:i[44]: 192.168.1.1 -> 192.168.2.254 (ICMP) len=84 id=62773
ICMP: type=0 code=0 echo reply id=10926 seq=0
[vs_0][fw_0] bond1:I[44]: 192.168.1.1 -> 192.168.2.254 (ICMP) len=84 id=62773
ICMP: type=0 code=0 echo reply id=10926 seq=0
[vs_0][fw_0] bond1.9:o[44]: 192.168.1.1 -> 192.168.2.254 (ICMP) len=84 id=62773
ICMP: type=0 code=0 echo reply id=10926 seq=0
[vs_0][fw_0] bond1.9:O[44]: 192.168.1.1 -> 192.168.2.254 (ICMP) len=84 id=62773
ICMP: type=0 code=0 echo reply id=10926 seq=0
[vs_0][fw_0] bond1.9:Oe[44]: 192.168.1.1 -> 192.168.2.254 (ICMP) len=84 id=62773
ICMP: type=0 code=0 echo reply id=10926 seq=0
[vs_0][ppak_0] bond1.9:Oe[44]: 192.168.1.1 -> 192.168.2.254 (ICMP) len=84 id=62773
ICMP: type=0 code=0 echo reply id=10926 seq=0

I looked at the tables reported in "fw monitor" section of R81.20 CLI Reference Guide in order to understand the meaning of "i, I, O,..." (as per your suggestion).

3. Questions:

3.1. I'm not sure what does "Oe" mean; something like Post-Outbound+Pre-Outbound VPN ?
3.2. By the traces (I'm focusing on traffic initiated by PC - point 1 - or reply traffic from PC - point 2) I see the "E" is missing; so the packet isn't encrypted...is it correct ? If it is correct, so why in SmartLog I'm seeing these packets as "Encrypted" (those initiated by PC - point 1):

Now I provide more info about this site-to-site VPN:

1. This VPN was setup long time ago (it's not brand new); this issue started just few days ago
2. We have other VPNs configured at the same manner (Check Point- Cisco Meraki) with no issue
3. No changes were made on Cisco Meraki or Check Point
4. On Cisco Meraki I see WAN packets exchanged by Cisco Meraki and Check Point (related to site-to-site VPN); (tunnel) packets leaving Cisco Meraki (point 2 above); but no (tunnel) packets from Check Point.

These are site-to-site details:

Encryption Method: IKEv1 (policy-based)

IKE - Phase 1

Encryption Algorithm: AES-128

Data Integrity: SHA1

Diffie-Hellman group: 2 (1024 bit)

Renegotiation: 480 minutes

IPSec - Phase 2

Encryption Algorithm: AES-128

Data Integrity: SHA1

PFS: Off

Renegotiation: 28800 seconds

Check Point

VPN Domain: 5 IP class

Cisco Meraki

VPN Domain: 1 IP class

I have learned that colleagues at Site B have experienced Internet connectivity slowdown issues which led them to open a support ticket with the local ISP. I am concerned that the ISP may have made changes (to their infrastructure) that would explain the described behavior.

However, I would like to be certain that the problem does not lie with Check Point.

[the_rock]

All valid questions @cyberluke365 . So Oe flafg means Post Outbound, encrypted, so it means leaving CP encrypted. What happens to it after, I have no idea.

Andy

[RS_Daniel]

Hello,

Yes, Oe means Outbound before encrypt, check this link:

https://community.checkpoint.com/t5/Security-Gateways/R80-x-Performance-Tuning-and-Debug-Tips-fw-mon...

It seems to me that firewall is encrypting the packets. You could check if some new NAT rule is being applied to this traffic, that should be reflected on logs. If there is no NAT, i would do a ping from 192.168.1.1 to 192.168.2.254 with size 500 bytes, and check that traffic on external interface, you should filter tcpdump with the peer ip address, esp or nat-t, and greater than 500. The size of the packet is just a way to identify them after encryption, so you could use any size. If traffic does not leave the firewall, check drops with fw ctl zdebug drop. If they leave the firwall, check the meraki device. If you say it was working before it should be ok, but i would double check which encryption domain are being negotiated in phase 2 (vpn tu tlist). Also, as always... should open a case with TAC. hth.

regards

[cyberluke365]

Hello @RS_Daniel,

thank you for sharing the link.

Could you please explain your statement "it seems to me that firewall is encrypting the packets" ? This part of the table at the link you provided:

The latest packet from fw monitor (PING) reports "Oe". Comparing it with table above, if I'm not mistaken, this indicates Pre-Outbound VPN, Outbound before encrypt. So it isn't encrypted yet, as the "OE" state is missing. It seems to me that packet is stuck between the Pre-Outbound VPN and Post-Outbound VPN, correct ? However from SmartLog I see the ICMP packets as encrypted (screenshot I post before). So maybe there is something wrong in the logic I wrote here.

I confirm there is no NAT. The ping -l 500 192.168.2.254 returns Request timeout. While doing it, I run the tcpdump of external interface and this is the capture results filtered by the public IP of Site B:

Legend

• *.66 - Public IP of Site B
• *.30 - Public IP of Site A

I also opened a case with TAC; I'm waiting for their first contact.

[the_rock]

What it means essentially is that traffic is going through the tunnel, hence the Oe flag in fw monitor.

Andy

[RS_Daniel]

Hello,

Yes, it is correct that Oe is before encryption, but if gateway is tagging this packet for encryption it is a good sign, it is normal not to see OE because the IP's change, they are encapsulated and you can only see the public IP's instead (that is why the tcpdump trick helps us to check if the packet is leaving). On the other hand, on the capture you provided, how long are ESP packets? if they are 548 bits or something like that, that is your ping being encrypted and sent to the remote peer, that would mean the firewall is doing its job.

Regards

[cyberluke365 · Accepted solution]

Hello @RS_Daniel, @the_rock, @PhoneBoy 

This morning, the site-to-site VPN started working again. I didn’t change anything in the configuration, and the investigation conducted (thank you for your useful information) confirms that the issue was not attributable to Check Point or Cisco Meraki but likely 'in the middle' (that is, within the Internet connectivity).

Thank you.

[the_rock]

Thats great to know!

Andy

[PhoneBoy]

When the packet hits OE, it will be fully encrypted (source of local gateway, destination of remote gateway).
fw monitor will only show it if the filter includes those IP addresses (otherwise, it won't show it).

[the_rock]

Also, forgot to ask again, did you make sure what I mentioned in my previous post? If you use combination of hosts/subnets. you should set tunnel mgmt tab in vpn community in smart console as "per gateway" and install policy.

Andy

[cyberluke365]

Hello @the_rock,
VPN Domain (on both sites) are subnets, no hosts. VPN Tunnel Sharing is set to One VPN tunnel per subnet pair.

[the_rock]

K, no issues on that front then.

Andy

[PhoneBoy]

When you initiate the ping, do you see the outgoing IPsec traffic leaving your local gateway?
If you do, and given it appears on your end the traffic is being properly encrypted (as evidenced by the Oe packet in fw monitor), the problem is unlikely to be on your end.