"Decrypted in community" vs "Traffic Accepted"

[johnnyringo]

I've brought up two site-to-site IPSec VPNs between a Cisco IOS router and two different CheckPoint R80.30 gateway clusters in GCP.   The tunnels are route-based, and both showing up/up on the Cisco end with valid 0.0.0.0/0 SAs generated.  However, while the first VPN is passing traffic just fine, the second is not.  I see the traffic leaving the Cisco going over the tunnel interface but never making it to be server behind the checkpoint.

On the working tunnel, the CheckPoint logs show the VPN -> Decrypt with "Decrypted in community" and the name of the VPN community in the message.  

On the non-working tunnel, CheckPoint logs show Firewall -> Accept.  Almost as if the traffic never went through a VPN.

I've double-checked settings both on the Gateway and also the VPN Communities - they look the same.  I've also verified VPN domains on the gateways and they look correct.  What could explain this difference?  

[PhoneBoy]

Did you verify the traffic actually came over a VPN (like with a tcpdump or similar)?

Accept in this context implies "not encrypted."

[johnnyringo]

On the other side (Cisco ISR), I can see the packets being sent over the tunnel interface and the "pkts encaps" in the IPSec SA incrementing.  

On the CheckPoint, if I do a tcpdump on eth0 (external/internet interface) I see activity.  But on eth2 (Internal interface) I see nothing.

[PhoneBoy]

You need to more precisely describe "activity" here.

[johnnyringo]

"activity" meaning udp/4500 traffic on the CheckPoint's external interface.  A simple ping going over the tunnel shows up like this:

[Expert@gcp-checkpoint-member-a:0]# tcpdump -i eth0 -n port not 80 and port not 443
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
21:44:49.376705 IP 203.0.113.251.ipsec-nat-t > 172.16.2.26.ipsec-nat-t: UDP-encap: ESP(spi=0xcd7c75b7,seq=0x12b), length 132
21:44:50.410255 IP 203.0.113.251.ipsec-nat-t > 172.16.2.26.ipsec-nat-t: UDP-encap: ESP(spi=0xcd7c75b7,seq=0x12c), length 132
21:44:51.449244 IP 203.0.113.251.ipsec-nat-t > 172.16.2.26.ipsec-nat-t: UDP-encap: ESP(spi=0xcd7c75b7,seq=0x12d), length 132

[PhoneBoy · Accepted solution]

Ok, so the traffic is getting to the gateway, clearly, but it's most likely getting dropped.
What does fw ctl zdebug drop say?

[johnnyringo · Accepted solution]

Now that's useful...

@;2147206;[cpu_3];[fw4_0];fw_log_drop_ex: Packet proto=6 192.168.1.19:33860 -> 10.22.33.44:80 dropped by fw_first_packet_xlation Reason: Dynamic object is already being resolved;

Due to how routing works inside GCP and some dependencies beyond our control, this flow is NAT'd to the gateway's first internal network (3rd NIC) using a dynamic object, which had not been created on this particular gateway.  

I created the dynamic object and traffic is flowing now.  In SmartConsole, I see "Decrypted in Community" which is expected with VPN traffic.

[PhoneBoy]

Nice, glad you found the issue.