Domain VPN vs static rout

[Josh_Allen]

I have a ticket open for this, but since I am waiting I thought I would see if anyone here had any ideas. I'm presenting this a little more simply than it is but hopefully it communicates the issue. 

For reasons beyond my control, we have the same subnet present at 2 sites, A (production) and B (Backup). Users at A need to only go to the connection at A and users at B need to only go to the connection at B. Pretty simple so far with just some static routes. 

However, we have a 3rd site (C) that also needs to get to the subnet on A. All 3 sites are in the same checkpoint mesh VPN. Initially I just tried to add the subnet at A to the VPN domain. That worked great for getting C to go to A but what also happened was users at site B started getting their traffic to their subnet at B routed to A as well. After some talk with Checkpoint we discovered that the VPN domain overrides even static local routes, so we needed to figure something else out. 

Fortunately, Site A and C happen to be serviced by the same ISP, and they are able to provide a VRF connection, basically just a private network, between the sites. So problem solved. We take the subnet out of the VPN domain, create a local route to the VRF on site C, a route back to the VRF for return traffic on A, and everything works great. As long as traffic is defined in the VPN it just stays in the VPN from C to A and back again, and if it needs to go to the special subnet, it routes to the VRF from C, Arrives at A, then gets returned Vice versa since it is not defined in the VPN. 

However, we recently changed out the firewalls at site A, and now what seems to be happening is some VPN traffic is working fine, for example ICMP echo request and file share. However other traffic, for example SSH, RDP and HTTPS leave C on the VPN, arrive at A on the VPN, and never make it back. If I take the static route back to C through the VRF out, the traffic does make it back. My conclusion is that for certain services the traffic is just staying in the VPN both ways, but for other services for some reason it is now hitting that static route on the way back and since it didn't come in from that way it can't go back a different way.

Has anyone seen this, where it seems like the Domain VPN is only overriding the static  route sometimes? 

[the_rock]

Hm...cant say I ever experienced that before myself. One thing that came to my mind was setting inside the VPN community to exclude certain services, though most people don't bother changing that.

[Timothy_Hall · Accepted solution]

When you swapped out your firewall did the code version change?  Because this behavior on Firewall A sounds to me like some kind of route/tunnel caching being performed by SecureXL whose code may have changed if you upgraded.  It would be interesting to exclude your VPN tunnels from acceleration via the vpn accel off command on firewall A, but be warned this will cause a disruption in the tunnels while they restart and should probably be done during a maintenance window.  You shouldn't need to turn off SecureXL acceleration on all firewalls, just firewall A. 

Beyond that you'll need to run fw ctl zdebug + drop and see if the Check Point inspection code is dropping the traffic that is not making it back, and if not run packet captures with tcpdump/cppcap to see if the lost traffic is somehow not getting re-encrypted properly and being dumped out an interface in the clear, or even possibly leaving on the wrong interface altogether. 

Also your problem sounds suspiciously similar to this: sk100276: Route based VPN sends traffic across the tunnel in clear text for TCP traffic

or possibly even this: sk165998: VPN traffic dropped with "dropped by vpn_ipsec_decrypt Reason: decryption failure: tunnel ...

[Josh_Allen]

Hey thanks for the idea, yeah it was a change at site A from version 80.40 to 81 as well. I'll try to keep this forum updated with the results when I get a chance. 

[Josh_Allen]

FWIW acceleration did turn out to be the issue. It still is an error with the code so we are debugging with TAC and sending to development. Thank You!

[Chris_Wilson]

I have found the Domain vpn always overrides static routes.  One thing that I have used  to override, is by configuring the vpn_route.conf file on the mgmt server and this get pushed to the firewall.  You define a subnet and the firewall for the site, then that subnet just does local routing.   That might help you or might have before you tried the VRF config.   so, in the file, I would put

<Name of subnet B fw object>  firewallB firewallB force_override