VPN community routing not working

[isuckatthis]

Hello smarter than I people!

I have a Checkpoint to Checkpoint VPN. We're using a community. The traffic gets to the Checkpoint FW VPN concentrator and is not routed across the VPN.

How do I troubleshoot this?

The encryption domain on the remote side shows the correct subnets.

I've attached images showing the community, logs showing the traffic is not encrypted, a traceroute showing the FW is sending the traffic back to the router then the router back to the FW then the FW back to the router until the end of time (or TTL expires).

I have no idea how to troubleshoot this. Help! 😞 

[the_rock]

How is routing set in vpn community object?

Andy

[isuckatthis]

We've got it configured as a hub and spoke. We're not using any dynamic routing protocols so we have Route Injection Mechanism (RIM) disabled.

[the_rock]

Which option applies to you here? center only?

Andy

VPN Routing Options

• To center only . No VPN routing actually occurs. Only connections between the satellite gateways and central gateway go through the VPN tunnel. Other connections are routed in the normal way

• To center and to other satellites through center . Use VPN routing for connection between satellites. Every packet passing from a satellite gateway to another satellite gateway is routed through the central gateway. Connection between satellite gateways and gateways that do not belong to the community are routed in the normal way.

• To center, or through the center to other satellites, to internet and other VPN targets . Use VPN routing for every connection a satellite gateway handles. Packets sent by a satellite gateway pass through the VPN tunnel to the central gateway before being routed to the destination address.

[isuckatthis]

Should I see VPN Community encryption domain routes here?

[the_rock]

You should.

Andy

[isuckatthis]

Looks like we have it configured the same as in your screen shot.

[the_rock]

Right, but is that correct? I ask because if there is routing supposed to happen through the tunnel, that that setting would be incorrect.

Andy

[isuckatthis]

Good question and perhaps I'm not understanding your ask. If you're asking dynamic routing protocols, no. We are expecting static routes via the VPN community to handle routing.

I understand that the encryption domain itself handles encryption and routing across the tunnel. Using my original image showing 10.59.78.0/24 within the encryption domain at the remote site, I'm expecting my hub to route the traffic across the tunnel.

Here's a simple diagram of what we're trying to accomplish. The VPN community lists 10.59.78.0/24 on the spoke side. The VPN community lists 10.59.0.0/16 on the hub side.

[the_rock]

Ok, so option you have is fine, BUT, here is your issue, you have supernet problem. 10.59.0.0/16 definitely contains the other subnet. Make sure below options are set to FALSE in guidbedit, push policy, reset the tunnel and try again.

Andy

ike_enable_supernet

ike_p2_enable_supernet_from_R80.20

ike_use_largest_possible_subnets

[isuckatthis]

Thanks Andy! I feel like we're getting closer now!

Do I understand correctly, the more specific mask network isn't automatically used, like in traditional routing?

I'll take a look at dbedit, I've seen someone use it before to hack their way through a problem. "oh, the firewall isn't doing what I want? Let's change registry settings!" 😆

Fingers crossed! Thanks for all your help so far, I'll be back!

[the_rock]

Glad we can help you. Btw, not sure how long you been around CP, but in the old days (R77 and before versions), that was big issue for supernet, specially with Cisco.

What I mean by that is say Cisco expects /28 subnet, but CP would always send largest possible, ie min /24 or larger. Thats why folks always had to go to guidbedit and change those values.

In R80 +, thats not really such a huge problem, but I would still verify.

Andy

[isuckatthis]

I've been around CP for about 4 months. I'm an amateur and hence the name "isuckatthis" 😐

[isuckatthis]

I found them! Two are true and one is global.

Since these are all "ike" tagged, is there any impact anywhere else within Checkpoint when I change these settings? We have no production VPNs from a checkpoint firewall so if this only affects IKE, I feel confident in changing these settings. If this could impact any other functionality, I don't feel comfortable until I understand the full impact.

The second option talks about being global and not a true or false.

[the_rock]

I would 100% change it to false. I had people change this probably 100+ times before and I had NEVER seen an issue, even when they had multiple vpn communities. Worst thing that could happen is if tunnel did go down, you just reset it and its back up.

R82 mgmt has all those values false by default.

Andy

[isuckatthis]

Bummer, no dice.

I set all three values to false, pushed policy to both firewalls, bounced the tunnel. 

Same behavior. The hub does not forward across the tunnel.

[the_rock]

That sucks...o well. Okay, question...is tunnel showing up from vpn tu?

Andy

[isuckatthis]

It is, I have SAs on both FWs

[the_rock]

But from vpn tu, does it show both ike and ipsec sa's as up? If yes, can you run fw ctl zdebug + drop | grep x.x.x.x command? Just replace x.x.x.x with IP thats failing.

Andy

[isuckatthis]

I'm pretty confident that since they're listed here, it means they are UP state.



I ran the fw ctl zdebug command using both the source and destination of our troubleshooting IPs. No traffic shows up, a lot of logs pop up outside of the IPs that I entered, none of it matches either of my IPs

I am running that command from the hub in our case.

I confirmed my guidbedit changes are still set to false

[the_rock]

Hang on, lets confirm something basic. What are vpn domains set on both ends? 

Andy

[isuckatthis]

You got it! I somewhat posted it above but I'll provide more detail. I'm blocking out the names because of the environment we're in, people would be mad if they saw names and especially IPs posted on the Internet.

I've obfuscated (someone took their Sec+ exam 😉 ) some of the text. The names do indeed match.

Behind my hub, I have a loopback on a router with the IP 10.59.40.69. I am trying to ping our remote site switch IP 10.59.78.2. 10.59.78.1/24 is a sub-interface on the remote FW and the default gateway for the switch.

The encdom VPN Domain object is configured for the HUB
The ThompsonTesting VPN Domain object is configured for the SPOKE

[Duane_Toler]

You can't do this.

You have overlapping subnets on both sides.  Your log screenshot shows traffic is "Accepted", when it should show "Encrypted".  You need to fix your VPN domains.  You also need to fix your firewall routing tables; both gateways have routing issues.  Your R1 router must also have a 10.0.0.0/8 route with next-hop of the firewall.  That's why you are seeing ping-pong traffic in your traceroute.

On one gateway, you have 10.59.0.0/16 being sent out via eth4 interface, which includes the 10.59.78.0/24 subnet of the other gateway.  Your external interface is eth1, however.  The firewall won't encrypt packets that are destined towards an internal interface ("internal" is defined as "not-external").

As @Lesley pointed out, you also have one side set for route-based VPN with a VTI, and the other is not.  The other gateway has a 10.0.0.0/8 route going via the VTI, but it has local interfaces within that supernet.  This also won't be very reliable, and it will cause "strange effects" for your network if you do this.

Your route table screenshots are blocking the names of the gateways, so we can't see which gateway is which.

[the_rock]

Hm...I know what you mean, though Im fairly sure this would work, as long as supernet is set to false, which seems like it is now.

Andy

[isuckatthis]

You have overlapping subnets on both sides. From a traditional routing perspective, I don't understand the problem with this. They're different subnets. They overlap but that's why routers use the most specific mask when matching traffic for a routing decision. The network at the remote site is an extension of the main utility network.

 Your log screenshot shows traffic is "Accepted", when it should show "Encrypted".  You need to fix your VPN domains. I don't understand what's wrong with them. I understand the VPN domains are the policy used to direct traffic over the VPN.
Example: domain1 = 10.0.1.0/24   domain2 = 10.0.2.0/24 
src: 10.0.1.10 dst: 10.0.2.20 = traffic goes across the VPN

src: 10.0.1.10 dst: 10.0.3.30 = traffic does not go across the VPN

You also need to fix your firewall routing tables; both gateways have routing issues.  Your R1 router must also have a 10.0.0.0/8 route with next-hop of the firewall.  That's why you are seeing ping-pong traffic in your traceroute.  -  Close, it's a 0/0 and I agree with what you're saying. I'm expecting the FW to receive the traffic from the router and send it over the VPN because the src and dst of the packets match the VPN domains. Instead, the FW is saying, "screw you VPN community, you mean nothing to me, I don't even know you man, you're dead to me, I'm just going to use this /16 and route traffic back to where it came from"

I don't think I'm understanding what people are telling me. I probably need to better understand order of operations on CP. As it stands right now, I 100% believe that the traffic arrives in a firewall, some part of the process is to match the traffic against the VPN community and if the source and destination match, it will choose that tunnel and forward the traffic. Obviously I'm wrong and it's possible everyone is telling me this but it's not clicking.

How do you tell a FW to forward traffic over a VPN built with a community?

On one gateway, you have 10.59.0.0/16 being sent out via eth4 interface, which includes the 10.59.78.0/24 subnet of the other gateway.  Your external interface is eth1, however.  The firewall won't encrypt packets that are destined towards an internal interface ("internal" is defined as "not-external").   From a routing perspective, that seems fine to me, I don't understand the problem. Often, I have summary routes then more specific routes. Maybe I want to traffic engineer so I'll use more specific routes to take certain links, I've never encountered an issue.

As @Lesley pointed out, you also have one side set for route-based VPN with a VTI, and the other is not.  The other gateway has a 10.0.0.0/8 route going via the VTI, but it has local interfaces within that supernet.  This also won't be very reliable, and it will cause "strange effects" for your network if you do this.  This is not fully accurate. The remote side has two tunnels. One terminates on an ASA and is a VTI. Unless you think this is causing all my problems, we don't need to worry about that. With this design, I'm again assuming longest mask match wins the routing decision and also assuming the VPN Domains handle everything with routing, that's what I was taught. 
Simplified, I was taught that the two FW peers receive information from the VPN communities and based on src and dst, it will encrypt and send the traffic across the tunnel. That's not happening so I was taught wrong or misunderstood.

A lot to digest here, thanks man! 😊

[Duane_Toler]

This is gonna get really nitty-gritty detailed, and more than you're asking, but the core answer for "how does the firewall know to do VPN?":

* There is a function inside the VPN code called "VPN tagging"

* This function only gets triggered *AFTER* a packet passes through the routing table (via a "route lookup layer") and begins to egress an external interface

* VPN tagging (in general) only occurs on an external-facing interface (your eth1)

* The VPN tagging function does a bunch of work to figure out what kind of VPN to use (domain-based, route-based; and this is the part where domain-based VPNs override route-based VPNs)

* The VPN peer must also be reachable out an external interface to trigger the IKE process.

* Remote VPN domains and peers must NOT be within the internal network (including anti-spoofing configurations on the interface).  That's because "VPN" by definition is "something external to the local gateway".

The VPN domain (encryption domain) does tell the local firewall what to encapsulate toward the remote peer, but the problem is that a packet is not going towards a remote peer.  The packet is simply returning back to your local network.

Yes, more-specific routes win, but you don't have a more-specific route for 10.59.78.0/24.

Go to the command line of your firewall and run "ip route get 10.59.78.2" and it will show you "via 10.59.40.9 dev eth4" when you (likely) want it to go via eth1.

See the attached screenshot of your routing table.

If you must have this routing table, then you need to add a more-specific static route for 10.59.78.0/24 via 10.57.0.9.  This will trigger the VPN code and start IKE towards the remote peer.  Hence, the packets will be "tagged" for VPN; "untagged" packets are passed in plain text.  Plain text packets are logged in the traffic log as an "Accept" log type.  "VPN tagged" packets are logged as "Encrypt" (or "Decrypt", if the packet is being received).

(hint: you can do this same route trick for Remote Access VPN office-mode subnets, and once a route is in the FIB of the gateway, you can select it for redistribution to dynamic routing protocol)

[Lesley]

Hmm looks like one firewall is route based VPN and other one domain based vpn. 

vpnt20 is route based. You have to pick between them. Either configure route based on both or domain based on both

[the_rock]

Looks like based on last screenshot it would be route based? It all shows 0.0.0.0/0

Andy

[isuckatthis]

More context for you, I didn't include it for simplicity sake.
The spoke has two VPNs. 
1) vpnt20 - goes to a Cisco ASA
2) Checkpoint community VPN magic

vpnt20 to the ASA is working with no issues. I have not considered an issue on the spoke because the hub isn't forwarding to the spoke. As far as the peer having a 0.0.0.10 IP, it's a dynamic IP while we're using Starlink for transport at this remote site, all it has is power basically, well, power and a clear view of the sky.


The hub only has one VPN, using the checkpoint community.

[Duane_Toler]

Then you *definitely* can't do this.  You have 10.0.0.0/8 out that VTI to the Cisco gateway.  However, you are also trying to do a domain-based VPN to your other gateway, also with networks within the 10.0.0.0/8 supernet.  Domain-based VPNs take priority over route-based VPNs (the VTI).  You are risking breaking the VTI to the Cisco gateway.  The reason is hasn't broken yet is because the hub hasn't seen packets to it (yet).

You *CAN* do route-based VTIs on all of the gateways and control packet pathing in the route tables.  This would be your safest option.  Be very careful and very judicious about domain-based VPNs with route-based VPNs.

Your first (and most significant) issue is the routing and networking on your hub gateway.  Having an interface configured like eth4 (the /29 subnet) in addition to a larger network route (10.59.0.0/16) out the same interface will also cause weird effects making debugging/troubleshooting difficult.  If you want eth4 to be a transit VLAN/subnet, then make it a real transit on its own transit subnet.

With the VPN domains configured like you have them, your logs will also be showing various VPN errors.