AWS to Onsite VPN Failing to work

[Secret-goblin-5]

We have a HA pair of firewalls onsite, our network uses static routes as it's pretty simple.

Some of our services are in AWS using a VPN tunnel, when this tunnel goes down we tend to get about 5 minutes of downtime, which is unacceptable.

Our support calls have come to the conclusion we should be using VTI tunnels with BGP as best practice, and it has my task to implement this.

The issue I have is that following instructions https://support.checkpoint.com/results/sk/sk108958 and modifying them to work with HA using post https://community.checkpoint.com/t5/Security-Gateways/VTI-interface-with-Cluster-XL/td-p/129100 (or a similar post, I forgot to bookmark the one I read) I can get the tunnel up, but cannot get BGP to work.

Please help.

How I built the VPN & BGP

#GATEWAY 1
add vpn tunnel 1 type numbered local x.x.x.1 remote x.x.x.17 peer AWS_Tunnel1
set interface vpnt1 state on
set interface vpnt1 mtu 1399
save config
#GATEWAY 2
add vpn tunnel 1 type numbered local x.x.x.2 remote x.x.x.17 peer AWS_Tunnel1
set interface vpnt1 state on
set interface vpnt1 mtu 1399
save config
#SMART CONSOLE
"Get Interfaces" -> "With Topology"
Then edit "vpnt1" to have a VIP of x.x.x.18

#3: Define Network Objects
New Interoperable Device called "AWS_Tunnel1"
Topology is "Empty_Group"

NOTE, "Network Management", in the "VPN Domain" section; I have left this as is for our other, already working VPN tunnels, rather than change to "Empty_Group"

Next I built the Star Gateway community following the instructions (nothing special here, just following the instructions)

OK, so far so good. VPN tunnel will be up at this point.

For the firewall rules I have tried two rules 

Neither rule seems to be grabbing the traffic.
An FW Monitor appears to show pings going out to the internet rather than down the tunnel.

I have concluded the issue is with BGP rather than the VPN configuration itself.

BGP has a Router ID of my public IP address
BGP has a Local Autonomous System Number of 65000
BGP has a Peer group of 65256 with a local address of x.x.x.18 and a peer of x.x.x.17

BGP has 0 messages sent and 0 messages received.

"Inbound Route Filters" has a BGP policy for 65256
"Route Redistribution" is advertising all static routes to BGP 65256

Where did I go wrong, why isn't this working?

Also worth nothing, some instructions said to use an Unnumbered VTI, when I did this BGP would never leave the "Idle" state and everything still failed to work.

[Nir_Shamir]

Hi,

first you need to edit your FW object under the VPN Community and change the encryption domain to empty (see attached). this will only affect this specific Community and not others you have.

after that , can you see in your logs BGP traffic encrypt / decrypt ?

[Secret-goblin-5]

Thanks, missed this, it was not in any of the instructions I have found.

Also missed that BGP traffic was being blocked. I have enabled a rule to allow BGP traffic through but still got problems.

This hasn't fixed our BGP issues, but has made them better.

[the_rock]

For what its worth, though I can only speak from my own experience with Azure and AWS, I always found that using empty enc. domains on both sides and UNNUMBERED vtis for BGP is what makes all this work well. Then, to route traffic, you can use those VTIs as DG when you create a route in web UI. By the way, also important, most people may "freak out" if  they realize that unnumbered VTI will look EXACTLY like the actual external interface in topology, but thats 100% normal and you can even give it exact same VIP as well, no issues there. Just make sure to do "get interfaces WITHOUT topology. Also, no need to have anti spoofing configured on those.

See if my post below helps.

https://community.checkpoint.com/t5/Security-Gateways/Route-based-VPN-tunnel-to-Azure/m-p/206179/emc...

Andy

[Secret-goblin-5]

Rebuilt the tunnels using your recommended configuration.

It is working, traffic is flowing (or appears to be) but BGP is still not working.

BGP Config commands listed below.

Gateway1> show bgp

Routing Process BGP
    State is on
    Local Autonomous System is 65000
    Default Weight is 0
    BGP Route Rank is 170
    ECMP is off
    IGP Synchronization is off
Gateway1> show bgp errors

PeerID            Last State        Last Event        Last Error
x.x.x.17    Idle              Start             None
Gateway1> show bgp summary

Routing Process BGP
    State is on
    Local Autonomous System is 65000
    Default Weight is 0
    BGP Route Rank is 170
    ECMP is off
    IGP Synchronization is off
Gateway1> show bgp stats

Peer: x.x.x.17
                      Received            Sent
    Opens                    0               1
    Notifications            0               0
    Updates                  0               0
    Keepalives              98              98
    Route Refresh            0               0

Gateway1> show bgp peers

Flags: R - Peer restarted, W - Waiting for End-Of-RIB from Peer

PeerID           AS           Routes  ActRts  State             InUpds  OutUpds  Uptime
x.x.x.17   65256        0       0       Active            0       0        00:00:00
Gateway1> ping x.x.x.17
PING x.x.x.17 (x.x.x.17) 56(84) bytes of data.
64 bytes from x.x.x.17: icmp_seq=1 ttl=254 time=14.9 ms
64 bytes from x.x.x.17: icmp_seq=2 ttl=254 time=14.0 ms
64 bytes from x.x.x.17: icmp_seq=3 ttl=254 time=14.1 ms
64 bytes from x.x.x.17: icmp_seq=4 ttl=254 time=14.1 ms
64 bytes from x.x.x.17: icmp_seq=5 ttl=254 time=17.1 ms
64 bytes from x.x.x.17: icmp_seq=6 ttl=254 time=14.0 ms
^C
--- x.x.x.17 ping statistics ---
6 packets transmitted, 6 received, 0% packet loss, time 5002ms
rtt min/avg/max/mdev = 14.055/14.750/17.121/1.100 ms

Gateway1> ping x.x.x.18
PING x.x.x.18 (x.x.x.18) 56(84) bytes of data.
^C
--- x.x.x.18 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 1999ms


Gateway1> show bgp peer x.x.x.17 detailed

----- Peer x.x.x.17 -----
State                         Active
Peer Type                     eBGP Peer
Remote AS                     65256
Local AS                      65000
Peer Capabilities             n/a
Our Capabilities              IPv4 Unicast,4-Byte AS Extension
Authentication                None
Multihop                      Off
Reachability Detection        Off
Graceful Restart              Off
Received
    IPv4 Routes               0 (0 active)
    IPv6 Routes               0 (0 active)
Sent
    IPv4 Routes               0
    IPv6 Routes               0
Gateway1>

[the_rock]

You are using unnumbered VTIs now? If so, did you make sure you have routes pointing to remote subnet(s) using those unnumbered VTI interfaces?

If yes to all, then I would do basic zdebug and see why its fialing.

Andy

[Secret-goblin-5]

---

You are using unnumbered VTIs now?

---

Yes

---

If so, did you make sure you have routes pointing to remote subnet(s) using those unnumbered VTI interfaces?

---

I have a destination route to x.x.x.17 only (The AWS side of the internal tunnel IPs)
Do I need more than 1 destination route?

---

If yes to all, then I would do basic zdebug and see why its fialing.

---

OK, will try this and see what I get.

[the_rock]

Do the route to the whole subnet on the other side using related VTI.

Andy

[Secret-goblin-5]

- 169.x.x.17/32 as a static route to vpnt1 
- 192.168.177.0/24 as static route to vpnt1 (AWS test network)
BGP shows "Idle"

- 169.x.x.16/30 as static route to vpnt1
- 192.168.177.0/24 as static route to vpnt1 (AWS test network)
BGP still show "Idle"

What am I missing?

[the_rock]

So tunnel itself is fine and all works, except bgp? If so, did you try simple zdebug, just filter for 179?

fw ctl zdebug + drop | grep "179"

Andy

[Nir_Shamir]

can you run "show configuration bgp" on your GW and paste it here ?

"idle" means there is no BGP activity at all.

also check "/var/log/routed_messages" for any BGP error messages.

[Secret-goblin-5 · Accepted solution]

Thanks for all the help and apologies for not replying.

It took a LOT of work to get this working, but we managed it.

I'll paste my instructions below after anonymizing them.

Check Point BGP VTI VPN Tunnel Run Book

Basic Plan

1. Make new AWS tunnel, add Transit Gateway on AWS end.

2. Build VTI interfaces on both firewalls for both tunnels (4 in total) on Check Point End

3. Resolve routing issues in AWS (remove static routes, add BGP)

4. Check BGP is advertising the new routes.

5. Test traffic flow down the new tunnels.

6. Test redundancy with failover  test etc.

1. Make a new VPN tunnel in AWS

We cannot use the old VPN tunnels as they do not support BGP.

Make new VPN tunnels, using BGP, and using the same Transit Gateway and Customer Gateway as the old tunnel uses.

Use the same name as the old tunnel, but with “-bgp” appended to the name.

We are doing this through Click Ops. this is not too difficult for people with basic AWS training.

Lastly, once the tunnel is built, download the configuration with the button, choose “Check Point” and R80.10+

2. Build Tunnels in Check Point

Prerequisites

Removing External IP from inside the VPN tunnel

Gateway Cluster properties -> Network Management -> VPN Domain

Tick "Exclude Gateway’s external IP from the VPN Domain"

Making sure we don’t break any other VPN tunnels

Set specific VPN domain we are working on to "Empty_Group" so that other VPN tunnels are not affected.

Disable Anti-Spoofing for packets just from “AWS_ ACCOUNTS-PLUS-VTI_PEERS”

We don’t want to turn on Anti-Spoofing for all external traffic, just for AWS traffic & all virtual tunnel peers.

The object already exists and can be used for all Gateway clusters, but will need the virtual tunnel peers added for each Gateway cluster we do.

PreRequisite Planning

Add New AWS Tunnels (Interoperable Devices)

The old names we use are too long, they get cut off in one of the fields in Smart Console, this makes them both the “same” name which does not work.

Work out new objects names from the old ones. For example AWS-TRANSIT-CORE-EU-WEST1-OFFICE3-TUNNEL2 turns into to AWS-TRANSIT-CORE-EU-WEST2-O3-T1.

This will allow each name to stay unique in the field where it is too long.

For now, just write this down, we don’t need to make this change in Smart Console as we will actually make these objects in CLISH

Tasks in WebUI / CLISH

Build VTI Interfaces (CLISH)

NOTE! Using objects we named in the previous step.

Download the instructions from the AWS website.

Find the tunnel instruction commands and modify them.

      add vpn tunnel 1 type numbered local 169.xx.xx.170 remote 169.xx.xx.169 peer AWS_Tunnel1

      set interface vpnt1 state on

      set interface vpnt1 mtu 1399

      save config

              TO

      add vpn tunnel 1 type numbered local 169.xx.xx.1 remote 169.xx.xx.169 peer AWS-TRANSIT-CORE-EU-WEST1- S0-T1

      set interface vpnt1 state on

      set interface vpnt1 mtu 1399

      save config

Note, not only has the name changed, but we have also put in a “fake” IP. This is because the cluster object uses the real, .170 IP. We use “1” as our IP as it helps us work out which tunnel this IP is linked to and is different enough from the real one that you cannot accidentally mistake it for one of the real IPs.

Don’t forget there are 4 tunnels in total, 1 for each Firewall / Gateway, plus 2 for the 2 tunnels, makes a total of 4. Make sure you make all four in CLISH (two per gateway).

Configure VPN Object

Use the same VPN object as the old tunnels, this is not an issue as nothing is changed until we publish the changes.

Add new objects we just made to Satellite objects locations. (This removes the old tunnels from use, and lines up our new one.)

Set VPN Domain to “EmptyGroup” if not already.

Turned off MEP

Turned off NAT

Set Phase 1 & 2 timers to 60 mins (known to resolve some bugs)

All other settings can stay the same.

Configure Gateway Network Properties

Go into the Gateway cluster’s properties, then go to Network Management.

Click “Get Interfaces” -> “Without Topology”

You will see the two new VPN Tunnels appear (only 1 in the picture below)

You must now configure the correct cluster IP. Double click on “vpnt1” in the list.

You will see the two “fake” IPs we created earlier, and an empty field for the clustered IP.

Place the real IP (.170 in our earlier example) into the IPv4 box.

Do the same for “VPNT2” with the correct IP for that tunnel.

Build NAT Rules

We need to build custom NAT rules because the clustered IP without custom NAT will stop BGP traffic being replied to.

We need to build two sets of NAT Rules.

• VPN Tunnel only rules
• All other traffic rules (IE turn NAT off)

VPN Tunnel NAT Rules

The VPN tunnel needs to translate the two “fake” IPs (Source) into the cluster IP (translated source) so that the return traffic has a real IP address to be sent to.

Make all the objects listed above, then create these rules.

• AWS-Tunnel1-VTI-IP-Local-Nodes = The “fake” IPs we made earlier (.1 for Gateway 1 & .2 for Gateway 2 in our example)
• AWS-Tunnel1_VTI_Remote = the AWS real IP (.169 in our example)
• AWS-VTI-Tunnel1-IP-Local-VIP = the real Check Point side IP (.170 in our example)

Then make the objects again for Tunnel 2 (see pic above for the correct layout)

Next we make the “Do not NAT” NAT. This turns off NAT for other traffic which goes down the tunnel we are making.

This rule is pretty simple, it takes all local traffic, and if it’s moving to AWS it keeps all NAT info to original, IE no NAT. The rule below is the same, but in reverse.

Adding in the Rules to the Policy

We need to make two major changes to the policy. We need to add in BGP rules, and remove the VPN setting for traffic we plan to move down the tunnel (or convert the VPN settings to be the same as the BGP Out rule)

As you can see, the rules allow BGP traffic from the Local tunnel addresses, to the remote tunnel addresses. The VPN settings are more unique though.

For the VPN settings we need to use “Directional Match Condition” VPN, we cannot use traditional VPN. If we do, it will not work. (A different daemon will process it, and it will not be pushed down the tunnel)

Hope this helps others.

[the_rock]

AMAZING!

[Duane_Toler]

If you're doing unnumbered VTI with BGP (and on a cluster), then you need to use a loopback interface.  The unnumbered VTI interface needs a proxy of some kind, but on a cluster a there is no physical interface with a cluster VIP to use for a proxy.  VTIs are configured PER PHYSICAL gateway, that's why you can't rely on a VIP ("V" in VIP is virtual; duh).

So the loopback creates that proxy-able interface for you.

cluster member 1: "add interface lo loopback 169.254.1.1/32"

cluster memebr 2: "add interface lo loopback 169.254.1.2/32"

Use the link-local APIPA range for this, that's what it's for, and 3rd octet of "1" is allowed for this usage.

You'll then need to attach this loopback interface to your VTI configuration (both cluster members:  "add vpn tunnel 1 unnumbered dev loop00 peer VPNPEER_GW_IN_SMARTCONSOLE").  The BGP remote AS peer IP is whatever you want it to be across the VTI.  Then you need 2 tricks:

1. static route to the BGP remote peer across the VTI (bot cluster members: "set static-route 172.16.1.1/32 nexthop gateway logical vpnt1 on")

2. Configure the BGP peer with ebgp-multihop and ttl 3 (set bgp external remote-as 65256 peer 172.16.1.1 multihop on; set bgp external remote-as 65256 peer 172.16.1.1 ttl 3).

Be sure both cluster members have the same BGP AS **AND** the same router-id.  Absolutely required.  In fact, I would suggest you set the router-id to be the IP of future VIP of the loop00 interface (see below for SmartConsole interfaces).  You could instead use whatever you want, of course, such as a cluster VIP of another interface.  Set this manually so that BGP doesn't auto-select one for you.  You NEED to know this is consistent.

Your BGP session is stuck in Idle because it can't find a route to the BGP peer, because the peer is across the VTI which can't be established because the route doesn't exist to reach said peer.  The static route solves this.

If you plan to get BFD involved, then you got some more work to do. 🙂  I won't include that here.

Fetch topology in SmartConsole.  The IP will be the same for the vpntX interfaces and loop00 interfaces. Leave this in place.  Create a cluster VIP for both of them manually just like any interface. (169.254.1.3/32)

You can use VPN encryption domains per community to use the empty groups rather than setting this directly on your VPN gateway objects.  This reduces risk of VPN domain errors for your other VPNs using the classic configuration.

Install policy. Celebrate the joy.

[the_rock]

"Throwing in" my notes as well. I know its Azure, but hope it helps.

Andy

**************************************

VPN CONFIG EXAMPLE:

Steps to build the route based VPN tunnel

Azure portal:

Create new VNG

SK Basic (100 Mbps Limit)

Route Based

No BGP/Active to Active (because basic SK)

New Resource Group

New VNET

10.0.0.0/16

New Public IP

VIP = x.x.x.x

New Local Network Gateway (This is a reference object for the Checkpoint Cluster/Lab Checkpoint)
DEVCheckpoint

IP address: x.x.x.x

Address space: 172.16.10.0/24, x.x.x.x/x, 192.168.10.0/24 (Must include internal/local subnets and the external WAN facing subnets)

Click , click Add Connection

Type: Site to Site

PSK Pleasework1!

IKEv2

Click the connection

Download Config (Cisco > IOS > IKEv2)

Verify Default Settings/VTI IPs

IKE aes-cbc-256, sha1, DH 2, SA lifetime 3600S

IPSec esp-aes 256, esp-sha256-hmac, SA lifetime 3600s, SA lifetime 102400000 KB

Configure an APIPA (169.254.x.x) address that does NOT overlap with any

! other address on this device. This is not visible from the Azure gateway.

Local on Checkpoint Side VTI IP 169.254.0.1/32

Remote (Azure) 169.254.0.2/32

If there is another tunnel use DIFFERENT IPs that DO NOT OVERLAP WITH PREVIOUS RouteBASED TUNNEL

-------

Access to Lab Checkpoint

SmartConsole x.x.x.x

SmartConsole Settings

Global Properties > VPN > Advanced > Enable VPN Directional Match

Add Interoperable Object for Azure GW with configured VIP

"AzureLabGW"
Topology > VPN Domain > Add an Empty Group

Publish & Install

Go to Gaia WebUi (172.16.10.189:4434)

Network Interface

Add VPN-Tunnel

vpnt1

Peer Name should be EXACT SAME AS INTEROPERABLE DEVICE NAME

Local IP 169.254.0.5 (Not used anywhere else)

Remote IP 169.254.0.6 (Not used anywhere else)

Add Static Route

Local IP/Subnet of Azure GW (Virtual Network = 10.0.0.0/16)

Gateway (IP) of Remote IP from VTI configured previously (169.254.0.6)

Go back to SmartConsole

Open Gateway Object/Cluster

Network Management > Topology > Get Interfaces WITHOUT Topology

Make sure VTI interface shows up, may need to set up vip obj for vpnt tunnel in cluster (make sure no overlap)

Install Policy

Create a new VPN Community (Star Topology)

Ensure both gateways use an EMPTY group for domain

Encryption (IKEv2 Only)

AES256, SHA1, Group2

AES 256, SHA256

Tunnel Management

Set perm tunnels on all tunnels in the community

One tunnel per gateway pair

VPN Routing

To center only

SharedSecret

Pleasework1!

Advanced

IKE Phase 1 480 Min

IPSec Phase 2 27000 seconds

Disable NAT inside VPN Community

Policies

Source & Destination

- Local Subnets included in the Local Gateway Object Config/Applicable Subnets

- Remote (AzureNetwork 10.0.0.0/16)

- RServ (Radius Server used for testing)

VPN > Directional Match

Internal to Community

Community to Internal

Community to Community

Publish & Install

GUIDBEdit DPD Enabled (Tunnel Test Settings)

Reset Connection on Azure Side

MAY NOT BE NEEDED REFRESH AND CHECK IF CONNECTED

Test

Create VM LabUbuntu

VIP x.x.x.x

Private IP 10.0.0.4

Enable Rule to allow Pings & SSH traffic in

[Duane_Toler]

I meant to add the "Why?" to this.  "Why" do all these gymnastics?  Because for the NEXT time you do a VTI with BGP to another peer, you can re-use ALL of this configuration.  You make a new VTI for the remote peers, re-use the same loop00 (don't make another one), add the static route to the new peer across that peer's VTI, add the new peer's BGP configuration, add the new VTIs in SmartConsole, add the VPN community (vpn domain per community, too), and you're done.

You'll always need route-maps per remote AS, and at least 1 route-map that has the "allow" rule.

set routemap FOO_in id 10 on
set routemap FOO_in id 10 allow
set routemap FOO_out id 10 on
set routemap FOO_out id 10 match protocol direct  # to match locally-attached interfaces
set routemap FOO_out id 20 on
set routemap FOO_out id 20 match protocol static # to match static routes
set bgp external remote-as 64526 import-routemap FOO_in priority 1 on
set bgp external remote-as 64526 export-routemap FOO_out priority 1 on