BGP advice

[Sandgirl]

Hi guys, 

Let me start with saying I have very little experience with Checkpoints - only few weeks, so please bear that in mind 🙂 

I'm trying to establish a BGP connectivity between our two Checkpoints in Azure (CloudGuard) and the Azure Route Server (ARS). 

I have two setups: 

One is the Checkpoint Server Manager and ARS. 

The other one is Checkpoint Firewall and ARS. 

In the first instance, I can see on Checkpoint (CLI) that there is a two way communication: SYN, SYN ACK, ACK (but also F, P and R). But the neighbourhood is not estaliblished - the peers show on Checkpoint SM as either 'active' or 'idle'. 

With the second setup, I can only see traffic (in CLI) coming from ARS. Checkpoint does not respond at all. I have set up ASN in the GUI and peers, but there is absolutely no response. Is there any other setting somewhere I need to enable/setup? 

Finally, the above setups are just in my lab. When we deploy the solution, the Checkpoints will be behind Azure Load Balancer. Is this supported? I have read somewhere on here that it might not be? 

Any help would be greatly appreciated!

[the_rock]

Hey @Sandgirl ,

No need to apologize, we are here to help. So, below is good article explaining BGP different statuses:

https://www.ciscopress.com/articles/article.asp?p=2756480&seqNum=4

Now, I dont know for certain if BGP is supported when CPs are behind Azure load balancer, so maybe someone else can confirm that for you.

Now, here are few thing to check if show bgp summary command from clish does not show you established. 

-verify BGP settings match on both sides

-do fw ctl zdebug | grep x.x.x.x (other peer BGP ip address) and see why its getting dropped (if it is)

-do tcpdump or fw monitor as per below port (BGP)

tcpdump -nni any host x.x.x.x and port 179 (again, replace with other side peer IP)

fw monitor -e "accept host(x.x.x.x) and port(179)

Alternatively, you can also do new version of fw monitor flag

Idea is this -> fw monitor -F "src ip, src port, dst ip, dst port, protocol" -F "srcip, src port, dst ip, dst port, protocl"

So, lets say your IP is 1.1.1.1 and dst IP is 2.2.2.2 and port is BGP (179) and protocol can be anything, you can do this

fw monitor -F "1.1.1.1,0,2.2.2.2,179,0" -F "2.2.2.2,0,1.1.1.1,179,0"

We dont really care about source, only destination port.

Keep in mind all these commands are done in expert mode.

Hope that helps.

Andy

[Sandgirl]

Thank you so much Andy!

So I run the 
fw ctl zdebug drop 

and I'm getting the following message: 

fw_log_drop_ex: Packet proto =6 x.x.x.x:y -> z.z.z.z:179 dropped by fw_send_log_drop Reason: Rulebase drop - DEFAULT POLICY

Where would I find this policy so I can override it?

[the_rock]

Sorry for the late reply, not sure how I missed the email notification, apologies. So to verify right policy, can you run fw stat command on the CP fw? If default policy or default filter is on, that would block everything.

[Sandgirl]

Thank you so much everyone! 

After a lot of looking around, I have managed to unblock TCP 179 (as well as SSH and ICMP) after implementing the required rules and also turning off address  spoofing. 

I can now see the traffic being accepted by the firewall, and I can see SYN, ACK, Push, Finish and Reset. Tcpdump capture shows a flow - but only in one direction, from the ARS to the checkpoint, again I can see SYN, ACK, BGP Open... but then, I can see BGP Notification (which looks like the timeout waiting for the response) and FIN retransmission of FIN and Reset. 

So the traffic seems to be getting there, but Checkpoint is still not willing to establish the BGP connection. 

As mentioned before, I am trying to set it up with ARS, which only asks for IP and ASN. 

Am I missing some setting on the Checkpoint that I need to enable to get it to work? 

[the_rock]

K, so something is not matching with the other side for sure. What do they see on Cisco side?

Can you send us output of below commands from clish? Please blur out any sensitive info.

Andy

From expert mode, just type clich and then run following commands (one by one)

show bgp errors

show bgp groups

show bgp memory

show bgp paths

show bgp peer

show bgp peers

show bgp routemap

show bgp stats

show bgp summary

[Sandgirl]

Unfortunately I'm not using Cisco at the other end, it's ARS - Azure Route Server, which provides very little troubleshooting options. 
I can see the BGP traffic from ARS arriving at Checkpoint, but I cannot see Checkpoint trying to communicate with ARS. So that's why I'm thinking I missed something out on Checkpoint. 

[the_rock]

Sorry @Sandgirl , I guess I cant spell properly...lol. I read ASR, so I assumed it was Cisco ASR, my bad, apologies. Ok, so if thats the case, yea, I know, Azure in general does not sadly provide many troubleshooting options at all. Can you actually run fw monitor -F flag I gave in my first response? Also, run ip r g command and then simply put IP address you are trying to reach, so we can confirm the routing part (example ip r g 8.8.8.8)

Andy

[Sandgirl]

The fw -F command didn't return anything. 

ip r g command helped - the traffic is going via the other interface, so once I change the settings on both ends, I can now see the traffic going both ways... although it's still failing... the connection is timing out on Checkpoint end it seems, the ARS sends a message that its Hold time has expired... Or am I reading this wrong? 

[the_rock]

If you check my very first response on your post, I gave an example of fw monitor -F command.

idea is this:

fw monitor -F "srcip,srcport,dstip,dstport,protocol" -F "srcip,srcport,dstip,dstport,protocol"

(just an example -> fw monitor -F "1.1.1.1,0,2.2.2.2,179,0" -F "2.2.2.2,0,1.1.1.1,179,0"

I really think that capture would help us here.

Andy 

[Sandgirl]

Hi Andy, 

I run the fw monitor command for few minutes and this is what it returned:

[vs_0][ppak_0] eth1:i[40]: 10.0.4.4 -> 10.6.1.4 (TCP) len=40 id=27946
TCP: 64818 -> 179 ..R.A. seq=a532c7d3 ack=2ef94736
[vs_0][fw_0] eth1:i[40]: 10.0.4.4 -> 10.6.1.4 (TCP) len=40 id=27946
TCP: 64818 -> 179 ..R.A. seq=a532c7d3 ack=2ef94736
[vs_0][fw_0] eth1:I[40]: 10.0.4.4 -> 10.6.1.4 (TCP) len=40 id=27946
TCP: 64818 -> 179 ..R.A. seq=a532c7d3 ack=2ef94736
[vs_0][ppak_0] eth1:i[44]: 10.0.4.4 -> 10.6.1.4 (TCP) len=61 id=27947
TCP: 64938 -> 179 ...PA. seq=cce9b89b ack=80f42139
[vs_0][fw_1] eth1:i[44]: 10.0.4.4 -> 10.6.1.4 (TCP) len=61 id=27947
TCP: 64938 -> 179 ...PA. seq=cce9b89b ack=80f42139
[vs_0][fw_1] eth1:I[44]: 10.0.4.4 -> 10.6.1.4 (TCP) len=61 id=27947
TCP: 64938 -> 179 ...PA. seq=cce9b89b ack=80f42139
[vs_0][ppak_0] eth1:i[40]: 10.0.4.4 -> 10.6.1.4 (TCP) len=40 id=27948
TCP: 64938 -> 179 F...A. seq=cce9b8b0 ack=80f42139
[vs_0][fw_1] eth1:i[40]: 10.0.4.4 -> 10.6.1.4 (TCP) len=40 id=27948
TCP: 64938 -> 179 F...A. seq=cce9b8b0 ack=80f42139
[vs_0][fw_1] eth1:I[40]: 10.0.4.4 -> 10.6.1.4 (TCP) len=40 id=27948
TCP: 64938 -> 179 F...A. seq=cce9b8b0 ack=80f42139
[vs_0][ppak_0] eth1:i[40]: 10.0.4.4 -> 10.6.1.4 (TCP) len=40 id=27949
TCP: 64938 -> 179 F...A. seq=cce9b8b0 ack=80f42139
[vs_0][fw_1] eth1:i[40]: 10.0.4.4 -> 10.6.1.4 (TCP) len=40 id=27949
TCP: 64938 -> 179 F...A. seq=cce9b8b0 ack=80f42139
[vs_0][fw_1] eth1:I[40]: 10.0.4.4 -> 10.6.1.4 (TCP) len=40 id=27949
TCP: 64938 -> 179 F...A. seq=cce9b8b0 ack=80f42139
[vs_0][ppak_0] eth1:i[44]: 10.0.4.4 -> 10.6.1.4 (TCP) len=52 id=27950
TCP: 65059 -> 179 .S.... seq=f24889d5 ack=00000000
[vs_0][fw_0] eth1:i[44]: 10.0.4.4 -> 10.6.1.4 (TCP) len=52 id=27950
TCP: 65059 -> 179 .S.... seq=f24889d5 ack=00000000
[vs_0][fw_0] eth1:I[44]: 10.0.4.4 -> 10.6.1.4 (TCP) len=52 id=27950
TCP: 65059 -> 179 .S.... seq=f24889d5 ack=00000000
[vs_0][ppak_0] eth1:i[40]: 10.0.4.4 -> 10.6.1.4 (TCP) len=40 id=27951
TCP: 65059 -> 179 ....A. seq=f24889d6 ack=6ecba4fd
[vs_0][fw_0] eth1:i[40]: 10.0.4.4 -> 10.6.1.4 (TCP) len=40 id=27951
TCP: 65059 -> 179 ....A. seq=f24889d6 ack=6ecba4fd
[vs_0][fw_0] eth1:I[40]: 10.0.4.4 -> 10.6.1.4 (TCP) len=40 id=27951
TCP: 65059 -> 179 ....A. seq=f24889d6 ack=6ecba4fd
[vs_0][ppak_0] eth1:i[44]: 10.0.4.4 -> 10.6.1.4 (TCP) len=91 id=27952
TCP: 65059 -> 179 ...PA. seq=f24889d6 ack=6ecba4fd
[vs_0][fw_0] eth1:i[44]: 10.0.4.4 -> 10.6.1.4 (TCP) len=91 id=27952
TCP: 65059 -> 179 ...PA. seq=f24889d6 ack=6ecba4fd
[vs_0][fw_0] eth1:I[44]: 10.0.4.4 -> 10.6.1.4 (TCP) len=91 id=27952
TCP: 65059 -> 179 ...PA. seq=f24889d6 ack=6ecba4fd
[vs_0][ppak_0] eth1:i[40]: 10.0.4.4 -> 10.6.1.4 (TCP) len=40 id=27953
TCP: 64938 -> 179 ..R.A. seq=cce9b8b1 ack=80f42139
[vs_0][fw_1] eth1:i[40]: 10.0.4.4 -> 10.6.1.4 (TCP) len=40 id=27953
TCP: 64938 -> 179 ..R.A. seq=cce9b8b1 ack=80f42139
[vs_0][fw_1] eth1:I[40]: 10.0.4.4 -> 10.6.1.4 (TCP) len=40 id=27953
TCP: 64938 -> 179 ..R.A. seq=cce9b8b1 ack=80f42139

[Sandgirl]

For some reason I cannot reply to this comment with the output from the fw monitor... 

[Sandgirl]

Hi Andy, 

I've attached the output from fwmonitor as the text file - hopefully it will update this time. 

[the_rock]

Small favor @Sandgirl ...I need to know your local CP ip address, as well as remote side. Also, can you send the actual pcap file, rather than txt, as I would like to examine it in wireshark. Please send me offline message or blur out any sensitive info.

Cheers.

[Sandgirl]

I've attached the capture. It's my lab environment. 

[the_rock]

Thanks! Can you also please tell us the IP addresses involved (CP and other side)?

Cheers,

Andy

[Sandgirl]

Checkpoint's IPs are: 10.6.0.4 and 10.6.1.4 (the 1.4 is the one in the capture). The other end has two IPs: 10.0.4.4 and 10.0.4.5 (but the capture is just for 10.0.4.4).

[the_rock]

Excellent, let me review in a bit. So just to clarify, at this point, you see traffic from CP to the other side, but nothing coming back, right?

Andy

[Sandgirl]

Hi, 

No, 

I can see that ARS is trying to set up BGP with CP, and CP seems to be responding, at least up to a point. 

I cannot see CP trying to set up BGP with ARS. 

[the_rock]

I had a quick look at the file you sent and I see lots of FIN-ACKs, as well as retransmissions when I do below filters in wireshark:

tcp.port==179 and ip.addr=10.0.4.4

Here is good command to see what could be happening on CP side and if we see any drops at this point. Please run the tests again and leave this command running on CP firewall (expert mode)

fw ctl zdebug + drop | grep "10.0.4.4" | grep "179"

This will check for any drops for that IP, as well as BGP port (179)

Cheers,

Andy

[Sandgirl · Accepted solution]

Hi Andy, 

I managed to sort it out! 
It turns out I had to tick the 'eBGP Multihop' under Next Hop settings of the peers for the session to get established. 

Thank you so much for all your help! 🙂 

[the_rock]

Ah, yea, those settings are sometimes easy to overlook. Good job! Glad its fixed 💪

[Sandgirl]

Hi Andy, 

Another question. 
So I am now able to set up BGP peering between single Checkpoint and ARS (and Cisco router). 

However when I try to set up a peering between a Cluster and ARS (or Cisco), I'm not able to. 

Is there any additional setting that needs to be enabled somewhere on the Cluster to get it to work? 

[the_rock]

No worries, we can start another thread if needed or message me offline, we can do remote session. I know making this work with Cisco is bit tricky, my colleague and I spent 3-4 months until we finally got it working (dont let that discourage you, its probably because took forever to get everyone involved on the same page lol)

Check below settings, make sure they match.

Andy

 

[Sandgirl]

Hi, 

In the routed_messages and routed.log I keep getting this error: bgp_set_nexthop_address(9933) x.x.x.x [eBGP AS YYYYY] no IPv4 address found to connect. 

in routed.log I also get this message: bgp_use_protopeer x.x.x.x [eBGP AS YYYYY] Unable to find interfaces to reach this peer

When I googled one of these messages, it mentioned a VIP, but I already had VIP set up. 

Finally, I only built a Cisco device to help me troubleshoot the connection - I really want to set it up with Azure Route server. 

[the_rock]

I know that with Fortinet and Cisco (not sure if its same with PAN) when you have a cluster, there is no VIP, so every change you made on whichever member is active, it replicates on standby, so logically, in this case, you would most likely enter the IP address of active member. On CP side, its VIP, as per documentation.

Can you verify thats what you actually have set up?

Andy

[Sandgirl]

Hi Andy, 

With some help from the 3rd party I managed to figure it out. Turns out, that I need to set up the BGP peering to the VIP of the 'outside' interface (eth0) not the 'inside' (eth1). Which means I also need  to set up the router ID as the VIP. Once this has been set up, and  the corresponding routing added (route  to the ARS pointing to the default gateway of the 'outside' interface), it all started to work. 

Following from that, I struggled with the failover - the BGP peering and the connectivity only worked when I was using the primary Checkpoint, This is due to Azure's setup. When deploying Checkpoint from the Azure template, the Checkpoint is made a Contributor of the Resource Group it's build. By default this Resource Group also includes a Vnet in which Checkpoint is placed in, but in my setup it wasn't. So, Checkpoints didn't have permissions to move the VIP from primary Checkpoint to Secondary (I found it by running the $FWR/scripts/azure_ha_test.py script). Once I've granted Checkpoint cluster members appropriate permissions, it's all working. 

Phew! That was a difficult one. I feel like I've learnt a lot about Checkpoints in the process! 
Thank you again for all your help! 🙂 

[the_rock]

Excellent work @Sandgirl 💪👍👌

[HBK]

Hi

I am also implementing ExpressRoute between my Azure cloud to my on-prim datacenter. Checkpoint firewall's are working as  cluster. Could you please provide me the complete configuration steps to achieve this. Thanks 

[Luke_2023]

Hi, I have same issue but in a cloudguard cluster XL; do you mind pasting the cli bgp config that you used?

(show configuration bgp)

Thank you