topic Re: CheckPoint Cluster behind F5 Load balancer unable to reach to internet in Firewall and Security Management

CheckPoint Cluster behind F5 Load balancer unable to reach to internet

Wei_Soon_Heng — Tue, 21 Sep 2021 18:37:01 GMT

Hi All,

I am facing a strange issue where a pair checkpoint cluster(located behind F5) unable to reach internet. We need checkpoint cluster to have internet access to download geolocation package from CP cloud, client want to enable the geolocation feature.
CheckPoint cluster is not holding any public IP , it will being nated at F5 when go over internet.

Troubleshooting step that have been done:
-Ping from both cluster member to F5 devices is success, but ping from checkpoint cluster to external(e.g 8.8.8.8) , packet is being forwarded from gateway via output of tcpdump but no reply packet is received.

-Output of tcpdump in F5 showing that echo-reply have been returned to checkpoint but checkpoint does not show any receiving of icmp reply packet. Checked in checkpoint that there is no drop in firewall rule or kernel and interfaces level.

-Arp table in F5 devices shows that the mac address of CheckPoint VIP is bind to active member

-Meanwhile, this cluster have few working site-to-site vpn tunnels that established via through F5 devices.

-Tried failover of cluster member, it still does not resolve the issue.

-We have another single distributed checkpoint gateway that connect to the same F5 devices, it is able to reach internet and download the geolocation packages.

I am wondering where is the icmp reply packet goes? since F5 can see icmp reply is forwarded to checkpoint VIP.
I suspect it is related to checkpoint VIP.

Does anyone experienced the similar issue?

Checkpoint management server and cluster version is R80.30.

Thanks

Re: CheckPoint Cluster behind F5 Load balancer unable to reach to internet

the_rock — Tue, 21 Sep 2021 18:41:39 GMT

Your description definitely helps us hopefully give you a step in right direction. Here are some things I would try:

-if you check route to say 8.8.8.8 on CP, what do you get (from active member run ip route get 8.8.8.8)

-on active fw, if you run fw monitor -e "accept host(8.8.8.8);" ...what do you see?

have you tried running fw ctl zdebug | grep 8.8.8.8 while simultaneously pinging 8.8.8.8 from a duplicate window to see if anything gets dropped?

Andy

Re: CheckPoint Cluster behind F5 Load balancer unable to reach to internet

Chris_Atkinson — Wed, 22 Sep 2021 00:20:46 GMT

Check the mac-address are correct in the packet captures, also is "auto last hop" enabled on the F5? (Refer: sk83420)

Re: CheckPoint Cluster behind F5 Load balancer unable to reach to internet

mcatanzaro — Wed, 22 Sep 2021 03:13:13 GMT

Adding to what Chris said, if the F5 is sending return traffic to the wrong MAC then you should look at enabling VMAC mode on the cluster.

https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk50840&t=1632275904143

Re: CheckPoint Cluster behind F5 Load balancer unable to reach to internet

Wei_Soon_Heng — Wed, 22 Sep 2021 03:21:59 GMT

-Yes, the next hop is F5 devices if the destination is external.
-Only inspection point o and O are seen, no reply packet as same as the output of tcpdump.
-No drop in output of zdebug drop

I am wondering how does vpn tunnels is working because those initiation ike traffic also pass through the same F5 devices in order to establish with their peers.

Re: CheckPoint Cluster behind F5 Load balancer unable to reach to internet

Wei_Soon_Heng — Wed, 22 Sep 2021 03:23:28 GMT

How can we check in checkpoint if the F5 is sending return traffic to wrong MAC?

Re: CheckPoint Cluster behind F5 Load balancer unable to reach to internet

mcatanzaro — Wed, 22 Sep 2021 03:32:43 GMT

You can use the -e option with tcpdump on the F5 if it supports that flag. I imagine it would but am not the most familiar with that vendor.

Re: CheckPoint Cluster behind F5 Load balancer unable to reach to internet

Daniel_ — Wed, 22 Sep 2021 15:20:42 GMT

f5 use linux just as system to access HW. Most (arp, tcp, ssl,...) is handled inside tmm. So you can check ARP/auto-last-hop inside tmsh
show sys connection cs-client-addr <IP-of-Firewall> all-properties

BTW: For outgoing traffic CP doesn't use VMAC!