CheckPoint Cluster behind F5 Load balancer unable to reach to internet

[Wei_Soon_Heng]

Hi All,

I am facing a strange issue where a pair checkpoint cluster(located behind F5) unable to reach internet. We need checkpoint cluster to have internet access to download geolocation package from CP cloud, client want to enable the geolocation feature.
CheckPoint cluster is not holding any public IP , it will being nated at F5 when go over internet.

Troubleshooting step that have been done:
-Ping from both cluster member to F5 devices is success, but ping from checkpoint cluster to external(e.g 8.8.8.8) , packet is being forwarded from gateway via output of tcpdump but no reply packet is received.

-Output of tcpdump in F5 showing that  echo-reply have been returned to checkpoint but checkpoint does not show any receiving of icmp reply packet. Checked in checkpoint that there is no drop in firewall rule or kernel and interfaces level.

-Arp table in F5 devices shows that the mac address of CheckPoint VIP is bind to active member

-Meanwhile, this cluster have few working site-to-site vpn tunnels that established via through F5 devices.

-Tried failover of cluster member, it still does not resolve the issue.

-We have another single distributed checkpoint gateway that connect to the same F5 devices, it is able to reach internet and download the geolocation packages.

I am wondering where is the icmp reply packet goes? since F5 can see icmp reply is forwarded to checkpoint VIP.
I suspect it is related to checkpoint VIP.

Does anyone experienced the similar issue?

Checkpoint management server and cluster version is R80.30.

Thanks

[the_rock]

Your description definitely helps us hopefully give you a step in right direction. Here are some things I would try:

-if you check route to say 8.8.8.8 on CP, what do you get (from active member run ip route get 8.8.8.8)

-on active fw, if you run fw monitor -e "accept host(8.8.8.8);" ...what do you see?

have you tried running fw ctl zdebug | grep 8.8.8.8 while simultaneously pinging 8.8.8.8 from a duplicate window to see if anything gets dropped?

Andy

[Wei_Soon_Heng]

-Yes, the next hop is F5 devices if the destination is external.
-Only inspection point o and O are seen, no reply packet as same as the output of tcpdump.
-No drop in output of zdebug drop 

I am wondering how does vpn tunnels is working because those initiation ike traffic also pass through the same F5 devices in order to establish with their peers.

[Chris_Atkinson]

Check the mac-address are correct in the packet captures, also is "auto last hop" enabled on the F5?  (Refer: sk83420)

[mcatanzaro]

Adding to what Chris said, if the F5 is sending return traffic to the wrong MAC then you should look at enabling VMAC mode on the cluster. 

https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...

[Wei_Soon_Heng]

How can we check in checkpoint if the F5 is sending return traffic to wrong MAC?

[mcatanzaro]

You can use the -e option with tcpdump on the F5 if it supports that flag. I imagine it would but am not the most familiar with that vendor. 

[Daniel_]

f5 use linux just as system to access HW. Most (arp, tcp, ssl,...) is handled inside tmm. So you can check ARP/auto-last-hop inside tmsh
show sys connection cs-client-addr <IP-of-Firewall> all-properties

BTW: For outgoing traffic CP doesn't use VMAC!