Policy push seems to cause connectivity issues with FortiGate cluster on same VLAN

[KHodgsonSHQ]

This is a bit of a strange one, so please bear with me here. I've not yet had the opportunity to do any in-depth investigation but I wanted to get some thoughts and opinions.

We're experiencing an issue where installing a policy on a Check Point ClusterXL pair appears to indirectly disrupt internet connectivity for a separate FortiGate HA cluster that shares the same internet-facing VLAN. The working theory is that the Check Point policy installation may trigger unexpected ARP or MAC address changes that confuse the Cisco core switch.

After a Check Point policy install, internet access through the FortiGate cluster gradually fails, with outages occurring anywhere from a few minutes to several hours later. Connectivity is typically restored by rebooting the secondary FortiGate node.

My suspicion is that this is actually an issue with either the Cisco switch or the FortiGate cluster.

[simonemantovani]

Hello

how is configured the Fortigate cluster? It's strange that you need to need to reboot the secondary Fortigate to restore Internet (and not to reboot the primary Fortigate); the issue happens even if the secondary Fortigate is configured as the active member of that cluster?

[KHodgsonSHQ]

Hi.

Yes, I agree it's strange that rebooting the secondary resolves the issue but that seems to be the case. As far as I'm aware it's a standard HA Cluster. I'm not sure if the behaviour is the same following a failover of the Fortigate cluster but that's something to check. I'm still trying to gather as much information as I can. Some initial research suggests that it might be related to large amounts of GARP traffic being generated by the Check Point cluster after a policy push.

[simonemantovani]

Ok, and whe the issue happens is there any logs on the Cisco, maybe about ARP or similar?

[KHodgsonSHQ]

We're going to schedule some time to check this. Thanks.

[CaseyB]

I would compare the ARP table on the Internet-facing VLAN before and after the policy push.

[KHodgsonSHQ]

That's the plan. We'll be scheduling a maintenance window to do this.

[Chris_Atkinson]

Is the CP firewall commonly encountering failover events otherwise?

Is the cluster object recovery options set to switch to higher priority member or maintain active?

What version and JHF?

[KHodgsonSHQ]

I don't see any evidence of frequent cluster failovers, and certainly not around the time of a policy push. Cluster recovery is set to higher priority member, connection persistence is set to rematch connections, version and JHF is R82 with take 122.

[JozkoMrkvicka]

What is HW/SW on CP cluster members  and FG cluster members ?

I can imagine that after policy push of CP policy, the failover happens and that might cause issue with ARP. IPv4 GARP for CP cluster IP is sent from active CP cluster member every 60 seconds. During CP failover, GARPs are sent from newly active CP member 3 times imidiatelly once CP member became Active (and then within 60 seconds interval).

Once internet is working, try to do cpstop on standby CP member and the same on secondary FG member to see if something changes after policy push on active CP member.