topic Re: External check to determine active/standby state of cluster members? in Cloud Firewall

External check to determine active/standby state of cluster members?

johnnyringo — Fri, 30 Sep 2022 20:23:44 GMT

Background: We use Google Cloud Global HTTP(S) load balancers in front of a CheckPoint CloudGuard HA cluster to handle SSL termination and give each service a different external IP addresses. The CheckPoint cluster members are the load balancer's backend and we're left with a fundamental problem:

The load balancer does a basic TCP check on port 443 to detect a down member.
Since both members pass the check regardless of active/standby state, the traffic is distributed 50/50 assuming both are up.
The max NAT sessions are 16384/66 which obviously is 99/1, not 50/50

That NAT issue is described here and I still don't have a fix for it. So I'm thinking if I can somehow just get the traffic to go 100/0, this fixes the issue.

Is there a way to externally check the active/standby status? Perhaps there's a service that runs only on the active and is shutdown when it goes standby?

Re: External check to determine active/standby state of cluster members?

Chris_Atkinson — Mon, 03 Oct 2022 03:23:23 GMT

Something seems off with this deployment, would suggest investigating the details/specifics further with TAC.

Also tagging @Shay_Levin in case he has any ideas

Re: External check to determine active/standby state of cluster members?

johnnyringo — Sun, 02 Oct 2022 19:05:39 GMT

Ahh yes, the old "surely, this problem is the customer's fault" response. Good 'ol CheckPoint, lol.

I'm currently on my 3rd support case. The NAT sessions issue was identified over 9 months ago:

https://community.checkpoint.com/t5/Security-Gateways/R80-40-NAT-port-exhaustion-why-do-cluster-members-show-vastly/m-p/140417#M27006

There's been a series of "bug fixes" for this in R80.40, but none that address the root issue

Re: External check to determine active/standby state of cluster members?

Chris_Atkinson — Tue, 18 Oct 2022 04:19:25 GMT

That's not the intent of my response. Actually it seems we have a mix of HA & Active/Active architectures in play.

We typically have different templates depending on HA or Active/Active (Autoscale) requirements per:

CloudGuard Network High Availability R80.30 and above for Google Cloud Platform Administration Guide

CloudGuard GCP Auto Scaling Managed Instance Group (MIG) for R80.30 and Higher Administration Guide

If the Cluster is a true HA cluster only the "VIP" (Cluster's External IP) should be a backend target of the LB, not the individual gateway / cluster member IPs (standby won't process traffic).

Additionally (but not so relevant to a load-balancer discussion) with normal cluster XL HA gateways you could query via SNMP to determine the active state i.e.

The OID 1.3.6.1.4.1.2620.1.5.6.0

The expected output is:

From the Active member:
SNMPv2-SMI::enterprises.2620.1.5.6.0 = STRING: "Active"

From the Standby member:
SNMPv2-SMI::enterprises.2620.1.5.6.0 = STRING: "Standby"

Re: External check to determine active/standby state of cluster members?

johnnyringo — Mon, 03 Oct 2022 02:31:35 GMT

We typically have different templates depending on HA or Active/Active (Autoscale) requirements per:

It's standard active/standby HA deployment (no auto-scaling or active/active)

If the Cluster is a true HA cluster only the "VIP" should be a backend target of the LB, not the individual gateway / cluster member IPs

Not quite following here. I known in AWS, you can set an IP address for the backend server. But as far as I know, GCP only allows instance groups, target pools, or NEGs. In all cases, these would reference instance vNICs, not just an IP address.

(standby won't process traffic).

I'd have to disagree with that, based on experience. The standby will load the same policy as the active and run all services. It definitely processes traffic just fine as long as asymmetric routing is accounted for.

Additionally with normal cluster XL HA gateways you could query via SNMP to determine the active state i.e.

That's good to know, but GCP load balancers definitely don't support use of SNMP polls for healthchecks. Perhaps I could create a Python script via 1-minute cron job to do the query and then update the LB backend accordingly, but that's a really ugly hack.

Re: External check to determine active/standby state of cluster members?

Chris_Atkinson — Mon, 03 Oct 2022 05:23:01 GMT

The admin guide for HA talks specifically about directing traffic for the Cluster's External IP towards the "Active" member, not both.

Therefor the "standby" node in HA shouldn't receive traffic, load-sharing is not supported in this mode.

Your load-balancer pool configuration should be such that it takes this into account.

Re: External check to determine active/standby state of cluster members?

johnnyringo — Mon, 03 Oct 2022 15:42:06 GMT

The admin guide for HA talks specifically about directing traffic for the Cluster's External IP towards the "Active" member, not both...Your load-balancer pool configuration should be such that it takes this into account.

Again, this is not possible with GCP load balancers, because there's no way to direct traffic to a specific IP address. If CheckPoint is expecting this to work, they need to submit a feature request to Google.

FortiGate doesn't have this problem because the two members act in very strict active/standby mode (i.e. the data plane on the standby is essentially shut down and won't even respond with a SYN+ACK)

Re: External check to determine active/standby state of cluster members?

Chris_Atkinson — Sun, 23 Oct 2022 04:50:19 GMT

With using TCP/443 for the health-probe from the LB, my assumption is it's likely being accepted by implied rules for multi-portal infrastructure including GAiA web portal etc.

In the firewall logs what do you see as the destination for the TCP/443 health probes coming from the LB?

There is a way to influence the implied rule behavior via kernel parameter (sk105740) but whether this will assist may depend on the target of the health probe and the ability to create specific rules to handle this traffic in a more granular way.

Otherwise the standby member of a clusterXL cluster shouldn't forward traffic (sk63942) and if you see something contrary to this it's definitely something for TAC.

Source: https://sc1.checkpoint.com/documents/R80.40/WebAdminGuides/EN/CP_R80.40_ClusterXL_AdminGuide/Topics-CXLG/Viewing-Cluster-State.htm

Re: External check to determine active/standby state of cluster members?

johnnyringo — Tue, 04 Oct 2022 01:36:12 GMT

the standby member of a clusterXL cluster shouldn't forward traffic

Here's what a non-443 TCP healthcheck looks like:

[Expert@xxx-member-b:0]# tcpdump -i eth0 -n port 29413 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes 21:27:07.494666 IP 35.191.11.202.59692 > 100.64.100.29413: Flags [S], seq 2790787222, win 65535, options [mss 1420,sackOK,TS val 2593715553 ecr 0,nop,wscale 8], length 0 21:27:07.495411 IP 100.64.100.29413 > 35.191.11.202.59692: Flags [S.], seq 3524699649, ack 2790787223, win 65535, options [mss 1420,nop,nop,sackOK,nop,wscale 6,nop,nop,TS val 616471966 ecr 2593715553], length 0 21:27:07.495563 IP 35.191.11.202.59692 > 100.64.100.29413: Flags [.], ack 1, win 256, options [nop,nop,TS val 2593715554 ecr 616471966], length 0 21:27:07.495584 IP 35.191.11.202.59692 > 100.64.100.29413: Flags [F.], seq 1, ack 1, win 256, options [nop,nop,TS val 2593715554 ecr 616471966], length 0 21:27:07.495766 IP 100.64.100.29413 > 35.191.11.202.59692: Flags [.], ack 1, win 4096, options [nop,nop,TS val 616471966 ecr 2593715554], length 0 21:27:07.495779 IP 100.64.100.29413 > 35.191.11.202.59692: Flags [F.], seq 1, ack 2, win 4096, options [nop,nop,TS val 616471966 ecr 2593715554], length 0 21:27:07.496021 IP 35.191.11.202.59692 > 100.64.100.29413: Flags [.], ack 2, win 256, options [nop,nop,TS val 2593715555 ecr 616471966], length 0 ^C 7 packets captured 7 packets received by filter 0 packets dropped by kernel [Expert@xxx-member-b:0]# cphaprob state ID Unique Address Assigned Load State Name 1 10.12.34.68 100% ACTIVE xxx-01 2 (local) 10.12.34.69 0% STANDBY xxx-02

Re: External check to determine active/standby state of cluster members?

Chris_Atkinson — Wed, 05 Oct 2022 22:58:29 GMT

From what I can see you will need to use HTTP LB on TCP port 8117 for the health probe with the supporting configuration per: sk114577 (Section 7)

Check Point CloudGuard IaaS reference architecture for Google Cloud Platform - [7] Setting up a load balancer

Re: External check to determine active/standby state of cluster members?

Sorin_Gogean — Wed, 05 Oct 2022 20:31:00 GMT

hello,

Can I ask, why do you have traffic routed specifically through the secondary appliance?

Shouldn't you route through HSRP IP of the cluster ?

Ty,

Re: External check to determine active/standby state of cluster members?

Nir_Shamir — Wed, 05 Oct 2022 21:39:29 GMT

yes, but not HTTP , it's TCP 8117.

Re: External check to determine active/standby state of cluster members?

Nir_Shamir — Wed, 05 Oct 2022 21:40:37 GMT

Google Clusters don't have an Internal VIP address. all the interfaces are configured as private.

So we can use an LB to route traffic to the ACTIVE member in the Cluster.

Re: External check to determine active/standby state of cluster members?

Chris_Atkinson — Thu, 06 Oct 2022 00:53:45 GMT

Yeah that was a reference to the type of LB, edited for clarity.

Re: External check to determine active/standby state of cluster members?

Sorin_Gogean — Thu, 06 Oct 2022 09:47:15 GMT

Understood, and initially you said "We use Google Cloud Global HTTP(S) load balancers in front of a CheckPoint CloudGuard HA cluster to handle SSL" and that the "load balancer does a basic TCP check on port 443 to detect a down member" .
So you are stating that Google Cloud Load Balancer checks the HTTPS response from each CheckPoint Cluster Member ? Why ???
I was expecting that Google Cloud Balancer would address smth behind the CheckPoint Cluster, therefore your routing from Google LB to the devices behind CKP Cluster should go through the CKP Cluster VIP .
Like in our case, our cluster on LAN side has 10.xx.3.2 (on CKP node A) and 10.xx.3.2 (on CKP node B) and the HA is 10.xx.3.1 (on CKP ACTIVE node) . So we route internal towards the 10.x.3.1 and we don't care about which node is active or standby .

So don't you have a similar situation ? Can you sketch smth in paint ?

ty,

Re: External check to determine active/standby state of cluster members?

_Val_ — Sat, 08 Oct 2022 14:13:49 GMT

@Shay_Levin can you advise please?

Re: External check to determine active/standby state of cluster members?

johnnyringo — Sat, 08 Oct 2022 18:03:02 GMT

@Sorin_Gogean wrote:
Understood, and initially you said "We use Google Cloud Global HTTP(S) load balancers in front of a CheckPoint CloudGuard HA cluster to handle SSL" and that the "load balancer does a basic TCP check on port 443 to detect a down member" .
So you are stating that Google Cloud Load Balancer checks the HTTPS response from each CheckPoint Cluster Member ? Why ???

Ummm...TCP healthchecks work at Layer 4/5 of the OSI model. They don't care about the HTTPS response. But the standby checkpoint responses to either type, any port, so it's a moot point anyway.

Here's a diagram of the setup. Note that the standby receiving traffic wouldn't be an issue if not for the max of 66 NAT connections

Re: External check to determine active/standby state of cluster members?

Chris_Atkinson — Sun, 09 Oct 2022 01:35:55 GMT

Have you reviewed / tried TCP 8117 with the supporting config as suggested above?

Re: External check to determine active/standby state of cluster members?

johnnyringo — Sun, 09 Oct 2022 04:00:32 GMT

@Chris_Atkinson wrote:

Have you reviewed / tried TCP 8117 with the supporting config as suggested above?

It's my understanding this only works on R81.10. We're on R80.40, and in cloud deployments, the upgrade to R81.10 is non trivial.

Even under R81.10, I'm really not down the the whole tweaking kernel parameters thing. There's too much of a risk of that setting being reset upon subsequent upgrades.

Re: External check to determine active/standby state of cluster members?

Dmitry_Gorn — Sun, 09 Oct 2022 11:01:57 GMT

Hi,

The GCP HA solution was released with a reference architecture describing Site-to-Site VPN use case and egress/E-W inspection via routing rule updates to forward traffic to the Active member, and VIP for ingress. A Load Balancer use case for ingress/egress was not designed nor certified.

For ingress inspection we recommend using the auto-scaling MIG solution.

Having said that, as Chris mentioned, with R81.10 we have firewall-kernel code that replies to GCP load balancer health probes - it is in use by the auto-scaling MIG solution from R81.10. On HA, only the Active will respond to the probes on port 8117.

For 2023 roadmap, we are evaluating the possibility to update the HA architecture to rely on load balancers for ingress/egress, or officially support ingress architecture with Load Balancers. In that case, the health probe will automatically be configured during deployment (as happens on other platforms).

Thanks