AWS ClusterXL High Availability

[Warren_T]

We are facing an issue with our Check Point firewall cluster deployed in AWS and would appreciate your guidance.

Environment:

Check Point Version: R81.20
Deployment: AWS ClusterXL High Availability

Issue:

When Member A is Standby, Internet or ping to any host on internet working normally
As soon as Member A becomes Active (after failover or manual switch), Internet connectivity stops for users and even from firewall it stops.
The cluster status appears healthy, and tarrfic exiting from WAN interface but no return traffic observed.

Troubleshooting performed:

Verified ClusterXL status.
Verified firewall policy and NAT.
Same route table on both gateways.

Could anyone suggest what AWS or Check Point components should be checked? Specifically, are there any known issues related to:

AWS route tables
Source/Destination Check
Elastic IP failover
ENI attachment
ClusterXL synchronization
CloudGuard Controller or failover automation

[Chris_Atkinson]

Everything good with your IAM roles, any errors with the HA script?

The APIs do take some time to enact the fail over from memory, how soon after are you testing before failing back again?

[Nir_Shamir]

Verify that your GW's can send API calls to AWS. run:

$FWDIR/scripts/aws_ha_test.py 

and see if there are any errors.

[Warren_T]

Here is output of the below command python3 $FWDIR/scripts/aws_ha_test.py

Testing if DNS is configured...
Primary DNS server is: 10.11.0.2

Testing if DNS is working...
DNS resolving test was successful

Testing metadata connectivity...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- 0:00:14 --:--:-- 0curl: (7) Failed to connect to 169.254.169.254 port 80: Connection timed out
Traceback (most recent call last):
File "/opt/CPsuite-R81.10/fw1/scripts/aws_ha_test.py", line 97, in test
region = get(META_DATA + '/placement/availability-zone')[:-1]
File "/opt/CPsuite-R81.10/fw1/scripts/aws_ha_test.py", line 58, in get
token = subprocess.check_output(cmd)
File "/opt/CPsuite-R81.10/fw1/Python/lib/python3.7/subprocess.py", line 411, in check_output
**kwargs).stdout
File "/opt/CPsuite-R81.10/fw1/Python/lib/python3.7/subprocess.py", line 512, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['curl_cli', '--request', 'PUT', 'http://169.254.169.254/latest/api/token', '--header', 'X-aws-ec2-metadata-token-ttl-seconds: 60']' returned non-zero exit status 7.
Error:
Failed in metadata connectivity test
Verify that outgoing connections over TCP port 80 (HTTP) to 169.254.169.254 are
allowed by the firewall security policy.
See:
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html

[Nir_Shamir]

so you have API access issue , can't connect to 169.254.169.254.

check routes, internet access , NSG etc.

[Warren_T]

Hi,

Both gateways have identical routes.

[Nir_Shamir]

I am talking about the entire environment. both GW's are just going out to the Cloud Infra and from there is takes the Cloud routes. So you need to check why you can't reach that IP.

if it's routes or some kind of Security block (from GWs or Cloud NSG's).

can the GW's access the Internet ?

[Warren_T]

As of now the gateway which is having the issue is standby and having the internet access. As we only manages firewall we don't have visibility to NSG and need to check that with different teams.