topic Re: Problem accessing standby cluster member from non-local network in Firewall and Security Management

Problem accessing standby cluster member from non-local network

Vladimir — Wed, 31 Jan 2018 16:40:34 GMT

Log shows accepted traffic on SSH and 443, cluster members connected to number of Cisco switches with VLANs in L2 mode.

No problem accessing both members from connected network.

vMAC in the cluster object IS ENABLED.

Any suggestions will be appreciated.

Thank you.

Re: Problem accessing standby cluster member from non-local network

AlekseiShelepov — Wed, 31 Jan 2018 16:57:32 GMT

How about fwha_forw_packet_to_not_active? Helps with similar situations with ping also.

Cluster debug shows "FW-1: fwha_forw_ssl_handler: Rejecting ssl packets to a non-active member"

Try with just entering # fw ctl set int fwha_forw_packet_to_not_active 1, and if works, enable on permanent basis in fwkern.conf.

I hope there is still fwkern.conf on R80.10

Re: Problem accessing standby cluster member from non-local network

Kaspars_Zibarts — Wed, 31 Jan 2018 17:19:14 GMT

You probably trying to connect to IP that's on the "other" side of the firewall and not locally connected IP. If you have only one route to the firewall VIP then traffic will get dropped as spoofed between firewalls. You can create manual static route for IP on the other side to point to standby memebr's locally connected IP interface address

Hope it makes sense

In other words if you have inside member-act 1.1.1.1 and member-stb 1.1.1.2 with VIP 1.1.1.3. And outside has IPs 2.2.2.1 and .2 and .3 then to connect to 2.2.2.2 via inside interface you will need to add static /32 on the router: 2.2.2.2 next-hop 1.1.1.2

Or I misunderstood maybe the problem?

Re: Problem accessing standby cluster member from non-local network

Vladimir — Wed, 31 Jan 2018 17:31:33 GMT

I'll give it a shot shortly and let you know if it works.

Thank you.

Re: Problem accessing standby cluster member from non-local network

Vladimir — Wed, 31 Jan 2018 17:42:55 GMT

Kaspars,

The situation is as you described, i.e. accessing from the "other" side.

However, ssh and https traffic to the management interface of the standby member logged as "accepted".

That being said, I'll try adding the route and see if it does the trick.

Thanks!

Re: Problem accessing standby cluster member from non-local network

Kaspars_Zibarts — Wed, 31 Jan 2018 17:54:38 GMT

This is what happens if you don't have the specific /32 to the standby. Packet arrives on active FW. It gets accepted and needs to be forwarded to standby box. It will do so based on topology, that is outside interface. But when this packet arrives to standby on outside interface with source IP from inside network, it will get dropped as spoofed. Unless spoofing is off

Re: Problem accessing standby cluster member from non-local network

Vladimir — Wed, 31 Jan 2018 18:01:46 GMT

Ah, but I've run the infamous fw ctl zdebug drop on the standby member (not yet in production), and have not seen the drops there.

Re: Problem accessing standby cluster member from non-local network

Vladimir — Wed, 31 Jan 2018 18:47:43 GMT

Kaspars and Alexei, thank you guys!

Ended-up using /32 routes as per Kaspars suggestion since I've recalled being in similar situation before and this being a less intrusive solution.

I will keep Alexei's solution in my toolbox.

That said, I still am baffled as to why zdebug drop did not yield anything on standby member when it was failing.

Regards,

Vladimir

Re: Problem accessing standby cluster member from non-local network

Kaspars_Zibarts — Wed, 31 Jan 2018 18:54:12 GMT

Will try tomorrow in one of clusters. Interesting. I just recall seeing spoofing drops in logs. Let you know.

Re: Problem accessing standby cluster member from non-local network

Kaspars_Zibarts — Thu, 01 Feb 2018 06:46:22 GMT

Here you go, fw monitor shows packet being accepted on incoming interface on active cluster member. But no outgoing packet there

fw ctl zdebug shows on active member

Note that this is without enabling fwha_forw_packet_to_not_active. After I enable it it started working immediately. Without static routes.

But I have seen your case not that long ago and that's why I remembered about /32 option..

Sorry won't have much time to dig into my past notes what happened there.

Re: Problem accessing standby cluster member from non-local network

Vladimir — Thu, 01 Feb 2018 14:40:28 GMT

Thank you again for looking into it: I have not seen the drops because I was looking for them on the standby member, thinking that if I see "accepts" in the log, the primary was not dropping them.

Apparently it drops them on egress.

Of two solutions available, which one do you like better?

Re: Problem accessing standby cluster member from non-local network

AlekseiShelepov — Thu, 01 Feb 2018 14:57:26 GMT

Hehe, I think I found exactly your situation

Connection from one side of the ClusterXL destined to the physical IP address of a non-Active cluster member on the other side of the ClusterXL fails - sk42733

I just wanted to mention that this fix with fwha_forw_packet_to_not_active is also connected to many other possible issues:

Simultaneously pinging the cluster members and the VIP address...

"Contract entitlement check failed" error on policy installation failure

Cluster debug shows "FW-1: fwha_forw_ssl_handler: Rejecting ssl packets to a non-active member"

"ERR_CONNECTION_REFUSED" error is displayed in web browser when connecting to Gaia Portal

Updates For Anti-Virus/Anti-Bot/Application Control/URLF blades are not working on standby ClusterXL member

So, it might be even a "best practice" But I have only experience with versions before R80, don't know how it is there. Also these "strange" routes might be a bit confusing for a new administrator for example.

Re: Problem accessing standby cluster member from non-local network

Vladimir — Thu, 01 Feb 2018 15:03:15 GMT

I agree that given how many issues this setting resolves, it may make better sense to have it on by default and have an option of commenting it out.

I wander what is the reason for it not to be and what possible side effects of it being set are.

Vladimir Yakovlev

973.558.2738

vlad@eversecgroup.com

Re: Problem accessing standby cluster member from non-local network

Vladimir — Mon, 18 Jun 2018 13:55:51 GMT

In the process of deploying new cluster of 15400s and ended-up using this suggestion.

Just wanted to tip my hat to you.

Thanks,

Vladimir

Re: Problem accessing standby cluster member from non-local network

Huseyin_Rencber — Mon, 18 Jun 2018 14:47:31 GMT

I had similar problem, solved after setting "fw ctl get int fwha_forw_packet_to_not_active" value to 1. sk42733 was helpful

Re: Problem accessing standby cluster member from non-local network

Kaspars_Zibarts — Mon, 18 Jun 2018 14:56:56 GMT

Re: Problem accessing standby cluster member from non-local network

Jerry — Tue, 06 Nov 2018 08:31:58 GMT

got similar story where

standby device in A/S HA is accessible by IP but GAIA Portal (http2 daemon) isn't working at all

I've regenerated self-signed certs

Still no go

httpd won't start

I can ssh to the standby device but one thing isn't accessible - https/http either on 443 or any other port simply HTTP daemon won't start on that device (there were no chances to that cluster since nearly 1y in terms of the PKI/FQDN etc.)

any ideas folks? alreago got a SR with TAC just from yesterday.

Cheers

Jerry

Re: Problem accessing standby cluster member from non-local network

Kaspars_Zibarts — Tue, 06 Nov 2018 11:19:16 GMT

I would say it's a separate issue as you can SSH to standby cluster member. So L3 routing is working end to end. There are multiple SKs regarding dead httpd, really depends on SW version you are running and actual errors you see. You might want to try this

How to debug the Gaia Portal

Re: Problem accessing standby cluster member from non-local network

Jerry — Tue, 06 Nov 2018 11:22:11 GMT

I did debug even with Ottawa TAC man - still no joy

Cheers Kaspars, I do appreciate what you've wrote just now but all this is already known, we know it isn't routing or access lists issue but httpd dead as you called it.

now clue what's the problem but that HA runs pretty much latest "takes" ...

Re: Problem accessing standby cluster member from non-local network

Kaspars_Zibarts — Tue, 06 Nov 2018 11:24:13 GMT

I just meant to say that this thread probably is irrelevant to your case so best would be to start new one