topic Re: Standby member no internet in Firewall and Security Management

Standby member no internet

melcu — Sat, 28 Sep 2024 17:04:38 GMT

Hi Mates,

So it's been discussed a lot but my story is a little bit different. I have a client with a bunch of Active/Standby ClusterXL clusters in which the Standby member cannot access he internet at all.

Long story short: I almost ran out of search keywords in this forum and on google regarding the issue. First of all, sk43807 was followed line-by-line with no luck. then fwha_forw_packet_to_not_active 1/0 - no change at all and this is why! - please see the diagram. There is more than 1 interface but you get the picture.

Both members are running only on private IP addresses. All traffic is NAT hidden behind a public IP address and the CORE router knows to route the /32 of that public IP address to the VIP address of the cluster. When the ACTIVE node (doesn't matter, fw1 or fw2) sends any packets it's NAT-ed behind that public IP address and sent on it's way. The return traffic is forwarded by the router to the VIP which and everything works (as VIP is bounded to the Active member).

When the Standby member tries to access everything I can see (and I'm very sorry but I cannot put real captures here due to IP address privacy) that packets that originates from Standby are forwarded to the Active member over the SYNC interface. The Active member then matches the traffic to it's rulebase, applies NAT and packets go out to CORE and then to internet. The return traffic is funny. It arrives on the Active member and there vanishes. It's not dropped (fw ctl zdebug +drop) , it simple vanishes and is not forwarded to the Standby member (which is a function by design I presume).

So eventually I've lost all my hops in making this work.

Any help or guidance will really be apreciated.

Wish all the best,

Re: Standby member no internet

_Val_ — Sun, 29 Sep 2024 10:51:58 GMT

Forwarding outbound traffic through the sync interface is correct in this case. Return packet disappearing without forwarding back is not. Check if you have any associated drops. Also, if stuck, please open a support request.

Re: Standby member no internet

AkosBakos — Sun, 29 Sep 2024 11:11:17 GMT

Hi @melcu

Run am #fw ctl zdebug + drop on the standby member. Maybe will appear something meaningful.

Re: Standby member no internet

the_rock — Sun, 29 Sep 2024 13:38:14 GMT

If I were you, apart from what the guys already asked, I would make sure routes are exactly the same on both members. Also, to confirm, navigate to cluster object in smart console, open network, then topology and click get interfaces WITHOUT topology, just to verify there are no errors.

Andy

Re: Standby member no internet

melcu — Sun, 29 Sep 2024 13:42:13 GMT

I've created a lab with the exact IP schema from the above diagram. Pinging google DNS I can see this on the Active member, but nothing on the standby member. See the attached.

Later edit: Also interfaces and routes are identical.

[Expert@gw01:0]# ip ro ls
10.134.0.0/24 dev eth1 proto kernel scope link src 10.134.0.11
10.144.70.0/24 dev eth0 proto kernel scope link src 10.144.70.21
10.144.0.0/16 via 10.144.70.1 dev eth0 proto routed
default via 10.134.0.1 dev eth1 proto routed

[Expert@gw02:0]# ip ro ls
10.134.0.0/24 dev eth1 proto kernel scope link src 10.134.0.12
10.144.70.0/24 dev eth0 proto kernel scope link src 10.144.70.22
10.144.0.0/16 via 10.144.70.1 dev eth0 proto routed
default via 10.134.0.1 dev eth1 proto routed

Getting interfaces with or without topology goes well. At least in my lab as I don't have any access in their environment. Here I can do whatever I want as it's a lab .. no impact 🙂

Re: Standby member no internet

the_rock — Sun, 29 Sep 2024 13:40:48 GMT

Can you send output of below command from expert mode of both members please?

Andy

ip r g 8.8.8.8

Re: Standby member no internet

the_rock — Sun, 29 Sep 2024 13:49:17 GMT

@melcu

For the context this is what I see in my lab cluster and fw2 is standby currently.

Andy

[Expert@CP-FW-01:0]#
[Expert@CP-FW-01:0]# ip r g 8.8.8.8
8.8.8.8 via 172.16.10.1 dev eth0 src 172.16.10.248
cache
[Expert@CP-FW-01:0]# ssh admin@172.16.10.247
admin@172.16.10.247's password:
Last login: Fri Sep 27 08:35:02 2024 from 100.65.16.2
[Expert@CP-FW-02:0]# ip r g 8.8.8.8
8.8.8.8 via 172.16.10.1 dev eth0 src 172.16.10.247
cache
[Expert@CP-FW-02:0]# ^C
[Expert@CP-FW-02:0]# cphaprob roles

ID Role

1 Master
2 (local) Non-Master

[Expert@CP-FW-02:0]# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=113 time=9.13 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=113 time=6.36 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=113 time=6.00 ms
^C
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 6.008/7.169/9.133/1.399 ms
[Expert@CP-FW-02:0]#

Re: Standby member no internet

melcu — Sun, 29 Sep 2024 13:52:03 GMT

[Expert@gw01:0]# ip ro get 8.8.8.8
8.8.8.8 via 10.134.0.1 dev eth1 src 10.134.0.11
cache ipid 0x40ef mtu 1500 advmss 1460 hoplimit 64

[Expert@gw02:0]# ip ro get 8.8.8.8
8.8.8.8 via 10.134.0.1 dev eth1 src 10.134.0.12
cache mtu 1500 advmss 1460 hoplimit 64

fw1 is MASTER

fw2 is NON-MASTER (as it's the standby unit) If I flip them internet works in FW2 but not on FW1 🙂

Re: Standby member no internet

the_rock — Sun, 29 Sep 2024 14:00:48 GMT

I have a gut feeling I know what could be wrong. So its 100% NOT the specific member if an issue happens regardless which is master.

Can you check how below is set? No need to send a screenshot, just verify.

Andy

Re: Standby member no internet

the_rock — Sun, 29 Sep 2024 14:04:17 GMT

@melcu I know this may sound silly (trivial), but can you confirm 100% that you do indeed have a proper rule in smart console allowing the traffic?

Andy

Re: Standby member no internet

melcu — Sun, 29 Sep 2024 14:08:55 GMT

In LAB is wire. Specific rule for VIP and members to access internet and the unsafe from ANY to members and CLU (but public IP is protected by an IPS profile - just in case).

Re: Standby member no internet

the_rock — Sun, 29 Sep 2024 14:16:27 GMT

To clear ANY doubts, run ping in one ssh window, then below in another and send what you get.

Andy

[Expert@CP-FW-02:0]# fw up_execute dst=8.8.8.8 ipp=0
Rulebase execution ended successfully.
Overall status:
----------------
Active clob mask: 2
Required clob mask: 0
Match status: POSSIBLE
Match action: Accept

Per Layer:
------------
Layer name: network
Layer id: 0
Match status: POSSIBLE
Match action: Accept
Possible rules: 3 4 6 7 8 9 16777215

Layer name: appc+urlf
Layer id: 6
Match status: MATCH
Match action: Accept
Matched rule: 5
Possible rules: 5 16777215

Layer name: content-awareness-layer
Layer id: 3
Match status: MATCH
Match action: Accept
Matched rule: 1
Matched rules: 1

Layer name: final-allow-layer
Layer id: 7
Match status: MATCH
Match action: Accept
Matched rule: 1
Matched rules: 1

[Expert@CP-FW-02:0]#

Re: Standby member no internet

melcu — Sun, 29 Sep 2024 14:19:17 GMT

Re: Standby member no internet

the_rock — Sun, 29 Sep 2024 14:21:36 GMT

K, so if its accepted, rules are fine. Not sure then, maybe some kernel parameter...lets see if anyone else may have an idea. Anyway, I have to go now, get ready for some biking event.

Hope you find the resolution soon.

Andy

Re: Standby member no internet

melcu — Sun, 29 Sep 2024 14:32:24 GMT

Told ya! It's driving me crazy!

Even looking in the logs it shows that traffic is accepted by fw01, is NATted by the public IP and goes out. Even in the Fortigate I see traffic coming from the NAT IP (when initiated by the standby member) but when it returns it gets dropped or something by fw01.

Perimetral firewall (Fortigate- but irrelevant):

FGT-FW01 (vdom-) # diagnose sniffer packet any 'host 8.8.8.8 and host 91.208.215.149'
interfaces=[any]
filters=[host 8.8.8.8 and host 91.208.215.149]
11.250797 91.208.215.149 -> 8.8.8.8: icmp: echo request
11.250813 91.208.215.149 -> 8.8.8.8: icmp: echo request
11.250814 91.208.215.149 -> 8.8.8.8: icmp: echo request
11.280356 8.8.8.8 -> 91.208.215.149: icmp: echo reply
11.280358 8.8.8.8 -> 91.208.215.149: icmp: echo reply
11.280368 8.8.8.8 -> 91.208.215.149: icmp: echo reply
11.280369 8.8.8.8 -> 91.208.215.149: icmp: echo reply

So it sees the traffic originating from standby node and NATTed behind 91.208.215.149

The return traffic reaches the VIP (on the active node)

[Expert@gw01:0]# tcpdump -vvv -ni eth1 host 8.8.8.8
tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 96 bytes
17:27:25.095506 IP (tos 0x0, ttl 59, id 0, offset 0, flags [none], proto: ICMP (1), length: 84) 8.8.8.8 > 91.208.215.149: ICMP echo reply, id 10267, seq 8, length 64
17:27:25.095623 IP (tos 0x0, ttl 58, id 0, offset 0, flags [none], proto: ICMP (1), length: 84) 8.8.8.8 > 91.208.215.149: ICMP echo reply, id 10267, seq 8, length 64
17:27:25.095908 IP (tos 0x0, ttl 57, id 0, offset 0, flags [none], proto: ICMP (1), length: 84) 8.8.8.8 > 91.208.215.149: ICMP echo reply, id 10267, seq 8, length 64
17:27:25.095948 IP (tos 0x0, ttl 56, id 0, offset 0, flags [none], proto: ICMP (1), length: 84) 8.8.8.8 > 91.208.215.149: ICMP echo reply, id 10267, seq 8, length 64
17:27:25.096033 IP (tos 0x0, ttl 55, id 0, offset 0, flags [none], proto: ICMP (1), length: 84) 8.8.8.8 > 91.208.215.149: ICMP echo reply, id 10267, seq 8, length 64
17:27:25.096062 IP (tos 0x0, ttl 54, id 0, offset 0, flags [none], proto: ICMP (1), length: 84) 8.8.8.8 > 91.208.215.149: ICMP echo reply, id 10267, seq 8, length 64
17:27:25.096142 IP (tos 0x0, ttl 53, id 0, offset 0, flags [none], proto: ICMP (1), length: 84) 8.8.8.8 > 91.208.215.149: ICMP echo reply, id 10267, seq 8, length 64
17:27:25.096177 IP (tos 0x0, ttl 52, id 0, offset 0, flags [none], proto: ICMP (1), length: 84) 8.8.8.8 > 91.208.215.149: ICMP echo reply, id 10267, seq 8, length 64
17:27:25.096261 IP (tos 0x0, ttl 51, id 0, offset 0, flags [none], proto: ICMP (1), length: 84) 8.8.8.8 > 91.208.215.149: ICMP echo reply, id 10267, seq 8, length 64

But on the standby member .. mumu 🙂 sees only the outgoing packets but nothing back.

[Expert@gw02:0]# tcpdump -vni any host 8.8.8.8
tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 96 bytes
17:30:39.739541 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto: ICMP (1), length: 84) 91.208.215.149 > 8.8.8.8: ICMP echo request, id 10325, seq 176, length 64
17:30:40.739875 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto: ICMP (1), length: 84) 91.208.215.149 > 8.8.8.8: ICMP echo request, id 10325, seq 177, length 64
17:30:41.740411 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto: ICMP (1), length: 84) 91.208.215.149 > 8.8.8.8: ICMP echo request, id 10325, seq 178, length 64
17:30:42.739779 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto: ICMP (1), length: 84) 91.208.215.149 > 8.8.8.8: ICMP echo request, id 10325, seq 179, length 64

First I thought that the packet will return to a different interface and that's why I've used "-i any". It doesn't come back from FW01.

Pretty sure this is Kernel issue. Btw, even turning fwaccell off doesn't solve.

Re: Standby member no internet

the_rock — Sun, 29 Sep 2024 16:56:40 GMT

Just came back from my race...I thought about this while running and now that I checked your topology, I am certain your issue is the fact you have external if configured as sync. Can you create SEPARATE sync interface and test?

Andy

Re: Standby member no internet

melcu — Sun, 29 Sep 2024 17:05:13 GMT

Hmm SYNC was not on this interface

Before:
eth0       UP                    sync(secured), unicast
eth1       UP                    non sync(non secured), unicast

Virtual cluster interfaces: 2
eth0            10.144.70.20
eth1            10.134.0.10

After:
eth0       UP                    non sync(non secured), unicast
eth1       UP                    sync(secured), unicast

Virtual cluster interfaces: 2

eth0            10.144.70.20
eth1            10.134.0.10

Behavior is the same..

Just tested. ClusterXL_admin down on FW1 and FW2 instantly reaches internet.

SO definitely something in the Kern.

Re: Standby member no internet

the_rock — Sun, 29 Sep 2024 18:47:45 GMT

But then how come your topology shows it is?? Can you send topology screenshot again? And also commands below from BOTH members?

Andy

If you got time, lets do remote, since it a lab

Let me know

[Expert@CP-FW-01:0]# cphaprob -a if

CCP mode: Manual (Unicast)
Required interfaces: 4
Required secured interfaces: 1

Interface Name: Status:

eth0 (LM) UP
eth1 (LM) UP
eth2 (LM) UP
eth3 (S) UP

S - sync, HA/LS - bond type, LM - link monitor, P - probing

Virtual cluster interfaces: 3

eth0 172.16.10.246
eth1 192.168.10.246
eth2 172.31.10.246

[Expert@CP-FW-01:0]# cphaprob -i list

There are no pnotes in problem state

[Expert@CP-FW-01:0]# cphaprob syncstat

Delta Sync Statistics

Sync status: OK

Drops:
Lost updates................................. 0
Lost bulk update events...................... 0
Oversized updates not sent................... 0

Sync at risk:
Sent reject notifications.................... 0
Received reject notifications................ 0

Sent messages:
Total generated sync messages................ 1458146
Sent retransmission requests................. 0
Sent retransmission updates.................. 0
Peak fragments per update.................... 1

Received messages:
Total received updates....................... 3962798
Received retransmission requests............. 0

Sync Interface:
Name......................................... eth3
Link speed................................... 1000Mb/s
Rate......................................... 25340 [Bps]
Peak rate.................................... 802520[Bps]
Link usage................................... 0%
Total........................................ 27707 [MB]

Queue sizes (num of updates):
Sending queue size........................... 512
Receiving queue size......................... 256
Fragments queue size......................... 50

Timers:
Delta Sync interval (ms)..................... 100

Reset on Sun Sep 22 12:15:49 2024 (triggered by fullsync).

[Expert@CP-FW-01:0]#

Re: Standby member no internet

the_rock — Sun, 29 Sep 2024 21:06:57 GMT

Forgot to mention, IF topology shows something different, go to smart console cluster object, network and then edit topology, click "get interfaces WITHOUT topology", make sure it saves without errors, publish, install policy, test.

Andy

Re: Standby member no internet

PhoneBoy — Mon, 30 Sep 2024 13:59:27 GMT

So an upstream router is basically providing the NAT here only for the public VIP?
I'm with @AkosBakos, we probably need to see fw ctl zdebug + drop output from the active member while trying to initiate communication from the secondary.