Cannot ping only ove device after failover to backup member

Jean-Francois_G — Tue, 21 Feb 2023 22:00:37 GMT

Hello im running Gaia R81.10 Take 87 in cluster

When im on the primary firewall the server 192.168.10.xx can ping all device in network 192.168.71.0/24

Also the firewall can ping any device in the network 192.168.71.0/24

When we do a failover to the backup fireall the server 192.168.10.xx stop pinging 192.168.71.13 only. The server can ping all other device in the network 192.168.71.0/24 except 192.168.71.13. Also note that the backup firewall can ping the device 192.168.71.13

The device 192.168.71.13 is a Lantronix MSSLITE Version V3.6/4(000628)

When we failover to the primary firewall the server 192.168.10.xx can ping 192.168.71.13 again

Ive ran a tcpdump from the primary firewall

tcpdump -nni eth2.71 'host 192.168.71.13 and host 192.168.10.xx'

we are seing

ICMP echo request

ICMP echo reply

But when im running the same command on the backup firewall im only seing the

ICMP echo request and no echo reply from the 192.168.71.13

Ive also ran these 2 commands on the backup firewall and nothing was drop

fw ctl zdebug + drop | grep 192.168.71.13

fw ctl zdebug + drop | grep 192.168.10.xx

Any idea what could be the issue ?

Thanks !

Re: Cannot ping only ove device after failover to backup member

Chris_Atkinson — Tue, 21 Feb 2023 22:40:02 GMT

Have you checked things like the default gateway specified on the end device itself, does it reference the cluster VIP or a specific cluster members physical IP in error?

Also what does the firewall and device think about the ARP entries it has for the relevant addresses...

Re: Cannot ping only ove device after failover to backup member

Jean-Francois_G — Wed, 22 Feb 2023 15:09:54 GMT

Yes the default GW is the VIP

#When firewall main active
[Expert@infFire:0]# arp | grep 192.168.71.13
192.168.71.13 ether 00:80:a3:67:51:7e C eth2.71

[Expert@infFire:0]# ping 192.168.71.13
PING 192.168.71.13 (192.168.71.13) 56(84) bytes of data.
64 bytes from 192.168.71.13: icmp_seq=1 ttl=59 time=9.85 ms
64 bytes from 192.168.71.13: icmp_seq=2 ttl=59 time=9.20 ms
64 bytes from 192.168.71.13: icmp_seq=3 ttl=59 time=9.62 ms
64 bytes from 192.168.71.13: icmp_seq=4 ttl=59 time=9.64 ms
^C
--- 192.168.71.13 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3003ms
rtt min/avg/max/mdev = 9.200/9.581/9.853/0.266 ms

[Expert@infFire-Backup:0]# arp | grep 192.168.71.13

192.168.71.13 ether 00:80:a3:67:51:7e C eth2.71

#When firewall backup active
[Expert@infFire:0]# arp | grep 192.168.71.13
192.168.71.13 ether 00:80:a3:67:51:7e C eth2.71

[Expert@infFire-Backup:0]# arp | grep 192.168.71.13
192.168.71.13 ether 00:80:a3:67:51:7e C eth2.71

[Expert@infFire-Backup:0]# ping 192.168.71.13
PING 192.168.71.13 (192.168.71.13) 56(84) bytes of data.
64 bytes from 192.168.71.13: icmp_seq=1 ttl=60 time=12.6 ms
64 bytes from 192.168.71.13: icmp_seq=2 ttl=60 time=12.5 ms
64 bytes from 192.168.71.13: icmp_seq=3 ttl=60 time=3.49 ms
64 bytes from 192.168.71.13: icmp_seq=4 ttl=60 time=11.3 ms
^C
--- 192.168.71.13 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 3.495/10.021/12.668/3.804 ms

If we try to ping 192.168.71.13 from any pc that work when on the primary firewall it don't ping

Here is what im seeing with tcpdump

tcpdump -nni eth2.71 'host 192.168.71.13 and host 192.168.10.xx'

10:01:45.935176 IP 192.168.71.13.3001 > 192.168.10.xx.6250: Flags [.], ack 648, win 3584, length 0
10:01:45.955295 IP 192.168.71.13.3001 > 192.168.10.xx.6250: Flags [P.], seq 1401:1409, ack 648, win 3584, length 8
10:01:46.004000 IP 192.168.10.xx.6250 > 192.168.71.13.3001: Flags [.], ack 1409, win 63035, length 0
10:01:47.770382 IP 192.168.10.xx > 192.168.71.13: ICMP echo request, id 1, seq 6554, length 40
10:01:47.777560 IP 192.168.71.13 > 192.168.10.xx: ICMP echo reply, id 1, seq 6554, length 40
10:01:47.957213 IP 192.168.10.xx.6250 > 192.168.71.13.3001: Flags [P.], seq 648:657, ack 1409, win 63035, length 9
10:01:48.017390 IP 192.168.71.13.3001 > 192.168.10.xx.6250: Flags [P.], seq 1409:1465, ack 657, win 3584, length 56
10:01:48.066413 IP 192.168.10.xx.6250 > 192.168.71.13.3001: Flags [.], ack 1465, win 62979, length 0
10:01:50.035461 IP 192.168.10.xx.6250 > 192.168.71.13.3001: Flags [P.], seq 657:666, ack 1465, win 62979, length 9
10:01:50.088928 IP 192.168.71.13.3001 > 192.168.10.xx.6250: Flags [.], ack 666, win 3584, length 0
10:01:50.098961 IP 192.168.71.13.3001 > 192.168.10.xx.6250: Flags [P.], seq 1465:1473, ack 666, win 3584, length 8
10:01:50.144595 IP 192.168.10.xx.6250 > 192.168.71.13.3001: Flags [.], ack 1473, win 62971, length 0
10:01:52.113583 IP 192.168.10.xx.6250 > 192.168.71.13.3001: Flags [P.], seq 666:675, ack 1473, win 62971, length 9
10:01:52.171054 IP 192.168.71.13.3001 > 192.168.10.xx.6250: Flags [P.], seq 1473:1481, ack 675, win 3584, length 8
10:01:52.222841 IP 192.168.10.xx.6250 > 192.168.71.13.3001: Flags [.], ack 1481, win 62963, length 0
10:01:52.770650 IP 192.168.10.xx > 192.168.71.13: ICMP echo request, id 1, seq 6555, length 40
10:01:52.782173 IP 192.168.71.13 > 192.168.10.xx: ICMP echo reply, id 1, seq 6555, length 40
10:01:54.191719 IP 192.168.10.xx.6250 > 192.168.71.13.3001: Flags [P.], seq 675:684, ack 1481, win 62963, length 9
10:01:54.252931 IP 192.168.71.13.3001 > 192.168.10.xx.6250: Flags [P.], seq 1481:1510, ack 684, win 3584, length 29
10:01:54.301080 IP 192.168.10.xx.6250 > 192.168.71.13.3001: Flags [.], ack 1510, win 62934, length 0
10:01:56.270478 IP 192.168.10.xx.6250 > 192.168.71.13.3001: Flags [P.], seq 684:693, ack 1510, win 62934, length 9
10:01:56.324751 IP 192.168.71.13.3001 > 192.168.10.xx.6250: Flags [.], ack 693, win 3584, length 0
10:01:56.334775 IP 192.168.71.13.3001 > 192.168.10.xx.6250: Flags [P.], seq 1510:1541, ack 693, win 3584, length 31
10:01:56.379123 IP 192.168.10.xx.6250 > 192.168.71.13.3001: Flags [.], ack 1541, win 62903, length 0
10:01:57.770500 IP 192.168.10.xx > 192.168.71.13: ICMP echo request, id 1, seq 6556, length 40
10:01:57.776662 IP 192.168.71.13 > 192.168.10.xx: ICMP echo reply, id 1, seq 6556, length 40
10:01:58.348143 IP 192.168.10.xx.6250 > 192.168.71.13.3001: Flags [P.], seq 693:702, ack 1541, win 62903, length 9
10:01:58.406776 IP 192.168.71.13.3001 > 192.168.10.xx.6250: Flags [P.], seq 1541:1572, ack 702, win 3584, length 31
10:01:58.457355 IP 192.168.10.xx.6250 > 192.168.71.13.3001: Flags [.], ack 1572, win 62872, length 0
10:02:00.426422 IP 192.168.10.xx.6250 > 192.168.71.13.3001: Flags [P.], seq 702:711, ack 1572, win 62872, length 9
10:02:00.488548 IP 192.168.71.13.3001 > 192.168.10.xx.6250: Flags [P.], seq 1572:1580, ack 711, win 3584, length 8
10:02:00.535582 IP 192.168.10.xx.6250 > 192.168.71.13.3001: Flags [.], ack 1580, win 62864, length 0

Those 2 commands don't show any drop also

fw ctl zdebug + drop | grep 192.168.71.13

fw ctl zdebug + drop | grep 192.168.10.xx

[Expert@infFire-Backup:0]# fw monitor -e 'accept host(192.168.10.xx) and host(192.168.71.13);'
PPAK 0: Get before set operation succeeded of fwmonitor_kiss_enable
PPAK 0: Get before set operation succeeded of simple_debug_filter_off
PPAK 0: Get before set operation succeeded of kiss_debug_force_kdprintf_enable
PPAK 0: Get before set operation succeeded of fwmonitorfreebufs
************************************************************** NOTE **************************************************************
*** Using "-e" filter will not monitor accelerated traffic. To monitor and filter accelerated traffic please use the "-F" filter ***
************************************************************************************************************************************
FW monitor will record only ip & transport layers in a packet
For capturing the whole packet please do -w
PPAK 0: Get before set operation succeeded of fwmonitor_ppak_all_position
monitor: getting filter (from command line)
monitor: compiling
monitorfilter:
Compiled OK.
monitor: loading
monitor: monitoring (control-C to stop)
PPAK 0: Get before set operation succeeded of fwmonitormaxpacket
PPAK 0: Get before set operation succeeded of fwmonitormask
PPAK 0: Get before set operation succeeded of fwmonitorallocbufs
PPAK 0: Get before set operation succeeded of printuuid
[vs_0][fw_1] eth2.100:i[44]: 192.168.10.xx -> 192.168.71.13 (ICMP) len=60 id=62744
ICMP: type=8 code=0 echo request id=1 seq=6627
[vs_0][fw_1] eth2.100:I[44]: 192.168.10.xx -> 192.168.71.13 (ICMP) len=60 id=62744
ICMP: type=8 code=0 echo request id=1 seq=6627
[vs_0][fw_1] eth2.71:o[44]: 192.168.10.xx -> 192.168.71.13 (ICMP) len=60 id=62744
ICMP: type=8 code=0 echo request id=1 seq=6627
[vs_0][fw_1] eth2.71:O[44]: 192.168.10.xx -> 192.168.71.13 (ICMP) len=60 id=62744
ICMP: type=8 code=0 echo request id=1 seq=6627
[vs_0][fw_2] eth2.100:i[44]: 192.168.10.xx -> 192.168.71.13 (ICMP) len=60 id=62749
ICMP: type=8 code=0 echo request id=1 seq=6628
[vs_0][fw_2] eth2.100:I[44]: 192.168.10.xx -> 192.168.71.13 (ICMP) len=60 id=62749
ICMP: type=8 code=0 echo request id=1 seq=6628
[vs_0][fw_2] eth2.71:o[44]: 192.168.10.xx -> 192.168.71.13 (ICMP) len=60 id=62749
ICMP: type=8 code=0 echo request id=1 seq=6628
[vs_0][fw_2] eth2.71:O[44]: 192.168.10.xx -> 192.168.71.13 (ICMP) len=60 id=62749
ICMP: type=8 code=0 echo request id=1 seq=6628
[vs_0][fw_0] eth2.100:i[44]: 192.168.10.xx -> 192.168.71.13 (ICMP) len=60 id=62756
ICMP: type=8 code=0 echo request id=1 seq=6629
[vs_0][fw_0] eth2.100:I[44]: 192.168.10.xx -> 192.168.71.13 (ICMP) len=60 id=62756
ICMP: type=8 code=0 echo request id=1 seq=6629
[vs_0][fw_0] eth2.71:o[44]: 192.168.10.xx -> 192.168.71.13 (ICMP) len=60 id=62756
ICMP: type=8 code=0 echo request id=1 seq=6629
[vs_0][fw_0] eth2.71:O[44]: 192.168.10.xx -> 192.168.71.13 (ICMP) len=60 id=62756
ICMP: type=8 code=0 echo request id=1 seq=6629
[vs_0][fw_0] eth2.100:i[44]: 192.168.10.xx -> 192.168.71.13 (ICMP) len=60 id=62761
ICMP: type=8 code=0 echo request id=1 seq=6630
[vs_0][fw_0] eth2.100:I[44]: 192.168.10.xx -> 192.168.71.13 (ICMP) len=60 id=62761
ICMP: type=8 code=0 echo request id=1 seq=6630
[vs_0][fw_0] eth2.71:o[44]: 192.168.10.xx -> 192.168.71.13 (ICMP) len=60 id=62761
ICMP: type=8 code=0 echo request id=1 seq=6630
[vs_0][fw_0] eth2.71:O[44]: 192.168.10.xx -> 192.168.71.13 (ICMP) len=60 id=62761
ICMP: type=8 code=0 echo request id=1 seq=6630
^C monitor: caught sig 2
monitor: unloading
PPAK 0: Get before set operation succeeded of fwmonitor_kiss_enable
PPAK 0: Get before set operation succeeded of simple_debug_filter_off
PPAK 0: Get before set operation succeeded of kiss_debug_force_kdprintf_enable
PPAK 0: Get before set operation succeeded of fwmonitorfreebufs
[Expert@infFire-Backup:0]#

And as soon that i failover back to my primary firewall my server 192.168.10.xx can ping 192.168.71.13

Thanks for helping me

Re: Cannot ping only ove device after failover to backup member

Jean-Francois_G — Wed, 22 Feb 2023 15:42:17 GMT

Yes Gateway is the VIP

arp is the same on both firewall

[Expert@infFire:0]# arp | grep 192.168.71.13
192.168.71.13 ether 00:80:a3:67:51:7e C eth2.71

[Expert@infFire-Backup:0]# arp | grep 192.168.71.13
192.168.71.13 ether 00:80:a3:67:51:7e C eth2.71

topic Cannot ping only ove device after failover to backup member in Firewall and Security Management

Cannot ping only ove device after failover to backup member

Re: Cannot ping only ove device after failover to backup member

Re: Cannot ping only ove device after failover to backup member

Re: Cannot ping only ove device after failover to backup member