Gateway not replying to pings

[gtzakis]

Hello all,

we upgraded one Security Gateway to R80.20 and we have a really strange behavior.

The gateway doesn't reply to ping requests. 

We see logs that the request is accepted, and the tcpdump and fwmonitor shows that the requests successfully reach the gateway, but both tcpdump and fwmonitor don't show replies. Also on zdebug we don't see any drops at all.

We disabled SecureXL with "fwaccel off", because it has caused some problems on others upgrades and the issue persists.

It is really weird, and we cannot think what may cause this problem.

Find below tcpdump output with some requests but without replies!

08:27:18.461003 IP 10.x.78.154 > 10.x.78.1: ICMP echo request, id 6556, seq 38729, length 87
08:27:19.462044 IP 10.x.78.154 > 10.x.78.1: ICMP echo request, id 6556, seq 38730, length 87
08:27:20.463021 IP 10.x.78.154 > 10.x.78.1: ICMP echo request, id 6556, seq 38731, length 87

The 10.x.78.1 is the VIP of the cluster, and the server with 10.x.78.154 is an esxi that has to ping the default gateway as a Keep Alive mechanism.

Can you think of something to investigate, because we have reached a wall.

Thank you all

[Maarten_Sjouw]

Do you have vMAC enabled? If not enable it and see how that works?
There is no impact in doing so, it will take a little while for the new vMAC to be used on all equipment, so give it a few minutes.
Also look at the latest jumbo. There have been a lot problems with R80.20, including memory leaks, that are resolved in the latest jumbo.

[Benedikt_Weissl]

Can you initiate the ping to the esxi from the gateway? Is IPS active? Do you see anything in the logfiles?

[gtzakis]

Yes we can initiate the ping from the gateway to esxi, and works fine.

IPS is not active, and the logs show that the icmp is accepted as excepted.

[Benedikt_Weissl]

Very strange. Did you check NAT-Rules and Anti-Spoofing configuration? Can you please post a censored screenshot of SmartLog?

[gtzakis]

Yes we checked all of them. NATing, antispoof, static routes, anything. We couldn't find anything that's why is so strange.

[Benedikt_Weissl]

Anything strange if you execute "dmesg"? Also check "netstat -s" starting at "Icmp:", you should see something like this

XXXX ICMP messages received
XXX input ICMP message failed.
ICMP input histogram:
destination unreachable: XXXX
timeout in transit: XX
echo requests: XXX
echo replies: XX
XX ICMP messages sent
X ICMP messages failed
ICMP output histogram:
destination unreachable: X
time exceeded: XX
echo request: X
echo replies: XXX.

[gtzakis]

Thank you for your help.

In dmesg we don't see anything weird about the interfaces and the IPs that we see the problem.

Here is the netstat -s output for icmp

Icmp:
4979831 ICMP messages received
36 input ICMP message failed.
ICMP input histogram:
destination unreachable: 28880
echo requests: 4950917
echo replies: 28
984185 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
destination unreachable: 70178
time exceeded: 27
echo request: 36
echo replies: 913944
IcmpMsg:
InType0: 28
InType3: 28880
InType8: 4950917
OutType0: 913944
OutType3: 70178
OutType8: 36
OutType11: 27

[Timothy_Hall]

How are you filtering your tcpdump?  The destination IP of the ping request is being NATted from 10.x.78.1 to 10.x.78.3, is 10.x.78.3 the dedicated IP address of the active firewall?  Is there an echo request coming back sourced from 10.x.78.3?

This sounds a bit like this:

sk26874: Cannot simultaneously ping Virtual IP address of the cluster and IP addresses of physical i...

[gtzakis]

Hello,

yes the 10.x.78.3 is the original IP of the gateway interface. And yes indeed, sometimes we see an echo request from the Gateway(10.x.78.3) targeting the servers.

The simultaneously ping was our first thought, this one of the first changes we made to fix the issue but made things even worse for some reason.

Also the tcpdump is filtered like "tcpdump -nni eth2.xx host <server-IP> and icmp".  The filter is alright we see the whole traffic(and replies) if there are any.