Maestro return traffic dropped by other SGM

[Maarten_Sjouw]

We have a strange issue where we have a server indirectly attached to a Maestro VSX environment and the VS has a host route for this host. Now when this host pings a specific host on the other side of the VS we see the traffic pass through, lets say SGM 1_2, but we see the return traffic being dropped by SGM 1_4.

With SSH and RDP sessions we sometimes see them completing and working and about 30% of the time also the return traffic is dropped.

The weird thing is that this is only happening between these specific hosts.

Other sessions all seem to work just fine.

Version MHO: R80.20SP JHF 304

Version SGM: R80.30SP JHF 49

Distribution mode: Auto Topology/L4 enabled, however L4 disabled has also been tested, same result.

[Kaspars_Zibarts]

What's the actual drop reason? No matching rule for return traffic? That is B -> A? It's been couple of months since I touched scalable platform and it was R76SP but feels like flow correction is failing for some reason? Can you manually calculate which SGMs are supposed to be involved? Not too sure though how it looks in R80.. 🙂

[Maarten_Sjouw]

The ping reply was dropped with a no corresponding ICMP request and for the other connections we get a out of state packet drop. We had the same idea that for some reason only for this specific pair the flow correction is just not working.

[Jones]

On the environment you mentioned, I did a test where I only allowed 1 SGM to be active, meaning that I disabled all the other SGM's. The traffic was flowing fine without any problems! Once I enabled the other SGM's again, the errors came back.

An example error:

On SGM 1_3 an Echo Request came from server A to server B. In the same second an Echo Reply came from SGM 1_1 that server B to server A that was dropped because of the message "ICMP reply does not match a previous request".

When I changed the Distribution Mode from Auto Topology to Manual General, traffic was flowing fine and the issue was resolved.

Kind Regards,

Eamon Jones

[Danny]

Same issue here on R81 JHF 42.

We set up a new Maestro single-site environment with two 7000 appliances running in Active/Active mode.
Return packets are dropped, even in Active/Down mode via clusterXL_admin down.

Stopping the other SG member via cpstop temporarily fixes the issue.

• Drop reason for SSH return packets:
  • action:Drop sport:443 ssh_version_2-Protocol-Signature
• Drop reason for VPN return packets (separate 3rd party VPN server in a DMZ)
  
  • action:Drop sport:4500 snmp-Protocol-Signature
  • 

The drops seem to appear from the other member that is not correctly synced.

Load Balancing / Distribution mode is set to policy (Default).
The VPN symptoms only appear if we change distribution mode on the relavent interface to network (we are doing this because of other Maestro issues).

[Zbynek]

Hello,

Is there any fix or solution for this issue?

I have exactly the same issue - new installation of maestro environment version R81.10 HFA #22.

Thanks for reply,

Zbynek

[Danny]

The solution is described here.

[melkool]

I have quite a similar issue.

Dual site, dual MHO  with 8 SGMs and two SGs

SG1 has only one SGM from site1

SG2 has one SGM from site 1 and one SGM from site 2 -> working fine

in SG1 when I add another SGM (it doesn't matter if it's from site 1 or site 2) some strange traffic issues are reported. Some connections (ping, http, rdp, ssh) are not working but only for some users and not for all of them.

As I could not identify issue with logging (as I can't (or I don't know) how to filter for members (like 1_1 or 1_2) I could not relate this to a return traffic issue.

I do have to mention that all interfaces are bond with Active/Standby (Activ in MHO1, Standby in MHO2) - identical for both site1 and site 2.

Also L4 distribution is disabled and Distribution mode is auto (per-port).

Any idea on how to start the investigation ?

Really appreciate your effort.

[Jochen_Hoechner]

Hi, 

1. what is the topology of the interfaces (inbound and outbound) involved to the communication?

2. did you check the routing? Is asymmetric routing in place? 
     What are the C2S interfaces in the communication flow,
     what are the S2C interfaces in the communication flow?
     Did you perform g_tcpdump for this traffic?

3. General question: Is the chassis performing Number of Hide NAT sessions? 

Thanks,
Jochen

[Ryan_Ryan]

Could you be kind and explain to me like I am 5 on how to resolve this?

My scenario

Site to Site vpn. From the same subnet on checkpoint I can connect to one of the remote subnets no problem, but to another subnet all the return traffic is dropped.

The working one the traffic goes out and back via SG1_02

Broken subnet it goes out via SG1_02 but comes back via SG1_03 and all protocols get dropped with an out of state error 

New to Maestro so not really understanding what I need to do to fix. thanks

show distribution configuration
Distribution Mode: auto-topology (per-port)