Gateway on VMWare ESXi 6.X - Policy Push Randomly Fails - Error no. 10

[B_P]

We have had an issue since the beginning with our ESXi gateways where policy push randomly fails with the error: "Installation failed. Reason: TCP connectivity failure ( port = 18191 )( IP = 10.90.45.30 )[error no. 10].". Sometimes policy push works fine, the next minute we'll get the error. It doesn't really matter when you're pushing, if it's just Access or Threat policy, it's a roll of the die if it will work.

We have six separate ESXi hosts, one gateway on each that make up three separate clusters. This issue happens on three out of the six gateways. In one cluster, if you set the "backup" gateway to primary, it pushes 100% of the time. Another cluster works 100% of the time regardless of which is the primary. Comparing the configuration between all of them, we cannot see any real difference between them. We thought it was TSO or something but no luck. Management isn't the source of the issue because it can push just fine to some of the systems 100% of the time. There's no firewall between Management and the gateways.

Also, you cannot transfer large files from the gateways, anything over about 10MB fails with an error like "Incorrect MAC received on packet" with WinSCP, etc. Get a similar error with Putty sometimes. Note, that's MAC as in "message authentication code" with SSH.

All of the systems that work are Dell. The rest are HP and one Fujitsu. We've dug into the BIOS to see if there were NIC settings, etc, but can't find anything.

All of the VMs use VMXNET3. We have Promiscuous mode, MAC address changes and Forged transmits enabled on the Port Groups. E1000 NICs don't help.

Using ethtool -S nic# we see that the Tx Queue "ring full" is > 0 and Rx Queue "pkts rx out of buf" is > 0 on all gateways that have issues. Working gateways those values are 0.

[PhoneBoy]

Since you're using ESXi, the underlying hardware shouldn't matter.

What version/JHF level are the gateways in question?

One SK to check: Policy installation fails with 'TCP connection failure port=18191 [error no. 10]' and &apo... 

[B_P]

Picking this back up..... on R80.20GA (JHF 10) now and still having the issue. I ruled out SK98096.

[b95c3808-bd23-4]

I have the same exact issues. R80.10 SMS and R80.10 GW, both on VMWare 6.5.

I have tried to move these machines from one ESXI host to others, part of the same VMWare cluster, but the same issues persists. Servers where VMWare is installed are Dell PowerEdge R740. Did you find a solution for this kind of issues?

P.S: SMS seems fine, I think the only issues are within the GW.

BTW even the checkpoint support didn't find the root cause of the issues.

[B_P]

The vmware tools installed with R80.20 appear to be from the ESXi v4.x generation. The installed vmxnet drivers look ancient. Check Point says in the HCL that R80.20 is compatible with vmware esxi v6.7 but it really looks like it's built for v4.x. The E1000 drivers installed even seem to be from the ESXi v3 generation. I bet using decade old drivers isn't helping the situation here.

Also, I wonder how many vulnerabilities are in these old drivers?

[B_P · Accepted solution]

This ended up being an issue with AES-NI support in R80 running on VMware. It has been fixed in R80.30 (and a hotfix is now available for R80.20 take 33).