Lost Zabbix packets

[Mateusz_89]

Hey,
I have a problem with a CP 7000 cluster, Gaia R80.40.
The problem manifests itself in the loss of some packages from Zabbix. There are no rejected packets in the CP.. Most of the packets reach their destination, but some of them get lost along the way and cannot be seen on the destination servers. Between Zabbix and the servers there is only a CP cluster and switches. The traffic is on tcp 10050. When we change the port for the selected servers to a different one, the communication starts to work properly It looks like some queue is clogged or something like that when there are too many requests on tco 10050.

Please help diagnose the problem

[_Val_]

The first step is to make sure those packets are "lost" because of the firewall. Logs, traces, drop debugging, did you look into any of those?

[Mateusz_89]

Yes, I'm looking but the logs don't show dropped packets, fw ctl debug drop doesn't show anything either.
I thought it might be something related to:
https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...
but the proposed changes had no effect.
Behind the CP there are only switches and a server to which some packets do not reach, routing is ok because most sessions work properly so I assume that the problem is on the CP
Can you suggest what to check next?

[Chris_Atkinson]

Have you looked at the interface level counters using the likes of cpview or netstat -i / ifconfig / ethtool -S 

[_Val_]

If you do not see dropped packets, it is likely they just do not reach the FW. Try snooping outside of FW to prove that.

[Mateusz_89]

I can't catch packets before CP. The topology looks like this:
Zabbix - Cisco Nexus - CheckPoint Cluter (active-passive)-another Cisco Nexus - destination servers.
I do not have physical access to the infrastructure.
I catch packets on CP:
fw monitor -T -w -e "accept (src=10.120.58.98 and dst=10.120.61.148) or (src=10.120.61.148 and dst=10.120.58.98);" -o /var/log/test.cap
10.120.58.98 - zabbix
I  see only SYN 

[_Val_]

Assuming there is no NAT involved, a reason to see only SYN could be that the packets are accelerated. Use -F flag instead of -e to look for the accelerated packets as well. Mind the filtering with "-F", see sk30583 for more details.

[Mateusz_89]

fw monitor -F "10.120.58.98,0,10.120.61.148,0,0" -o /var/log/test.cap
Pcap screenshots in the attachments.

There is another DC in the infrastructure with the same devices. The pcap in the second DC looks very similar, there is also a lot of malformed packet and nothing else is visible in pcap. Traffic from zabbix to the servers in the other DC is working fine. There are the same models of CP and switches with the same firmware.
Any suggestions what else I can check?

[G_W_Albrecht]

Open a SR# with CP TAC! But i fear that only being able to sniffer on CP GW  will not help here...

[Timothy_Hall]

Wireshark is reporting the packets are malformed because fw monitor only captures the first 40 bytes of a packet (the snaplen) and not the whole thing, pass the -w flag to capture the entire packet and that warning will go away.  So that is a red herring.

All packets in your firewall capture are appearing 4 times at all 4 capture points so they are passing through the firewall just fine.  Please post the output of netstat -ni on the firewall to this thread; assuming packets are not being lost at the NIC level there it appears they are getting across the firewall just fine, and your problem lies elsewhere with an improperly defined bond or errors racking up on some interface somewhere.  You need to check the network error counters on all firewall/Nexus/Zabbix/Servers in the path, I guarantee you are taking interface errors somewhere which is why packets are randomly not making it.