topic Re: Lost Zabbix packets in Firewall and Security Management

Lost Zabbix packets

Mateusz_89 — Fri, 27 Jan 2023 13:29:37 GMT

Hey,
I have a problem with a CP 7000 cluster, Gaia R80.40.
The problem manifests itself in the loss of some packages from Zabbix. There are no rejected packets in the CP.. Most of the packets reach their destination, but some of them get lost along the way and cannot be seen on the destination servers. Between Zabbix and the servers there is only a CP cluster and switches. The traffic is on tcp 10050. When we change the port for the selected servers to a different one, the communication starts to work properly It looks like some queue is clogged or something like that when there are too many requests on tco 10050.

Please help diagnose the problem

Re: Lost Zabbix packets

_Val_ — Fri, 27 Jan 2023 17:09:39 GMT

The first step is to make sure those packets are "lost" because of the firewall. Logs, traces, drop debugging, did you look into any of those?

Re: Lost Zabbix packets

Mateusz_89 — Fri, 03 Feb 2023 11:02:24 GMT

Yes, I'm looking but the logs don't show dropped packets, fw ctl debug drop doesn't show anything either.
I thought it might be something related to:
https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk24960
but the proposed changes had no effect.
Behind the CP there are only switches and a server to which some packets do not reach, routing is ok because most sessions work properly so I assume that the problem is on the CP
Can you suggest what to check next?

Re: Lost Zabbix packets

Chris_Atkinson — Fri, 03 Feb 2023 13:53:30 GMT

Have you looked at the interface level counters using the likes of cpview or netstat -i / ifconfig / ethtool -S

Re: Lost Zabbix packets

_Val_ — Fri, 03 Feb 2023 13:24:54 GMT

If you do not see dropped packets, it is likely they just do not reach the FW. Try snooping outside of FW to prove that.

Re: Lost Zabbix packets

Mateusz_89 — Fri, 10 Feb 2023 09:39:13 GMT

I can't catch packets before CP. The topology looks like this:
Zabbix - Cisco Nexus - CheckPoint Cluter (active-passive)-another Cisco Nexus - destination servers.
I do not have physical access to the infrastructure.
I catch packets on CP:
fw monitor -T -w -e "accept (src=10.120.58.98 and dst=10.120.61.148) or (src=10.120.61.148 and dst=10.120.58.98);" -o /var/log/test.cap
10.120.58.98 - zabbix
I see only SYN

Re: Lost Zabbix packets

_Val_ — Fri, 10 Feb 2023 12:00:54 GMT

Assuming there is no NAT involved, a reason to see only SYN could be that the packets are accelerated. Use -F flag instead of -e to look for the accelerated packets as well. Mind the filtering with "-F", see sk30583 for more details.

Re: Lost Zabbix packets

Mateusz_89 — Fri, 17 Feb 2023 08:32:32 GMT

fw monitor -F "10.120.58.98,0,10.120.61.148,0,0" -o /var/log/test.cap
Pcap screenshots in the attachments.

There is another DC in the infrastructure with the same devices. The pcap in the second DC looks very similar, there is also a lot of malformed packet and nothing else is visible in pcap. Traffic from zabbix to the servers in the other DC is working fine. There are the same models of CP and switches with the same firmware.
Any suggestions what else I can check?

Re: Lost Zabbix packets

G_W_Albrecht — Fri, 17 Feb 2023 09:45:30 GMT

Open a SR# with CP TAC! But i fear that only being able to sniffer on CP GW will not help here...

Re: Lost Zabbix packets

Timothy_Hall — Fri, 17 Feb 2023 14:33:28 GMT

Wireshark is reporting the packets are malformed because fw monitor only captures the first 40 bytes of a packet (the snaplen) and not the whole thing, pass the -w flag to capture the entire packet and that warning will go away. So that is a red herring.

All packets in your firewall capture are appearing 4 times at all 4 capture points so they are passing through the firewall just fine. Please post the output of netstat -ni on the firewall to this thread; assuming packets are not being lost at the NIC level there it appears they are getting across the firewall just fine, and your problem lies elsewhere with an improperly defined bond or errors racking up on some interface somewhere. You need to check the network error counters on all firewall/Nexus/Zabbix/Servers in the path, I guarantee you are taking interface errors somewhere which is why packets are randomly not making it.