topic Re: Strange ( periodic ) packet loss in Firewall and Security Management

Strange ( periodic ) packet loss

EVSolovyev — Fri, 14 Jan 2022 07:47:27 GMT

Good afternoon.

We have a cluster (HA) of 15600 with R80.40 actual JHF. Faced with a problem - some time after restarting the active node, starts losing packets. I.e. everything is working fine, then suddenly a couple (usually 2) packets are lost. Rebooting the active device - the problem goes away for a while, but then comes back.

Our scheme - the CP cluster by 10 Gb SFP+ port on interface card is connected to the core switch, from which VLANs are going through a large multi-storey building. The gateway for all is the CP.

Looked for errors on the CP port toward the switch - no errors. Decided to find a new little switch and try to connect a test segment of users through it for test (now serching for witch). fw ctl zdebug drop show nothing.... While we are looking for a switch, maybe you can offer some ideas on debugging, please.

On screen we pings gateway for this net (on on CP's VLAN port) and google DNS server.

Re: Strange ( periodic ) packet loss

HeikoAnkenbrand — Fri, 14 Jan 2022 08:48:32 GMT

Hi @EVSolovyev

I would check the following:
1) Is your internet connection ok before the firewall?
2) Check if you have errors on the interfaces (RX-OVR, RX-ERR, RX-DRP)
     # netstat -in
3) Is multi queueing enabled for the 10G interface?
     # mq_mng --show -v
4) You can see a high utilisation of the software interrupts (si) in the SecureXL instances.
     # fw ctl affinity -l -> Check which cores are used for SecureXL (Interfaces)
     # top + 1                 -> View the cores of the SecureXL instances
5) Running "show asset" command returns "Line Card Type: N/A" rather than properly identifying an installed 4 Port 10GBase-F SFP+
    > show asset
6) Do you see any interface errors in the file /var/log/messages
    cat /var/log/messages | grep "NETDEV"

Re: Strange ( periodic ) packet loss

genisis__ — Fri, 14 Jan 2022 09:10:03 GMT

Are these VSX appliances using virtual switches?

Re: Strange ( periodic ) packet loss

EVSolovyev — Fri, 14 Jan 2022 09:14:14 GMT

VSX is disabled and not used.

Re: Strange ( periodic ) packet loss

genisis__ — Fri, 14 Jan 2022 09:31:31 GMT

ok thanks, I have a similar issue with VSX and VSWs, but as your not running VSX no point in bring this to the table.

Can you confirm what jump your running? I would ensure your running at least JHFA125 (GA is JHFA139)

Re: Strange ( periodic ) packet loss

EVSolovyev — Fri, 14 Jan 2022 10:57:46 GMT

Re: Strange ( periodic ) packet loss

genisis__ — Fri, 14 Jan 2022 11:24:29 GMT

looks ok to me, I know we had issues with DNS packets and soon as we updated to that Jumbo issue was resolved.

can you also confirm no debugging is running 'fw ctl debug 0'. So we ensure its not a resource issues because of that.

Additionally silly checks like duplex settings

run this:

ifconfig -a | grep encap | awk '{print $1}' | grep -v lo | grep -v bond | grep -v ":" | grep -v ^lo | xargs -I % sh -c 'ethtool %; ethtool -i %' | grep '^driver\|Speed\|Duplex\|Setting' | sed "s/^/ /g" | tr -d "\t" | tr -d "\n" | sed "s/Settings for/\nSettings for/g" | awk '{print $5 " "$7 "\t " $9 "\t" $3}' | grep -v "Unknown" | grep -v "\."

and of course what Heiko has suggested to check.

Re: Strange ( periodic ) packet loss

EVSolovyev — Fri, 14 Jan 2022 11:39:48 GMT

Hello, @HeikoAnkenbrand

Thank you for the detailed answer.

1. I think, internet connected well - at this moment I see no droped packet to internet (ping about 2k packets).
2. There is no errors, but I see drops:

3. Multi queueing enabled on 4 cores:

4. In some times I see full utilization on some cores. But this is a rare occurrence and I have not yet been able to catch the process that does this. I think that fw_worker.

5. No, all is good:

6. No, there is nothing....:

But....:

What it can be?

Re: Strange ( periodic ) packet loss

Alex- — Fri, 14 Jan 2022 11:58:04 GMT

You mention it happens on the active device. Is it always the same device displaying this behavior, or does it happen on whatever machine becomes active after reboot?

In addition to all what was explained here, you might want to do a failover and run the hardware diagnostics tool during a maintenance window. It could indicate if it's an issue with your NIC's.

Re: Strange ( periodic ) packet loss

EVSolovyev — Fri, 14 Jan 2022 12:06:27 GMT

Duplex - first thing, that was checked. ) I see no CPU utilization.

Re: Strange ( periodic ) packet loss

EVSolovyev — Fri, 14 Jan 2022 12:11:05 GMT

It happen on whatever machine becomes active after reboot, but not immediately - after some time. May be some hours, or days. In CP cluster we have 2 devices, but core switch is a single device with multiple line cards.

Re: Strange ( periodic ) packet loss

genisis__ — Fri, 14 Jan 2022 12:15:34 GMT

I have similar issues where after a reboot I get packet loss after about 4 weeks, but this is on a VSX system and R&D have confirmed a bug.

You may want to log a TAC case, just in case its a bug; additionally perhaps installed the latest GA Jumbo as TAC will likely ask for this to be done.

Re: Strange ( periodic ) packet loss

HeikoAnkenbrand — Fri, 14 Jan 2022 12:56:01 GMT

It can be seen that the RX-DRP are high. If the errors continue to increase means that the SNDs can no longer handle the traffic.
I would give the system more SND cores in the first step change from 4 to 6 cores. Thus, more cores should also be used for multi queueing. Thus, more cores should also be used for multi queueing.

Re: Strange ( periodic ) packet loss

genisis__ — Fri, 14 Jan 2022 13:23:10 GMT

I agree with that, we increased our SND allocation to 6 cores; and we are running 15600s as well.

Re: Strange ( periodic ) packet loss

Timothy_Hall — Fri, 14 Jan 2022 14:59:23 GMT

First off you need to make sure your cluster is stable, as losing 2 ping packets in a row will generally happen when there is a non-graceful failover between members. What is the failover count shown by cphaprob state from expert mode?

RX-DRPs could be the source of the drops, but those could also be non-IPv4 packets hitting the interface and getting dropped. Please provide updated output of netstat -ni along with ethtool -S eth3 so we can distinguish between ring buffer drops and unknown protocol drops. Based on the fact you are getting them on all interfaces they are probably unknown protocol drops. If they are actually ring buffer drops you can look at the history of when that counter is getting incremented with sar -n EDEV to see if it is slowly incrementing, or coming in clumps when you are experiencing the loss.

The high CPU utilization on some workers could be caused by elephant flows, and any "mice" trapped on that worker core with an elephant flow will be degraded and possibly lose packets. Any elephant flows in the last 24 hours reported by running fw ctl multik print_heavy_conn?

Re: Strange ( periodic ) packet loss

EVSolovyev — Fri, 14 Jan 2022 17:51:47 GMT

I'm sorry, but I don't understand your advice.... My SND configuration:

Here we can see:

Best Practice - We recommend to allocate an additional CPU core to the CoreXL SND only if all these conditions are met:

There are at least 8 processing CPU cores.
In the output of the top command, the idle values for the CPU cores that run the CoreXL SND instances are in the 0%-5% range.
In the output of the top command, the sum of the idle values for the CPU cores that run the CoreXL Firewall instances is significantly higher than 100%.

If at least one of the above conditions is not met, the default CoreXL configuration is sufficient.

Whan I see cpview for CPU (screens are upper), I see, that about 100% utilization CPUs are mapped for fw_workers. I see never 100% utilization of SND cores.

And I'm sorry, but I can't understand, how to add 2 cores from fw to SND.... Am I need to decrease number of fw_workers in cpconfog only and free cores are automatically were added to SND after rebooting? Or to increase the number of SND cores I need to go some other way, which I have not yet been able to find?

Re: Strange ( periodic ) packet loss

EVSolovyev — Mon, 17 Jan 2022 11:34:28 GMT

Hello.

Thank you for answer.

Yes, cluster is stable. "What is the failover count shown by cphaprob state from expert mode?" Info is here. 15 failovers is due to collegues rebooting the active device when losses start to occur. Reboot solves the problem for a few days.

Yes, sometime we have a big traffic connections:

Re: Strange ( periodic ) packet loss

Timothy_Hall — Mon, 17 Jan 2022 13:06:37 GMT

The constant rate of RX-DRP reported by sar would seem to indicate the presence of non-IPv4 protocols and probably not packet loss due to SNDs being overloaded, please provide the output of ethtool -S eth1-01 to be sure.

Looks like you have quite a few elephant flows squashing "mice" connections when they get trapped on a worker core with an elephant flow which could be the source of your packet loss. Make sure that priority queueing is enabled for when workers get fully loaded by running this command: fw ctl multik prioq

Beyond that you'll need to upgrade to R81 or later to take advantage of the pipeline paths that can spread the processing of elephant flows across multiple worker cores.

Also please provide the output of fw ctl pstat just in case it is a resource limitation on the firewall.

Re: Strange ( periodic ) packet loss

Dolev — Mon, 17 Jan 2022 18:22:07 GMT

Hi @EVSolovyev,

Do you have or had a SR opened with Check Point support on this issue by any chance? If so, can you reply to me with the SR#?

Thanks,

Dolev

Re: Strange ( periodic ) packet loss

EVSolovyev — Tue, 18 Jan 2022 09:19:54 GMT

Good afternoon.

No, we have not opened an SR at this time. The problem is that our support ran out, and we did not renew it in time. Now the process of buying it is underway. We are a state company and such processes are very slow. If we had tech support, I would have opened SR right away.