topic Re: BW Saturation in Firewall and Security Management

BW Saturation

Matlu — Wed, 29 Nov 2023 18:21:20 GMT

Hello,

We have a problem with our Internet service.

We currently have a network design similar to this:

LAN -> INT_CLUSTER -> EXT_CLUSTER -> INTERNET.

The contracted BW is 500MB, but the LAN network, is having slowness problems.
The ISP told us that the link is getting saturated, and this is maybe due to a bad practice of some of the LAN users.

Is there any way to know, which is the IP that is saturating the Internet link, from the Firewall point of view?

We have ClusterXL HA in version R81.10 with Take 110.

Greetings.

BW Saturation

Re: BW Saturation

the_rock — Thu, 30 Nov 2023 01:52:46 GMT

Buddy, what has been done so far? Have you ran any captures, checked interface errors, anything at all? Without at least basic info, it would be purely a guess as to what can be causing this.

Best regards,

Andy

Re: BW Saturation

Matlu — Thu, 30 Nov 2023 01:59:13 GMT

Hi, Andy.

I checked commands like CPview, where I noticed that the Hardware resources were "stable".

I checked the command "netstat -ni", but the result of this, I did not understand it well.
This command showed a column of "RX-ERR" and the interface facing the Internet, this column did "show" a numerical value.

I suspect this may be an "indication" that there is a problem at the ISP level.

Re: BW Saturation

the_rock — Thu, 30 Nov 2023 02:11:58 GMT

The best thing to do is see if the issue happens when you take CP firewall out of the equation. If it does, then its not the firewall, if the problem does not happen, then you know its the firewall issue and need to look further if its on the hw or software level.

Here are some commands to run.

ps -auxw

cpview (you already went through that, but you can also export it and review with command cpview -s export)

cpstat (bunch of values there for all the given blades, interfaces, etc)

ethtool command (use -S flag for specific interface...ie ethtool -S eth0)

top

free -m

fwaccel stats

Re: BW Saturation

PhoneBoy — Thu, 30 Nov 2023 15:34:27 GMT

Can you post the output of netstat -ni?
This could be some sort of cabling or flow control issue.

Re: BW Saturation

Matlu — Thu, 30 Nov 2023 16:19:44 GMT

Hello,

I share the result of "netstat -ni".

From this result, what is the "important" value to take into account?

The interface that has the public IP on my GW is eth1-03.

What are the values of the commands I have shared that are "important" to consider?

Cheers.

Re: BW Saturation

PhoneBoy — Thu, 30 Nov 2023 16:30:11 GMT

The fact you have a non-zero RX (receive) errors on the ISP interface suggest the issue is upstream of the firewall.
The fact you're got a lot of CRC errors suggest a cabling issue of some sort.
Receive errors result in retransmissions, which will definitely impact overall performance.

Re: BW Saturation

the_rock — Thu, 30 Nov 2023 16:35:07 GMT

CRC errors would 100% indicate some sort of cabling problem. Check out below.

Andy

https://www.cisco.com/c/en/us/support/docs/ios-nx-os-software/nx-os-software/217554-understand-cyclic-redundancy-check-crc.html#:~:text=to%20Host%2DB.-,CRC%20Error%20Definition,the%20device%20for%20the%20frame.

Re: BW Saturation

Timothy_Hall — Thu, 30 Nov 2023 16:44:28 GMT

eth1-03 does have a few CRC errors (usually a cabling problem but the number is really low) but also has a crapload of RX-OVR indicating an overrun of inbound frames into the NIC card itself resulting in packet loss. You need to use an interface with a faster line speed there, or create an Active-Active bond of multiple interfaces. Just be sure to set the Transmit Hash Policy to L3+4 on both sides of the bond to help ensure roughly equal distribution of traffic between the physical interfaces of the bond.

The easiest way to see the bandwidth hogs is take a look at the elephant/heavy flows the firewall detected in the last 24 hours with the fw ctl multik print_heavy_conn command. top_conns will also give you the live list of top connections consuming resources through the firewall.

Re: BW Saturation

the_rock — Thu, 30 Nov 2023 18:41:37 GMT

Hey bro, any progress today on this? Things any better?

Cheers,

Andy

Re: BW Saturation

Matlu — Thu, 30 Nov 2023 19:41:16 GMT

Buddy,

At the moment, the client is reviewing with his ISP the detected problem, since they are observing that his contracted BW is being saturated intermittently.

For the moment, from Check Point's side, we are only "monitoring" that we do not observe anything unusual.

I understand that the most relevant value of the "netstat -ni" command that helps me to detect a significant error is the RX-OVR, right?

Greetings.

Re: BW Saturation

the_rock — Thu, 30 Nov 2023 19:44:31 GMT

Thats right. Just follow what @Timothy_Hall said, he knows this probably more than anyone out there.

Best regards,

Andy

Re: BW Saturation

D_Schoenberger — Thu, 30 Nov 2023 20:22:36 GMT

In addition to what the others have mentioned, if you can collect a packet capture during the time of the bandwidth saturation, it is possible to analyze that capture using our CPMonitor tool to tell you what the top source/destination/services are.

Have a look at sk103212 for guidance on how to use the tool. If taking the capture on the external/ISP-facing interface of the firewall, you may want to also take (and analyze) captures on the internal interfaces as well to get a better idea of where the load is coming from.

Re: BW Saturation

the_rock — Thu, 30 Nov 2023 20:27:20 GMT

Damon, nice to see you here mate : - )

I know its you, since I recognize your car, its same photo you had on every time we would do zoom meetings. Good old https inspection issue, haha.

Hope you are doing well. Good to know about that tool, dont believe I ever used it before.

Cheers,

Andy

Re: BW Saturation

D_Schoenberger — Thu, 30 Nov 2023 20:40:38 GMT

Hey Andy!

I figured using my Zoom profile pic would be the quickest way to be recognized here 🙂

CPMonitor comes in handy when trying to track down floods of traffic, always good to keep in your back pocket.

Re: BW Saturation

the_rock — Thu, 30 Nov 2023 20:56:14 GMT

Good to know! I always ask TAC people when Im on the phone if they know you, because you are by far the most patient person I ever talked to : - )

Cheers,

Andy

Re: BW Saturation

Matlu — Thu, 30 Nov 2023 21:00:14 GMT

Hello,

I have tried the command "fw ctl multik print_heavy_conn" on my GWs, but I have no result.

[Expert@GW01:0]# fw ctl multik print_heavy_conn
[Expert@GW01:0]#
[Expert@GW01:0]#

Do I have to install something in particular?

Greetings.

Re: BW Saturation

Timothy_Hall — Fri, 01 Dec 2023 13:04:16 GMT

No that just means no elephant flows were detected, try top_conns.

Re: BW Saturation

D_Schoenberger — Thu, 30 Nov 2023 21:06:14 GMT

if you have no entries in the heavy_conn_table kernel table (fw tab -t heavy_conn_table -s), no output is expected from this command.

Re: BW Saturation

Matlu — Thu, 30 Nov 2023 21:30:56 GMT

Hello,

I got this result with the command you have shared.
I understand, that not having any result, I can't have a result either, in the command that Timothy shared, right?

[Expert@GW01:0]# fw tab -t heavy_conn_table -s
HOST NAME ID #VALS #PEAK #SLINKS
localhost heavy_conn_table 16 0 0 0 0 0

Cheers.