cancel
Showing results for 
Search instead for 
Did you mean: 
Create a Post
Highlighted

R80.x Performance Tuning Tip - Elephant Flows (Heavy Connections)

Jump to solution
Elephant Flow (Heavy Connections)

In computer networking, an elephant flow (heavy connection) is an extremely large in total bytes continuous flow set up by a TCP or other protocol flow measured over a network link. Elephant flows, though not numerous, can occupy a disproportionate share of the total bandwidth over a period of time.  When the observations were made that a small number of flows carry the majority of Internet traffic and the remainder consists of a large number of flows that carry very little Internet traffic (mice flows).

All packets associated with that elephant flow must be handled by the same firewall worker core (CoreXL instance). Packets could be dropped by Firewall when CPU cores, on which Firewall runs, are fully utilized. Such packet loss might occur regardless of the connection's type.

What typically produces heavy connections:

  • System backups
  • Database backups
  • VMWare sync.
Chapter

More interesting articles:

- R80.x Architecture and Performance Tuning - Link Collection
- Article list (Heiko Ankenbrand)

Evaluation of heavy connections


The big question is, how do you found elephat flows on an R80 gateway?

Tip 1
Evaluation of heavy connections (epehant flows)

A first indication is a high CPU load on a core if all other cores have a normal CPU load. This can be displayed very nicely with "top". Ok, now a core has 100% CPU usage. What can we do now? For this there is a SK105762 to activate "Firewall Priority Queues".  This feature allows the administrator to monitor the heavy connections that consume the most CPU resources without interrupting the normal operation of the Firewall. After enabling this feature, the relevant information is available in CPView Utility. The system saves heavy connection data for the last 24 hours and CPDiag has a matching collector which uploads this data for diagnosis purposes.

Heavy connection flow system definition on Check Point gateways:

  • Specific instance CPU is over 60%
  • Suspected connection lasts more than 10s
  • Suspected connection utilizes more than 50% of the total work the instance does. In other words, connection CPU utilization must be > 30%  
CLI Commands


Tip 2

Enable the monitoring of heavy connections.

To enable the monitoring of heavy connections that consume high CPU resources:

# fw ctl multik prioq 1

# reboot

Tip 3
Found heavy connection on the gateway with „print_heavy connections“

On the system itself, heavy connection data is accessible using the command:

# fw ctl multik print_heavy_conn
pq5.jpg

Tip 4
Found heavy connection on the gateway with cpview

# cpview                CPU > Top-Connection > InstancesX
pq3.png

 

Links


sk105762 - Firewall Priority Queues in R77.30 / R80.10 and above

 

 
1 Solution

Accepted Solutions

Re: R80.x Performance Tuning Tip - Elephant Flows (Heavy Connections)

Jump to solution

In the past years I had always been looking for a solution to find elephant flows. Check Point has built in a good solution.

View solution in original post

Tags (1)
13 Replies

Re: R80.x Performance Tuning Tip - Elephant Flows (Heavy Connections)

Jump to solution

Hi @HeikoAnkenbrand,

Thank you for all the interesting articles about Performance Tuning you wrote.

You could write a book out of this link collection 😀.

R80.x Architecture and Performance Tuning - Link Collection 

 

 

Re: R80.x Performance Tuning Tip - Elephant Flows (Heavy Connections)

Jump to solution

Hi @HeikoAnkenbrand,

This article has helped me very well.

I followed the steps and actually found a database backup connection. The connection caused about 70% CPU load on one core. We have now limited the bandwidth of the connection via QoS.

Best Regards

Paul

 

Re: R80.x Performance Tuning Tip - Elephant Flows (Heavy Connections)

Jump to solution

We were able to identify a very similar problem.

thx

 

0 Kudos

Re: R80.x Performance Tuning Tip - Elephant Flows (Heavy Connections)

Jump to solution

We also had the problem with the elephant flows. This is a good way to find them quickly and easily.

Re: R80.x Performance Tuning Tip - Elephant Flows (Heavy Connections)

Jump to solution

In the past years I had always been looking for a solution to find elephant flows. Check Point has built in a good solution.

View solution in original post

Tags (1)
Gaurav_B_
Ivory

Re: R80.x Performance Tuning Tip - Elephant Flows (Heavy Connections)

Jump to solution

I just tried that. This is a very interesting solution. A way to find elefant flows.

Thanks

Re: R80.x Performance Tuning Tip - Elephant Flows (Heavy Connections)

Jump to solution

👌

0 Kudos

Re: R80.x Performance Tuning Tip - Elephant Flows (Heavy Connections)

Jump to solution

We have several connections with 5-7% utilization.

What can we do here?

0 Kudos

Re: R80.x Performance Tuning Tip - Elephant Flows (Heavy Connections)

Jump to solution

So glad you asked this question.  🙂

I will be speaking at CPX New Orleans and Vienna on the CheckMates track with a presentation called "Big Game Hunting: Elephant Flows" that will go through how to track down elephant flows (a.k.a. heavy connections), all the different remediation options, and the pros and cons of each.  PhoneBoy will be delivering this presentation for me at CPX Bangkok because I'll be very busy that week, with, uh, something else...

 

Book "Max Power 2020: Check Point Firewall Performance Optimization" Third Edition
Now Available at www.maxpowerfirewalls.com

Re: R80.x Performance Tuning Tip - Elephant Flows (Heavy Connections)

Jump to solution

This is an interesting approach to detect heavy connections. I had checked this after this article and could identify some systems that were causing problems. We have now created QoS rules to limit the bandwidth. That worked well.

 

0 Kudos

Re: R80.x Performance Tuning Tip - Elephant Flows (Heavy Connections)

Jump to solution

Guys,

if you have a problem with elephant flow you may try this

SecureXL Fast Accelerator (fw fast_accel) for R80.20 and above - sk156672

0 Kudos

Re: R80.x Performance Tuning Tip - Elephant Flows (Heavy Connections)

Jump to solution
Do we have to enable PrioQ to support the "fw ctl multik print_heavy_conn" command? The article suggests it, but the Tip# list isn't execution step#.

Also is this supported on R77.30 and R76SP.50?
CCMA
0 Kudos

Re: R80.x Performance Tuning Tip - Elephant Flows (Heavy Connections)

Jump to solution

Priority Queues must be in mode 1 (Eviluator-only) to use that command; mode 1 is the default on a firewall that does not have USFW enabled. I'll be speaking about this very topic in detail at CPX New Orleans and Vienna.

Support for fw ctl multik print_heavy_conn was added in R80.20; I doubt it can be backported into earlier releases since I'm pretty sure it relies on the major changes introduced to SecureXL in R80.20.

 

Book "Max Power 2020: Check Point Firewall Performance Optimization" Third Edition
Now Available at www.maxpowerfirewalls.com
0 Kudos