topic Re: CPU intensive connections "TCP:empowerid" in Firewall and Security Management

CPU intensive connections "TCP:empowerid"

piou_piou — Fri, 24 Apr 2020 13:12:48 GMT

Hello everyone,

Due to the current situation, we all know lots of users are working remotely.

I am having a weird problem on my R80.30 cluster (5400 appliances) :

The CPU is dangerously increasing during daytime (it reached 100% today). I noticed via cpview that most of the CPU is consumed by a few connections :

These connections are always using TCP/7080 port and are displayed as "TCP:empowerid".

This is always from a 172.16.50.x host (which is our remote access VPN users pool) to 10.75.30.248 which is one of our Cisco Jabber server. Users and IP can be different, it seems to be happening randomly in the remote access VPN pool.

I managed to lower the CPU by manually disconnecting the involved users as you can see on the following graph at 14h and 16h (CPU is in yellow) :

Then as soon as someone is starting to work in the working, CPU is increasing like crazy... and eventually reaches 100%.

Same problem, same workaround for now, and here is the last graph I have :

I added a rule to drop the TCP/7080 service for now it's working properly, but we may need to accept this service later in order to make Jaber calls work when working remotely (it doesn't work for now and I've no clue why but its another topic).

Here's my GW's version :

Does someone has already seen this before? 😞

Thanks for your help!

piou_piou

Re: CPU intensive connections "TCP:empowerid"

Timothy_Hall — Sun, 26 Apr 2020 17:56:06 GMT

Yes, what you are seeing is classic elephant flow behavior; Check Point calls these heavy connections. Please see my CPX 2020 presentation titled "Big Game Hunting: Elephant Flows" based on a chapter of my book which goes through the available identification tools and possible remediation options:

https://community.checkpoint.com/fyrhh23835/attachments/fyrhh23835/member-exclusives/432/3/CPX_Big_Game_Hunting_FINAL2.cleaned.pdf

Edit: It is also possible these particular connections have to be processed F2F/slowpath for some reason, to see which path these connections are in run fwaccel conns and search for the connection and port attributes.

Re: CPU intensive connections "TCP:empowerid"

piou_piou — Sun, 26 Apr 2020 16:25:29 GMT

Hello Timothy,

Thanks a lot, very useful resource!

I'll try some of these remediation options in the next days.

piou_piou