topic Re: Concurrent Connections - R80.40 in Firewall and Security Management

Concurrent Connections - R80.40

j_silva — Mon, 06 Dec 2021 11:55:50 GMT

Hi,

I have an appliance 15600 with Gaia R80.40 JHF 126. Everyday when the connections is above 80000 all traffic to internet stops unexpectedly. I don´t understand because the fw ctl pstat output show me that the connection doesn´t have limit.

System Capacity Summary:
Memory used: 16% (7758 MB out of 47763 MB) - below watermark
Concurrent Connections: 28844 (Unlimited)
Aggressive Aging is enabled, not active

I don´t have issues related with CPU, memory or disk and no nat fragmentation as well. I saw on Smartconsole that is possible to change the connection value limit but i believe isn´t necessary to change it.

Could you help with this issue?

Re: Concurrent Connections - R80.40

_Val_ — Mon, 06 Dec 2021 11:57:50 GMT

Check if you do not exhaust your NAT Hide ports yet.

Re: Concurrent Connections - R80.40

j_silva — Mon, 06 Dec 2021 12:02:14 GMT

Hello

I don't have this problem, I've searched Smartlog and cpview. All related questions are apparently normal.

Re: Concurrent Connections - R80.40

Glen_Bayless — Mon, 06 Dec 2021 13:34:40 GMT

If you are able, a ticket with TAC will resolve the issue the quickest.

However some steps you can follow during the issue is to run the following in expert mode "fw ctl zdebug + drop" for a few seconds and see what error messages you get. TAC would request this information as well.

If you post it here someone will review but it needs to be during the issue.

Re: Concurrent Connections - R80.40

_Val_ — Mon, 06 Dec 2021 13:46:35 GMT

What did you search it for, exaclty? Also, when issue happens, look at your NAT kernel tables to see if they are not maxed.

Re: Concurrent Connections - R80.40

the_rock — Mon, 06 Dec 2021 14:45:02 GMT

@_Val_ actually brought up a very good point. I recall there was an sk from while back about this, probably R77 and before and it involved modifying some stuff on mgmt server file. I will see if I can find it.

Re: Concurrent Connections - R80.40

j_silva — Mon, 06 Dec 2021 14:53:11 GMT

Hi,

During the problem a NAT exhaustion is checked in cpview according to the image attached.

Is possible to check the NAT use with another command?

Re: Concurrent Connections - R80.40

the_rock — Mon, 06 Dec 2021 14:55:00 GMT

https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk32224

Re: Concurrent Connections - R80.40

j_silva — Mon, 06 Dec 2021 14:56:08 GMT

Hi Glen,

I have ran "fw ctl zdebug + drop" but i didn´t get messages about drop...

So, i already open a case with TAC but no issues about NAT was asked me until this moment.

Re: Concurrent Connections - R80.40

the_rock — Mon, 06 Dec 2021 14:57:49 GMT

This is one of sk's I was thinking of...check bottom for R80.x mgmt server (or R81 if thats what you use for mgmt)

https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk36708&partition=Advanced&product=ClusterXL

Re: Concurrent Connections - R80.40

j_silva — Mon, 06 Dec 2021 15:00:22 GMT

Hi,

I have found a SK that inform to modify the limit of connection -> /opt/CPsuite-R80.40/fw1/conf/objects_5_0.C

# :connections_limit (25000)

But when i run fw ctl pstat i have unlimited as below:

System Capacity Summary:
Memory used: 17% (8250 MB out of 47763 MB) - below watermark
Concurrent Connections: 50372 (Unlimited)
Aggressive Aging is enabled, not active

Hash kernel memory (hmem) statistics:
Total memory allocated: 8513806336 bytes in 2078566 (4096 bytes) blocks using 2 pools
Initial memory allocated: 5007998976 bytes (Hash memory extended by 3505807360 bytes)
Memory allocation limit: 40066088960 bytes using 512 pools
Total memory bytes used: 0 unused: 8513806336 (100.00%) peak: 6942913848
Total memory blocks used: 0 unused: 2078566 (100%) peak: 1812703
Allocations: 373539907 alloc, 0 failed alloc, 346496895 free

System kernel memory (smem) statistics:
Total memory bytes used: 11712721952 peak: 13962568616
Total memory bytes wasted: 123912475
Blocking memory bytes used: 119820396 peak: 815182716
Non-Blocking memory bytes used: 11592901556 peak: 13147385900
Allocations: 2250859302 alloc, 207 failed alloc, 2250776409 free, 0 failed free
vmalloc bytes used: 11447248008 expensive: no

Kernel memory (kmem) statistics:
Total memory bytes used: 6601477344 peak: 10154447436
Allocations: 2624246410 alloc, 207 failed alloc
2597134592 free, 0 failed free
External Allocations:
Packets:124969024, SXL:37052266, Reorder:0
Zeco:0, SHMEM:2176, Resctrl:0
ADPDRV:0, PPK_CI:13015296, PPK_CORR:0

Cookies:
1870268372 total, 809226293 alloc, 809155399 free,
3679877010 dup, 4020703328 get, 2165983218 put,
2327063251 len, 740443353 cached len, 26932 chain alloc,
26932 chain free

Connections:
540014320 total, 466588826 TCP, 71211150 UDP, 2203472 ICMP,
10872 other, 406247 anticipated, 1883050 recovered, 50372 concurrent,
124545 peak concurrent

Fragments:
92073 fragments, 44095 packets, 69 expired, 0 short,
0 large, 0 duplicates, 0 failures

NAT:
1779716794/0 forw, -431240825/0 bckw, 1135361697 tcpudp,
17091927 icmp, 1548710183-419225040 alloc

Re: Concurrent Connections - R80.40

j_silva — Mon, 06 Dec 2021 15:12:34 GMT

Hi the_rock

I've seen this sk, but my question is whether it's even necessary to adjust this parameter since the gateway has no connection limit according to fw ctl pstat output.
I have seen gateway with low capacity with no problem related nat or limits as well.

It´s very stranger to me.

Re: Concurrent Connections - R80.40

_Val_ — Mon, 06 Dec 2021 15:17:33 GMT

Limit 10000 but actual peak is 28000? Does not sound okay to me. Also, you are too close to that 28000, which might point in the right direction:

Re: Concurrent Connections - R80.40

j_silva — Mon, 06 Dec 2021 15:25:30 GMT

Hi,

I already asked it to TAC. According TAC i have 28 cpu core, so each core has a limit 10000, so i need to calculate 28x10000 = 280000.

I my opinion, this case i just loose the cache but the NAT still going to translate address normally.

Re: Concurrent Connections - R80.40

the_rock — Mon, 06 Dec 2021 15:25:49 GMT

@_Val_ is right, I just noticed the same thing actually. How come limit says 10000, but peak is 280000??!! That makes no sense...if you edit fw object in dashboard and go to optimization from left side menu, what do you have set there for capacity optimization?

Andy

Re: Concurrent Connections - R80.40

the_rock — Mon, 06 Dec 2021 15:27:06 GMT

First time ever I hear that sort of calculation...not really sure thats correct, maybe someone else can confirm.

Re: Concurrent Connections - R80.40

j_silva — Mon, 06 Dec 2021 15:30:11 GMT

Yes the_rock

TAC explained that to me during a chat. I´ll ask again.

Thanks for all feedback.

Re: Concurrent Connections - R80.40

the_rock — Mon, 06 Dec 2021 15:45:40 GMT

I will let others confirm, but I NEVER heard that before. I wish they gave you some document about it...not sure where that came from, honestly.

Re: Concurrent Connections - R80.40

genisis__ — Mon, 06 Dec 2021 21:14:34 GMT

I'd like to see the SK or document where it explains that.

Re: Concurrent Connections - R80.40

j_silva — Mon, 06 Dec 2021 21:20:44 GMT

Hi,

I just have a response during exchange message on the case.

Hi Juarez,

" Appreciating your update, You can clear the table and check if that will resolve as a first step. The limit 10000 is per firewall worker so I am thinking there are 28 workers, each worked can have up to 10K entries in that table, so 28 X 10000 = 280000 so that is why you see it at 280K. When we increase the limit it will be by firewall worker."

*** Your service request is not monitored outside of assigned engineers working hours, should you require immediate assistance, please feel free to call in. ***

Regards