1470 Capacity Optimization

[HristoGrigorov]

Hi,

I need your advice on a 1470 capacity optimization. This is how it is setup at the moment:

I was following sk39555 and I ran 'fw ctl pstat' to check what's the current status. There it is:

System Capacity Summary:
Memory used: 17% (184 MB out of 1051 MB) - below watermark
Concurrent Connections: 1% (1632 out of 149900) - below watermark
Aggressive Aging is not active

Hash kernel memory (hmem) statistics:
Total memory allocated: 192937984 bytes in 47104 (4096 bytes) blocks using 21 pools
Initial memory allocated: 109051904 bytes (Hash memory extended by 83886080 bytes)
Memory allocation limit: 408944640 bytes using 512 pools
Total memory bytes used: 76512184 unused: 116425800 (60.34%) peak: 178943980
Total memory blocks used: 21151 unused: 25953 (55%) peak: 44215
Allocations: 84585497 alloc, 0 failed alloc, 83774180 free

The SK talks that the Total memory allocated should be roughly the same as Memory pool size in the Optimization property. Unless I calculate something wrong, at the moment it is ~192 MB which is much more than the 20 MB set as initial memory pool and the max mem pool size. In fact, it says it extended it by ~83MB which is kind of not recommended and may cause memory fragmentation.

So, what I am not getting right here ? Anything I should change/adjust ?

[PhoneBoy]

In the vast majority of cases, you should just be using Automatic (versus manually setting these parameters).

What is the problem you're trying to solve?

[HristoGrigorov]

I did just that the other day. Set Automatic for max concurrent connections. Couldn't set it for hash and memory pool size because option is always disable for some reason.

Then yesterday Management Server reported appliance as not reachable and at the same time all IPSec tunnels went down. Strangely enough everything else seemed to work just fine. I was even able to SSH-in and reboot it. 

I came back to revert it to Manually (that was the default) and started to ask myself if these options are actually optimal because for example default max concurrent IKEs on locally managed 1470 is set to 20 or so and here it was like 100 by default.

And so I started reading about what these options mean and came across the above mentioned SK.

[D_W]

With the 1400 series (we have 3x 1470 and 2x 1490) we had the same issues like appliance unreachable, tunnels will not show up, no SIC but SSH is fine etc.

In $FWDIR/log/sfwd.elg i found lot of these entries:

sfwd_periodic_memory_rss: sfwd RSS (116 MB) is over the defined limit (80 MB). sfwd is exiting!

With this I found SK103501 and increased the limit to 300MB. Since then no issues with these devices!


Would be fine if that helps you too!

[HristoGrigorov]

Well, I went to check sfwd.elg and guess what... same error  Increased that limit to 300 as suggested. Certainly a very valuable info, thanx!

Btw, I noticed a lot of these in the same log file and I wonder if other see them as well ?

fw_kbuf_get_multik(instance: 0): ioctl(FWKBUF): Bad address

[HristoGrigorov]

Yay, talk to me about memory consumption:

 sfwd_periodic_memory_rss: sfwd RSS (300 MB) is over the defined limit (300 MB). sfwd is exiting!
[sfwd 21287 1738121216]@CPFW-1[24 May 22:14:41] sfwd: Thu May 24 22:14:41 2018

[D_W]

SK103501 describes also the following:

It is possible to completely disable this mechanism by running:

[Expert@1100gateway]# fw ctl set int fw_sfwd_max_rss_enforce 0

Maybe that will help you but maybe someone from CP can give more information what will happen when you disable this completely!

If you use IPS it's also recommended to use a "smaller" Profile for this kind of devices to lower the hardware usage.

[G_W_Albrecht]

Concerning the use of an optimized IPS profile for SMB please see my new document Optimizing an IPS profile for SMB.

[HristoGrigorov]

Thanx both of you. I have disabled that max RSS memory check. It does not seem to cause any harm so far. I have also lightened IPS profile as suggested by David and per Gunter's instructions. Let's see if that will cause any difference.

[G_W_Albrecht]

sk39555 may, may partly or may not be valid for SMB devices, so i would involve TAC for 1470 capacity optimization procedure.

[HristoGrigorov]

Defaults are probably fine. Although on SMB any extra byte matters and a doc on the subject would be really nice.

[G_W_Albrecht]

Only hint i have found is from sk112858:

Check for number of connections:

1. 1. fw tab -t connections -s (shows some of the connections and the rest will be shown as a number)

   2. fw tab -t connections (shows only the number of opened connections).

      1. Default value = 25,000.
      2. Maximum value = 10,000,000
      3. To edit this value: refer to 'Device > Advanced settings > Capacity Optimization > Maximum concurrent connections'

[HristoGrigorov]

Well, I did some experiments with this. In locally managed appliance the limit for max concurrent connections (MCC) is 150,000. I tested if lowering MCC will reduce memory usage as claimed in few places on the web. Could not confirm that, allocated memory is nearly the same.

Default hash table size is 131072. That in theory should handle somewhere around 45K average(!) concurrent connections which is high enough for this appliance I think. However, total hmem allocated as reported by 'fw ctl pstat' seem to include not only hash table but something else because as you can see in my case it is ~192MB and it was extended by around ~83MB from initial size. Which is not good because as explained in sk39555 leads to memory fragmentation. 

In SMS, default memory pool size for 1470 (does not specify is it hmem, kmem or something else) is 20 MB and max is 30MB. That somehow do not seem right to me but there is very little info on the net to explain myself these rather low numbers. None of the 'fw ctl pstat' values seems to be anywhere near these numbers.

That automatic mode is somehow strange to me because as explained in sk39555 certain regions of memory can only be increased and not decreased w/o reboot. Meaning probably, you set them low and the system will extend it as needed. But that will again cause memory fragmentation.... may be

[HristoGrigorov]

Anyone can explain the actual meaning of this requirement ? Why should max concurrent connections be no more than 25K ?

[PhoneBoy]

25k was the limit for connections back in the 1990s, and in fact it was the default until R75.40 with Gaia and "Automatic" mode.

Seems weird that, if manual, no more than 25,000 connections is considered "best practice."

Tomer Sole‌, what say you? 

[HristoGrigorov]

I keep wondering why SmartConsole is reporting CPU usage instead of Load Average. If it was mostly CPU-bound system I could understand it. But on a mixed CPU/IO-bound system load average gives much more realistic idea what the current system utilization is. 

Just my 2 cents.