Directory /fwtmp usage on centrally managed SMB

[HristoGrigorov]

This is probably question for R&D but anyone is welcome to comment...

There is the directory /fwtmp/opt/fw1/state/<CLUSTER OBJECT>/FW1 that is normally empty after reboot. However on first policy install there will be around 10MB files dumped here for no apparent reason. This will leave /fwtmp space with the dangerously low 5MB disk space. After reboot those files will be removed. That makes me think they are not really needed. I am worried that /fwtmp directory will eventually run out of disk space if appliance is not rebooted for a long time. 

My question is why are these files left there and is it OK to remove them?

[G_W_Albrecht]

The /fwtmp directory has a long history:

• In February 2014, we received sk98606 - Policy installation / fetch fails on Centrally Managed 1100 appliance due to insufficient .... Here we learn that 

The /fwtmp partition on an 1100 appliance is set to a size of 40MB, which is not enough for the operation to complete successfully in this specific scenario.

The partition was not enlarged, and there should not be a need to enlarge it. When IPS is used, policy should be adjusted according to sk105217.

If you choose not to upgrade the firmware, then as a workaround on versions lower than R75.20 HFA 70 / R77.20, you can increase the size of the /fwtmp partition to 250MB.

• In July 2016, there was sk112166: "fwtmp" folder is filling up, causing crashes on SMB appliances.

The fwtmp folder is filling up of its capacity, causing crash SMB appliances.

The fwtmp folder capacity is 40MB. The cpeps folder consuming 37MB.

If you wish not to upgrade, the following Workaround is available:

This will clean the existing cpeps tables.

• In June 2017, we find sk117912: Traffic counter cannot be reset back to "zero" on locally managed SMB appliance

After using the traffic counter for local machines, it cannot be reset back to "zero" value

This feature is not included in the product. 

As a workaround :

1. Clean /tmp and /fwtmp folders on the appliance.
2. Reboot the device.

• And in Mai 2018, we finally received sk126372: Policy installation on SMB appliances fails with "Load on Module failed - not enough disk ... !

The /pfrm2.0 partition is full, or getting full, by the temporary files of the policy that are being saved under $FWDIR/state/__tmp/FW1.

If your device has R77.20.80 installed, skip the Hotfix instructions and proceed to the solution below.

The feature added in the new image creates a soft link from $FWDIR/state/__tmp/FW1 to the storage partition.

After installing the Hotfix, manually enable the new feature:

1. In the appliance's WebUI, go to Device -> Advanced -> Advanced Settings. 
2. Set the value of "Move temporary policy files to storage" to 'true':

[HristoGrigorov]

Thanx for the detailed reply. Unfortunately the path I am talking about is different than the one mentioned in the SK. I can symlink it to /storage but before that I want R&D to confirm it is OK to do it. Hopefully they are monitoring this thread...

[G_W_Albrecht]

I would not think that R&D is monitoring these posts - i would rather involve TAC  if the issue is important to me... As long as there is no apparent issue caused by less space in /fwtmp than needed for operation i would not care at all.

Regarding the path: We all talk about partition /fwtmp that contains directories and symlinks. Some directories have developed to symlinks because of the need for more storage, e.g. /__tmp. In sk98606 we learn how to enlarge this partition, but: By design, 1100 appliance has shared volatile memory between mounted file system and RAM. Care should be taken when space is allocated to the file system, as available RAM is reduced accordingly.

[HristoGrigorov]

There is the third option.... I will symlink it and see what happens. But not Friday afternoon; next week 🙂

[HristoGrigorov]

Well, I double and triple checked it and although I have Device -> Advanced Settings -> Additional Management Settings - Move temporary policy files to storage set to true this is the result:

# ls -l /opt/fw1/state/__tmp
lrwxrwxrwx 1 root root 22 Feb 28 12:47 /opt/fw1/state/__tmp -> /flash/fw1/state/__tmp

# ls -l /opt/fw1/state/local
lrwxrwxrwx 1 root root 22 Feb 28 12:47 /opt/fw1/state/local -> /flash/fw1/state/local

Surprisingly this is not the /storage partition but /pfrm2.0 that is less in space. Go figure.... 🤔 

[PhoneBoy]

What these files are is the compiled policy for the device.
As these can be fetched from the management, it's safe to remove these once the policy has been loaded into memory (in theory).

On a non-SMB appliance the equivalent directory is persistent, allowing the device to load the policy from disk if for some reason the management is unavailable.
If there is nothing in the state directory AND the management server is unavailable, you end up in DefaultFilter territory on non-SMB gateways.
Not sure how this is handled on SMB appliances.

[HristoGrigorov]

On SMB directory /fwtmp/opt/fw1/state/<CLUSTER OBJECT>/FW1 is not existent right after reboot. It is created empty shortly after that and populated with files for first time after policy install.

The last installed policy is persistent on SMB as well. On boot it will compare local and management server policy versions and if they match it will be loaded from local storage. 

[HristoGrigorov]

Tom, if possible can you please run this command on that problematic appliance and paste output:

# du -sh /fwtmp/opt/fw1/cpeps/

[Tom_Hinoue]

Sure.

----------------------------------
[Expert@hostname]# du -sh /fwtmp/opt/fw1/cpeps/
34.4M /fwtmp/opt/fw1/cpeps/

It looks like the customer rebooted the gateway yesterday, but /fwtmp still is 100%.
Seems the cpeps directory is not cleared after reboot..? 🤔

[HristoGrigorov]

You can do it manually but there is downtime:

# cd /fwtmp/opt/fw1/cpeps/

# cpstop

# <copy files from cpeps somewhere, e.g. /storage/cpeps>

# rm -f *

# cpstart

[Tom_Hinoue]

Thanks 🙂
Will consider to do it if the investigation will take more time. (or simply just failover to standby)

but I wonder why this is happening in a cluster..
Hope we get some workarounds or tweaks in behavior regarding cpeps dir.

[HristoGrigorov]

It's a common problem. There is even SK about that but I can't find it at the moment. 

[HristoGrigorov]

While talking about temp directories is anyone able to explain why are 500MB RAM space reserved for appliance that has TE blade disabled?

tmpfs on /tetmp type tmpfs (rw,relatime,size=512000k)

cpInit: mkdir /tetmp
cpInit: mount tmpfs /tetmp -t tmpfs -o size=${te_size_mb}m
cpInit: ln -s /tetmp /opt/fw1/tmp/dlp
cpInit: ln -s /tetmp /opt/fw1/tmp/te

[HristoGrigorov]

Here is how to increase /fwtmp size...

Because  this is filesystem mounted in memory, if you can spare a bit of it you can allocate it to /fwtmp folder. For example, to increase it from 40MB to say 50MB, login and issue following command:

# fw_setenv fwtmp_dir_size 50

Reboot and enjoy...

# df -h /fwtmp
Filesystem Size Used Available Use% Mounted on
tmpfs 50.0M 25.7M 24.3M 51% /fwtmp

[Tom_Hinoue]

Actually, I have a ongoing case with TAC right now regarding a similar case.
A locally managed cluster with active member /fwtmp directory filled up 100% in approx 2 weeks after reboot.
Standby member /fwtmp seems average.

Current issue our customer is experiencing is that login to WEBUI fails even with correct credentials for a local administrator account, while SSH works fine. Also, noticed [cpwd_admin list] command does not work.. doesn't give us any output.

Still in investigation about the behavior. 

790 Appliance
R77.20.87 B960

[Jeroen_Demets]

Tom, this is a thread on centrally managed SMB and you have the issue on a 790 which cannot be centrally managed?

It's interesting to know it doesn't seem related to being centrally managed or not.

[motadata13]

i am facing the same issue right now do you found any solution for that  ??

i am using 1500 appliances on locally managed.

[Tom_Hinoue]

AFAIK after engaging with RnD back then, there were some issues regarding some files not automatically deleted from fwtmp directory that may caused to fill up ending up causing the device to become unstable. I believe this goes for not only locally managed but also centrally managed. 

Also, to adapt to the current customer traffic usage, the fwtmp directory was increased to 60MB which I heard is considered to be enough to not cause the directory filling up.

The small fixes for the issue and fwtmp size tweak should be included since R77.20.87 Build 990173072 for 700/1400 or R80.20.15 Build 992001451 for 1500s. Are you already on the latest build (R80.20.25 Build 992002077)? You may want to start there.