Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
Highlighted
Platinum

Directory /fwtmp usage on centrally managed SMB

This is probably question for R&D but anyone is welcome to comment...

There is the directory /fwtmp/opt/fw1/state/<CLUSTER OBJECT>/FW1 that is normally empty after reboot. However on first policy install there will be around 10MB files dumped here for no apparent reason. This will leave /fwtmp space with the dangerously low 5MB disk space. After reboot those files will be removed. That makes me think they are not really needed. I am worried that /fwtmp directory will eventually run out of disk space if appliance is not rebooted for a long time. 

My question is why are these files left there and is it OK to remove them?

 

0 Kudos
16 Replies
Highlighted
Sapphire

The /fwtmp directory has a long history:

The /fwtmp partition on an 1100 appliance is set to a size of 40MB, which is not enough for the operation to complete successfully in this specific scenario.

The partition was not enlarged, and there should not be a need to enlarge it. When IPS is used, policy should be adjusted according to sk105217.

If you choose not to upgrade the firmware, then as a workaround on versions lower than R75.20 HFA 70 / R77.20, you can increase the size of the /fwtmp partition to 250MB.

 

The fwtmp folder is filling up of its capacity, causing crash SMB appliances.

The fwtmp folder capacity is 40MB. The cpeps folder consuming 37MB.

If you wish not to upgrade, the following Workaround is available:

This will clean the existing cpeps tables.

After using the traffic counter for local machines, it cannot be reset back to "zero" value

This feature is not included in the product. 

As a workaround :

  1. Clean /tmp and /fwtmp folders on the appliance.
  2. Reboot the device.

The /pfrm2.0 partition is full, or getting full, by the temporary files of the policy that are being saved under $FWDIR/state/__tmp/FW1.

If your device has R77.20.80 installed, skip the Hotfix instructions and proceed to the solution below.

The feature added in the new image creates a soft link from $FWDIR/state/__tmp/FW1 to the storage partition.

After installing the Hotfix, manually enable the new feature:

  1. In the appliance's WebUI, go to Device -> Advanced -> Advanced Settings
  2. Set the value of "Move temporary policy files to storage" to 'true':

 

Highlighted
Platinum

Thanx for the detailed reply. Unfortunately the path I am talking about is different than the one mentioned in the SK. I can symlink it to /storage but before that I want R&D to confirm it is OK to do it. Hopefully they are monitoring this thread...

0 Kudos
Highlighted
Sapphire

I would not think that R&D is monitoring these posts - i would rather involve TAC  if the issue is important to me... As long as there is no apparent issue caused by less space in /fwtmp than needed for operation i would not care at all.

Regarding the path: We all talk about partition /fwtmp that contains directories and symlinks. Some directories have developed to symlinks because of the need for more storage, e.g. /__tmp. In sk98606 we learn how to enlarge this partition, but: By design, 1100 appliance has shared volatile memory between mounted file system and RAM. Care should be taken when space is allocated to the file system, as available RAM is reduced accordingly.

0 Kudos
Highlighted
Platinum

There is the third option.... I will symlink it and see what happens. But not Friday afternoon; next week 🙂

0 Kudos
Highlighted
Platinum

Well, I double and triple checked it and although I have Device -> Advanced Settings -> Additional Management Settings - Move temporary policy files to storage set to true this is the result:

# ls -l /opt/fw1/state/__tmp
lrwxrwxrwx 1 root root 22 Feb 28 12:47 /opt/fw1/state/__tmp -> /flash/fw1/state/__tmp

# ls -l /opt/fw1/state/local
lrwxrwxrwx 1 root root 22 Feb 28 12:47 /opt/fw1/state/local -> /flash/fw1/state/local

Surprisingly this is not the /storage partition but /pfrm2.0 that is less in space. Go figure.... 🤔 

0 Kudos
Highlighted
Admin
Admin

What these files are is the compiled policy for the device.
As these can be fetched from the management, it's safe to remove these once the policy has been loaded into memory (in theory).

On a non-SMB appliance the equivalent directory is persistent, allowing the device to load the policy from disk if for some reason the management is unavailable.
If there is nothing in the state directory AND the management server is unavailable, you end up in DefaultFilter territory on non-SMB gateways.
Not sure how this is handled on SMB appliances.
0 Kudos
Highlighted
Platinum

On SMB directory /fwtmp/opt/fw1/state/<CLUSTER OBJECT>/FW1 is not existent right after reboot. It is created empty shortly after that and populated with files for first time after policy install.

The last installed policy is persistent on SMB as well. On boot it will compare local and management server policy versions and if they match it will be loaded from local storage. 

0 Kudos
Highlighted
Platinum

Tom, if possible can you please run this command on that problematic appliance and paste output:

 

# du -sh /fwtmp/opt/fw1/cpeps/

0 Kudos
Highlighted
Copper

Sure.

----------------------------------
[Expert@hostname]# du -sh /fwtmp/opt/fw1/cpeps/
34.4M /fwtmp/opt/fw1/cpeps/

It looks like the customer rebooted the gateway yesterday, but /fwtmp still is 100%.
Seems the cpeps directory is not cleared after reboot..? 🤔

0 Kudos
Highlighted
Platinum

You can do it manually but there is downtime:

 

# cd /fwtmp/opt/fw1/cpeps/

# cpstop

# <copy files from cpeps somewhere, e.g. /storage/cpeps>

# rm -f *

# cpstart

0 Kudos
Copper

Thanks 🙂
Will consider to do it if the investigation will take more time. (or simply just failover to standby)

but I wonder why this is happening in a cluster..
Hope we get some workarounds or tweaks in behavior regarding cpeps dir.

0 Kudos
Highlighted
Platinum

It's a common problem. There is even SK about that but I can't find it at the moment. 

0 Kudos
Highlighted
Platinum

While talking about temp directories is anyone able to explain why are 500MB RAM space reserved for appliance that has TE blade disabled?

tmpfs on /tetmp type tmpfs (rw,relatime,size=512000k)

cpInit: mkdir /tetmp
cpInit: mount tmpfs /tetmp -t tmpfs -o size=${te_size_mb}m
cpInit: ln -s /tetmp /opt/fw1/tmp/dlp
cpInit: ln -s /tetmp /opt/fw1/tmp/te

0 Kudos
Highlighted
Platinum

Here is how to increase /fwtmp size...

 

Because  this is filesystem mounted in memory, if you can spare a bit of it you can allocate it to /fwtmp folder. For example, to increase it from 40MB to say 50MB, login and issue following command:

 

# fw_setenv fwtmp_dir_size 50

 

Reboot and enjoy...

 

# df -h /fwtmp
Filesystem Size Used Available Use% Mounted on
tmpfs 50.0M 25.7M 24.3M 51% /fwtmp

 

 

0 Kudos
Highlighted
Copper

Actually, I have a ongoing case with TAC right now regarding a similar case.
A locally managed cluster with active member /fwtmp directory filled up 100% in approx 2 weeks after reboot.
Standby member /fwtmp seems average.

Current issue our customer is experiencing is that login to WEBUI fails even with correct credentials for a local administrator account, while SSH works fine. Also, noticed [cpwd_admin list] command does not work.. doesn't give us any output.

Still in investigation about the behavior. 

790 Appliance
R77.20.87 B960

0 Kudos
Highlighted

Tom, this is a thread on centrally managed SMB and you have the issue on a 790 which cannot be centrally managed?

It's interesting to know it doesn't seem related to being centrally managed or not.

0 Kudos