Help with disk space issue - Checkpoint support not helping

[carl_t]

Hi Guys

We are having issues with space on our management server, we keep getting error messages saying "

Partition /opt has: XXXX of free space and it's lower than required: 2000 MB" error in SmartConsole

We have 1.5TB of disk space allocated, more than the recommended minimum.

We went through sk121198 and it didnt solve the issue.

We then logged a ticket to Checkpoint support, they havent been helpful and have closed the ticket twice on us even when its not been resolved, sending a link to an sk article which I can find myself is not very helpful when we pay alot of money for support.

The answer they give is allocate more space, we have done this before and it just fills up again, so what we need to do is find why it fills up and manage this.

Any help would be much appreciated

Many thanks

Carl

[_Val_]

Two things. 

1. Please send me the SR numbers in a private message. I looked up your support cases, and did not find anything relevant under "closed" category. 

2. Most Important. Please describe in details which partitions you resize. Please show how your partitioning is looking now. Also, did you manage to find what is consuming your root partition space, per sk121198?

[carl_t]

[Expert@-MNGMNT-01:0]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_splat-lv_current 32G 31G 1.2G 97% /
/dev/sda1 291M 43M 233M 16% /boot
tmpfs 7.8G 30M 7.7G 1% /dev/shm
/dev/mapper/vg_splat-lv_log 915G 909G 6.6G 100% /var/log
cgroup 7.8G 0 7.8G 0% /sys/fs/cgroup

[Bob_Zimmerman]

What output do you see when you run the commands 'lvs' and 'vgs'? Here's an example from one of my lab management servers:

[Expert@DallasSA]# lvs
  LV          VG       Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  hwdiag      vg_splat -wi-a-----  1.00g                                                    
  lv_TestSnap vg_splat -wi-a----- 14.50g                                                    
  lv_current  vg_splat -wi-ao---- 32.00g                                                    
  lv_fcd_GAIA vg_splat -wi-a-----  8.00g                                                    
  lv_log      vg_splat -wi-ao---- 60.00g                                                    
[Expert@DallasSA]# vgs
  VG       #PV #LV #SN Attr   VSize   VFree  
  vg_splat   1   5   0 wz--n- 356.69g 241.19g

I generally run my production SmartCenters with 50 GB allocated to lv_current. I give MDSs 200 GB.

31 GB of used space on a SmartCenter is odd, and the 'find' commands should help you figure out what's causing that. lvs and vgs should tell you how much room you have to expand.

[_Val_]

Hi Carl,

I received and reviewed the support case. I do agree with the support argument that you need to re-allocate disk partition and to add more space to the root partition, per the support engineer recommendation. 

I also see that one of your engineers misunderstood the Open Server partition guidance, which only lists the minimal requirements. The actual needs depend on many factors: number of policy packages, logs per day, etc., and may be much higher than the minimal requirements.

I have to say, your SMS does not seem to have a particularly large HDD, to begin with, and definitely, 1.5 GB is a very small partition to any reasonably sized production server.

As mentioned in sk121198, please review the largest files on your system to see if any of them can be removed.

1. find / -size +500M -type f -exec ls -l {} \;
2. find / -size +200M -type f -exec ls -l {} \;
3. find / -size +100M -type f -exec ls -l {} \;

[carl_t]

Hi

We have 1.5TB not GB, also it says on the SK that 1TB is "recommended" not minimum.

Also, the partition that is full is not what our logs are stored, these are not in the /opt

[_Val_]

Do you know what is taking all that space? Did you look into this? There should be an answer

[the_rock]

Here is something easy to try. Run this command from expert mode of mgmt server -> find /var/log -size +900M

That will look for ANY files in that dir bigger than 900 Mbs, check whats safe to delete.

Hope that helps.

Best,

Andy

[genisis__]

I'm just sizing up some Management servers and all of them will have 1TB of disk except the Logging Server which I'm going with 3TB.

The key thing in sizing this is what your daily logs + retention policy is.

If it helps, I did this for 7 months.
Daily total * 148 working days
Weekends/Bank Holidays = (Daily Log Total - 30%) * 65
This gave me a total estimated size for the log partition, then for everything else 600GB

[the_rock]

Super valid point.

Andy

[genisis__]

Another cause for increased diskspace on this server could be that upgrades, on top of upgrades where done.

[Amir_Senn]

Good way of investigating storage issue:

Change directory to /var/log

use: "du -s ./* | sort -n"

You can further change directories and repeat if needed.

[Eviatar_Abutbul]

Hi Carl, 

My name is Eviatar and I am a Group manager in Check Point support.

I understand that a team leader from my team has reached out to you and the partner, with our technology leader and aligned on the plan forward.

I will reach out to you on private to make sure we are all aligned and to understand the experience better.

Thanks and have a great day.

Eviatar

[carl_t]

Hi, we are all sorted now thanks.

[Eviatar_Abutbul]

Happy to hear, if my help is still required, feel free to reach out.

Thanks!