Disk Space issues on Gateway

[ESpataro]

I am trying to clear some disk space on one of our Gateways as teh Var/log area is 88% used.

However I am unsure on which files can be safely deleted , below is an output showing the directories which seem to be taking up space and they seem to older versions of checkpoint. These may be old files from previous upgrades but I am not sure

any help appreciated 

Expert@xxxx-xxx:0]# du -h --max-depth=1 /var/log/opt | sort -n -r
832K    /var/log/opt/CPcvpn-R80.30
665M    /var/log/opt/CPsuite-R80.20
380K    /var/log/opt/CPcvpn-R80.20
192K    /var/log/opt/CPshrd-R80
112M    /var/log/opt/CPsuite-R80
20G     /var/log/opt
8.8G    /var/log/opt/CPsuite-R80.30
7.1M    /var/log/opt/CPshrd-R80.30
5.2M    /var/log/opt/CPshrd-R80.20
5.1G    /var/log/opt/CPshrd-R80.40
2.2M    /var/log/opt/CPcvpn-R81.10
2.1G    /var/log/opt/CPsuite-R80.40
1.8G    /var/log/opt/CPshrd-R81.10
1.5M    /var/log/opt/CPcvpn-R80.40
1.5G    /var/log/opt/CPsuite-R81.10

[PhoneBoy]

Deleting any of the directories is not recommended.
Might try something like the following: https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...

[ESpataro]

Thanks , I have seen this sk article , but I have been told that this script is not compatible with R81.10 unfortunately 

[Hugo_vd_Kooij]

That SK has itself been deleted 😉

[the_rock]

I see the same, says has been deleted.

[_Val_]

It has been indeed un-published. Use https://support.checkpoint.com/results/sk/sk65330

[the_rock]

I always do something like this. First, run df -h and see what dir is the "fullest". Then, say it shows its /var/log at, for argument sake, at 90% capacity, do something like this:

find /var/log -size +500000000c 

That will look for ANY files bigger than 500 MB in /var/log. You can apply same method for any dir and any file size.

Andy

[ESpataro]

Thanks Andy , the issue I have though knowing which files can be safely deleted , as you can see from my output above , for example

8.8G    /var/log/opt/CPsuite-R80.30 - this directory is taking up 8.8G
within these directories I am not sure which files are safe to delete

[the_rock]

Ok, got it...can you run find command on that dir? So say find /var/log/opt/CPsuite-R80.30 -size +500000000c

Andy

[ESpataro]

This what I get from that output 

[Expert@xxxx-xxx:0]# find /var/log/opt/CPsuite-R80.30 -size +500000000c
/var/log/opt/CPsuite-R80.30/fw1/log/2020-09-05_000000.log
/var/log/opt/CPsuite-R80.30/fw1/log/2020-09-04_000000.log
/var/log/opt/CPsuite-R80.30/fw1/log/2021-02-01_000000.log
/var/log/opt/CPsuite-R80.30/fw1/log/2020-08-16_000000.log
/var/log/opt/CPsuite-R80.30/fw1/log/2020-08-17_000000.log

[ESpataro]

Is it safe to delete the above files ?

[G_W_Albrecht]

I would say yes 😎 But please also consult the following documents:

sk63361: How to clean up disk space on a Security Gateway or Security Management Server

sk114114: Disk space management tools do not delete logs from previous Security Management versions

[the_rock]

Its from 2021 and 2020, Covid years my friend : - ). I think safe to delete.

[G_W_Albrecht · Accepted solution]

CP special tipp:

I would suggest to run the following command as well:

find / -type f -size +100000 -exec ls -lh {} \; 2> /dev/null | awk '{ print $NF ": " $5 }' | sort -nk 2,2

This will display all files greater than 10MB and sort them in a readable and understandable way.

[Daniel_Kavan]

Has anyone seen kcore file, ok to delete?

find / -type f -size +100000000 -exec ls -lh {} \; 2> /dev/null | awk '{ print $NF ": " $5 }' | sort -nk 2,2
/opt/CPsuite-R81.20/fw1/te_file_analyzer/proc/kcore: 128T
/proc/kcore: 128T
/var/log/aspose/opt/CPsuite-R81.20/fw1/aspose_jail/proc/kcore: 128T
/var/log/files_repository/Archive_Tool/8785455D-CDCA-49C6-B22F-D9325B260629/601178/proc/kcore: 128T

Ok to remove these 81 & 81.10 files, now that I"m no 81.20?

/var/log/CPda/repository/CheckPoint#Major#All#6.0#5#2#R81_T392/Check_Point_R81_T392_Fresh_Install_and_Upgrade_v1.tgz: 3.4G
/var/log/CPda/repository/CheckPoint#Major#All#6.0#5#3#R81.10_ignis_main_T335/Check_Point_R81.10_T335_Fresh_Install_and_Upgrade.tgz: 3.4G
/var/log/CPda/repository/CheckPoint#Major#All#6.0#5#4#R81.20_ivory_main_T631/Check_Point_R81.20_T631_Fresh_Install_and_Upgrade.tgz: 3.7G
/var/log/CPda/repository/CheckPoint#Major#All#6.0#5#3#BLINK_R81_10_T335_JHF_T45_GW/Blink_image_1.1_Check_Point_R81.10_T335_JHF_T45_SecurityGateway.tgz: 5.2G

[the_rock]

This may explain...

Andy

https://www.linode.com/community/questions/21878/why-is-my-prockcore-file-so-large#:~:text=The%20nea....

[Daniel_Kavan]

Thanks!

Also, any harm getting rid of older stuff in the repository?  /var/log/CPda/repository    I have some R80.40 and R81 files not to mention R81.10 now.

How about these older R80.40 .dat files?

/var/log/files_repository/Raw_Files/D100C612-9D05-48AA-9F2A-F8C42528F100/600400/te_eu_fireball_11_9_990000400_600400.tgz: 505M
/var/log/opt/CPsuite-R81/fw1/log/mq_mng.elg: 657M
/var/log/opt/CPshrd-R80.40/cpview_services/CPViewDB_1616105839.dat: 853M
/var/log/opt/CPshrd-R80.40/cpview_services/cpview_services.dat: 872M
/var/log/opt/CPshrd-R80.40/cpview_services/CPViewDB_1600974992.dat: 879M
/var/log/opt/CPshrd-R80.40/cpview_services/CPViewDB_1611009647.dat: 927M
/var/log/opt/CPshrd-R81/cpview_services/CPViewDB_1628947028.dat: 953M

[the_rock]

I always delete whatever is there after the upgrade. Maybe you can confirm 100% with TAC if its safe, but I never had any issues after deleting old files form that dir. Just make sure NOT to delete any files from dir called LastTake (cant recall where its located now, I think under /var/log as well), as that would affect next jumbo install.

Andy

[the_rock]

@Daniel_Kavan , here is dir I was referring to (do NOT delete anything from here)

[Expert@CP-gw:0]# find / -name LastTake
/opt/CPda/backup/CheckPoint#CPUpdates#All#6.0#5#4#BUNDLE_R81_20_JUMBO_HF_MAIN#26/LastTake
[Expert@CP-gw:0]#

[aheilmaier]

I have the problem with some R81.20 machines.

I just wonder, because we added file deletion on the device object "Logs>Local Storage", when disk space is below 15%.
I for my point of view the device does not delete any files, so it looks like the disk space is not checked on the /var/log partition but only on the complete disk.

Am I right with that ? but this policy does not make any sense to me.

Filesystem                       Size  Used Avail Use% Mounted on
/dev/mapper/vg_splat-lv_current   32G   19G   14G  57% /
/dev/md0                         290M  116M  160M  42% /boot
tmpfs                             31G  543M   31G   2% /dev/shm
/dev/mapper/vg_splat-lv_log      192G  174G   19G  91% /var/log

[PhoneBoy]

Which is funny because this SK says the "hardcoded limit" for specifying this limit is 25% https://support.checkpoint.com/results/sk/sk182048
Which suggests you may want to contact the TAC.

[aheilmaier]

thank's for the sk on the mgmt server.

And I was wrong I had a look at the wrong directory, as always my backup files filled the /var/log partition.

Forgot to add the cronjob to delete the backups once per day.

[the_rock]

Personally, I always found once a week is good enough doing a backup.

Best,

Andy

[Hugo_vd_Kooij]

I find that cpview tends to leave about large files. Yesterday I was upgrading a cluster of 3100 appliances and /var/log was filled at 81%

In /var/log/opt/CPshrd-R80.40 there was a bunch of 5 large ond cpview_xxxxxxxxx.dat files. and getting rid of them lowered disk usage in /var./log to 31%

As far as old logs go if your gateway is supposed to send them of to the SmartCenter then any set of log files indicate connectivity iissues where the gateway is unable to send logs to the SmartCenter. If that happens a lot you need to investigate why this happens. As a normal safeguard you should  configure logging to forwar live logging but aalso roud up the logs once a day in case stuff gets left behind.

There is a bunch of notes that I must put infto some best practises documents some day. Propably also put it into a nice Ansible playbook as a lot of cleaning up can be automated in my view.

(Just don't hold you breath on it. It's on a too long to-do list.)