vg_splat-lv_current 100% full

[Marius-Edwards]

Hi Folks

I am experiencing an issue where my vg_splat-lv_current is running full. My other MGMT is not experiencing the same issue.

Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_splat-lv_current 200G 200G 20K 100% /
/dev/sda1 291M 83M 193M 31% /boot
tmpfs 31G 4.9M 31G 1% /dev/shm
/dev/mapper/vg_splat-lv_log 6.3T 3.2T 3.2T 50% /var/log
cgroup 31G 0 31G 0% /sys/fs/cgroup

Kind Regards

[the_rock]

Is this management HA?

[Marius-Edwards]

yes Sir,

[the_rock]

You can check things from the other reply, but in all honesty, considering that / dir is full, it sounds pretty serious, so I would work with TAC, just to be on the safe side, as if you delete wrong thing, you can corrupt the whole system, which would then require reinstall.

Andy

[the_rock]

You can also try below, but honestly, considering its root dir /, I would be super careful and may be worth TAC assistance.

Andy

find / -size +500000000c (this will search for any files bigger than 500 MBs in / dir)

You can also refer to below link

https://community.checkpoint.com/t5/Security-Gateways/Disk-Space-issues-on-Gateway/m-p/161537#M28601

Ultimately, you may need to do this

How to add hardware resources, such as log storage, to a Virtual Machine running Gaia OS (checkpoint...

[Marius-Edwards]

I will have a look at SK's and update on resolution.

Have TAC open, but taking a bit long to respond.

[the_rock]

Sounds good. If its urgent, I would pick up the phone and call and say you need to get this going, otherwise, via email, it wont be so fast.

Andy

[Marius-Edwards]

Looking the output form the command and drilling down:

[Expert@FWMGMT_CDC:0]# du -h --max-depth=1 /var/log/opt | sort -n -r
780G /var/log/opt/CPrt-R81.10
526M /var/log/opt/CPshrd-R81
247M /var/log/opt/CPshrd-R81.10
192M /var/log/opt/CPSmartLog-R81
146M /var/log/opt/CPsuite-R80.40
144M /var/log/opt/CPrt-R80.40
8.0K /var/log/opt/CPSmartLog-R80.40
3.6G /var/log/opt/CPsuite-R81
3.1T /var/log/opt
2.4T /var/log/opt/CPsuite-R81.10
2.2M /var/log/opt/CPshrd-R80.40
1.5G /var/log/opt/CPrt-R81
1.3M /var/log/opt/CPSmartLog-R81.10

[the_rock]

/var/log is 780GB, BUT, that would NOT fix your issue withg / dir...I would honestly call TAC and see what can be done. This is a bit of a tricky situation...

[GrassF]

Hi, How did you solved this issue with vg_splat-lv_current running full?

I'm getting the same issue. Also Full-HA - I already deleted old backups and snapshots on the 5000 appliance.

Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_splat-lv_current 32G 30G 639M 98% /
/dev/sda1 289M 103M 172M 38% /boot
tmpfs 7.7G 8.0M 7.7G 1% /dev/shm
/dev/mapper/vg_splat-lv_log 59G 37G 20G 66% /var/log
cgroup 7.7G 0 7.7G 0% /sys/fs/cgroup

Thank you

[Marius-Edwards]

Hi Grass,

There is a known issue where the secondary mgmt's root partition runs full. I got TAC involved whereby the provided a wrapper hotfix to be install on the secondary mgmt only. They also advised me to upgrade to R81.20 take 26 which we did a few weeks ago which resolved the issue.

Also a couple of other commands they asked to run before we installed the wrapper:

Vacuum the individual tables first

#psql_client cpm postgres

Then run:

SELECT nspname || '.' || relname AS "relation", pg_size_pretty(pg_total_relation_size(C.oid)) AS "total_size" FROM pg_class C LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace) WHERE nspname NOT IN ('pg_catalog', 'information_schema') AND C.relkind <> 'i' AND nspname !~ '^pg_toast' ORDER BY pg_total_relation_size(C.oid) DESC LIMIT 15;

Then vacuum the individual tables first, example:

vacuum (full, verbose) installstatus;
vacuum (full, verbose) abstractauditlogbase;
etc etc
From the smaller table over 100mb to the higher

[GrassF]

Hi Marius,

thank you for your quick reply. We are already running R81.20 take 26. I'll involve TAC on this issue.

Thank you

[the_rock]

I would do the same if I were you. I still recall while back, customer deleted wrong thing from / dir, did not have a snapshot, and after reboot, even restoring the backup did not help, so I ALWAYS caution people to be extra careful before deleting anything from root dir. I would say at least have snapshot generated, just in case.

Kind regards,

Andy

[just13pro]

check your /home/<user> directory, delete those unnecessary files.

Once 100%, you will not able to login to Gaia portal.

[the_rock]

Portal may still work, but that would be least of one's problems.

[CheckPointerXL]

i confirm the bad behavoir on r81.10 Take110 

my MDLS is getting bigger inside  /opt/CPshrd-R81.10/database/postgresql/data   (around 22GB) with / to 100% and server crashed (no communication with MDS)

added new disk to increase root volume to 100GB to resume MDLS and after 2days the postgresql folder magically dropped to 5GB.... check point mysteries...