Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
Marius-Edwards
Explorer

vg_splat-lv_current 100% full

Hi Folks

I am experiencing an issue where my vg_splat-lv_current is running full. My other MGMT is not experiencing the same issue.

Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_splat-lv_current 200G 200G 20K 100% /
/dev/sda1 291M 83M 193M 31% /boot
tmpfs 31G 4.9M 31G 1% /dev/shm
/dev/mapper/vg_splat-lv_log 6.3T 3.2T 3.2T 50% /var/log
cgroup 31G 0 31G 0% /sys/fs/cgroup

Kind Regards

 

0 Kudos
15 Replies
the_rock
Legend
Legend

Is this management HA?

0 Kudos
Marius-Edwards
Explorer

yes Sir,

0 Kudos
the_rock
Legend
Legend

You can check things from the other reply, but in all honesty, considering that / dir is full, it sounds pretty serious, so I would work with TAC, just to be on the safe side, as if you delete wrong thing, you can corrupt the whole system, which would then require reinstall.

Andy

0 Kudos
the_rock
Legend
Legend

You can also try below, but honestly, considering its root dir /, I would be super careful and may be worth TAC assistance.

Andy

find / -size +500000000c (this will search for any files bigger than 500 MBs in / dir)

You can also refer to below link

https://community.checkpoint.com/t5/Security-Gateways/Disk-Space-issues-on-Gateway/m-p/161537#M28601

 

Ultimately, you may need to do this

How to add hardware resources, such as log storage, to a Virtual Machine running Gaia OS (checkpoint...

0 Kudos
Marius-Edwards
Explorer

I will have a look at SK's and update on resolution.

Have TAC open, but taking a bit long to respond.

0 Kudos
the_rock
Legend
Legend

Sounds good. If its urgent, I would pick up the phone and call and say you need to get this going, otherwise, via email, it wont be so fast.

Andy

0 Kudos
Marius-Edwards
Explorer

Looking the output form the command and drilling down:

[Expert@FWMGMT_CDC:0]# du -h --max-depth=1 /var/log/opt | sort -n -r
780G /var/log/opt/CPrt-R81.10
526M /var/log/opt/CPshrd-R81
247M /var/log/opt/CPshrd-R81.10
192M /var/log/opt/CPSmartLog-R81
146M /var/log/opt/CPsuite-R80.40
144M /var/log/opt/CPrt-R80.40
8.0K /var/log/opt/CPSmartLog-R80.40
3.6G /var/log/opt/CPsuite-R81
3.1T /var/log/opt
2.4T /var/log/opt/CPsuite-R81.10
2.2M /var/log/opt/CPshrd-R80.40
1.5G /var/log/opt/CPrt-R81
1.3M /var/log/opt/CPSmartLog-R81.10

 

0 Kudos
the_rock
Legend
Legend

/var/log is 780GB, BUT, that would NOT fix your issue withg / dir...I would honestly call TAC and see what can be done. This is a bit of a tricky situation...

0 Kudos
GrassF
Contributor

Hi, How did you solved this issue with vg_splat-lv_current running full?

I'm getting the same issue. Also Full-HA - I already deleted old backups and snapshots on the 5000 appliance.

Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_splat-lv_current 32G 30G 639M 98% /
/dev/sda1 289M 103M 172M 38% /boot
tmpfs 7.7G 8.0M 7.7G 1% /dev/shm
/dev/mapper/vg_splat-lv_log 59G 37G 20G 66% /var/log
cgroup 7.7G 0 7.7G 0% /sys/fs/cgroup

Thank you

0 Kudos
Marius-Edwards
Explorer

Hi Grass,

There is a known issue where the secondary mgmt's root partition runs full. I got TAC involved whereby the provided a wrapper hotfix to be install on the secondary mgmt only. They also advised me to upgrade to R81.20 take 26 which we did a few weeks ago which resolved the issue.

Also a couple of other commands they asked to run before we installed the wrapper:

Vacuum the individual tables first

#psql_client cpm postgres

Then run:

SELECT nspname || '.' || relname AS "relation", pg_size_pretty(pg_total_relation_size(C.oid)) AS "total_size" FROM pg_class C LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace) WHERE nspname NOT IN ('pg_catalog', 'information_schema') AND C.relkind <> 'i' AND nspname !~ '^pg_toast' ORDER BY pg_total_relation_size(C.oid) DESC LIMIT 15;

Then vacuum the individual tables first, example:

vacuum (full, verbose) installstatus;
vacuum (full, verbose) abstractauditlogbase;

etc etc
From the smaller table over 100mb to the higher

0 Kudos
GrassF
Contributor

Hi Marius,

thank you for your quick reply. We are already running R81.20 take 26. I'll involve TAC on this issue.

Thank you

0 Kudos
the_rock
Legend
Legend

I would do the same if I were you. I still recall while back, customer deleted wrong thing from / dir, did not have a snapshot, and after reboot, even restoring the backup did not help, so I ALWAYS caution people to be extra careful before deleting anything from root dir. I would say at least have snapshot generated, just in case.

Kind regards,

Andy

0 Kudos
just13pro
Collaborator

check your /home/<user> directory, delete those unnecessary files.

Once 100%, you will not able to login to Gaia portal.

0 Kudos
the_rock
Legend
Legend

Portal may still work, but that would be least of one's problems.

0 Kudos
CheckPointerXL
Advisor
Advisor

i confirm the bad behavoir on r81.10 Take110 

my MDLS is getting bigger inside  /opt/CPshrd-R81.10/database/postgresql/data   (around 22GB) with / to 100% and server crashed (no communication with MDS)

added new disk to increase root volume to 100GB to resume MDLS and after 2days the postgresql folder magically dropped to 5GB.... check point mysteries...

0 Kudos

Leaderboard

Epsum factorial non deposit quid pro quo hic escorol.

Upcoming Events

    CheckMates Events