R80.x Multi-Domain backup size

[Kaspars_Zibarts]

Another question regarding R80 MDM backups. Ours has grown at the rate of 1GB per month since upgrade to R80 which seems excessive. The very first backup directly after R80 upgrade was just over 5GB and last one taken is over 11GB. Checkpoint has always been rather complacent regarding backup and it's contents - seems like directories are added in without any analyses if it's really needed or not. And no one can be bothered to make any improvements. I used to manually exclude lots of tmp files and directories to keep the backup size reasonable and even raised couple of service requests. But with R80 it got much more difficult as the main culprit is the PostgreSQL DB dump. The very first backup compressed DB was just under 300MB (mds_backup_dump.gz file within the MDS archive):

[Expert@mdms01:0]# ls -lh

total 5.3G
-rw-r--r-- 1 admin root 11M Mar 20 14:06 CPSmartLog-R80_opt.tgz
-rwxr-xr-x 1 admin root 261K Mar 20 13:51 gtar
-rwxr-xr-x 1 admin root 73K Mar 20 13:51 gzip

-rw-r--r-- 1 admin root 276M Mar 20 13:55 mds_backup_dump.gz
-rw-r--r-- 1 admin root 1.1G Mar 20 13:53 mds_backup_opt.tgz
-rw-r--r-- 1 admin root 129M Mar 20 14:06 mds_backup_opt_compkgs.tgz
-rw-r--r-- 1 admin root 457 Mar 20 14:05 mds_backup_supkgs.tgz
-rw-r--r-- 1 admin root 2.2K Mar 20 14:06 mds_backup_var_compkgs.tgz
-rw-r--r-- 1 admin root 3.8G Mar 20 14:05 mds_backup_varopt.tgz
-rwxr-x--- 1 admin root 18K Mar 20 13:51 mds_restore

and 5 months later it has grown to almost 5GB!

[Expert@mdms01:0]# ls -lh 

total 11G
-rw-r--r-- 1 admin root 13M Aug 15 15:06 CPSmartLog-R80_opt.tgz
-rwxr-xr-x 1 admin root 261K Aug 15 14:40 gtar
-rwxr-xr-x 1 admin root 73K Aug 15 14:40 gzip

-rw-r--r-- 1 admin root 4.6G Aug 15 14:54 mds_backup_dump.gz
-rw-r--r-- 1 admin root 1.5G Aug 15 14:43 mds_backup_opt.tgz
-rw-r--r-- 1 admin root 129M Aug 15 15:05 mds_backup_opt_compkgs.tgz
-rw-r--r-- 1 admin root 1.1K Aug 15 15:05 mds_backup_supkgs.tgz
-rw-r--r-- 1 admin root 2.2K Aug 15 15:05 mds_backup_var_compkgs.tgz
-rw-r--r-- 1 admin root 4.1G Aug 15 15:05 mds_backup_varopt.tgz
-rwxr-x--- 1 admin root 18K Aug 15 14:40 mds_restore

And this is running manual mds_backup trying to exclude as much as you can. 

Anyone with good tips to resolve this?

[PhoneBoy]

FYI I moved this and your other post into the https://community.checkpoint.com/community/management/maintenance-upgrades?sr=search&searchId=66fabc...‌ space as that's a more appropriate place for the questions you asked.

One of the major "under the hood" changes in R80+ is the fact the CMA configuration is now a proper database versus flat files.

That provides more capabilities, but obviously as you use it, the database will get bigger.

On the face, that seems like a huge increase, but I will admit: I don't know if that's normal or not.

I'll get someone more knowledgeable on the subject to chime in.

[Kaspars_Zibarts]

One interesting thing i noticed was the change in compression rate. Technically it's a flat text file (the DB dump). In the first example when it was 276MB, when decompressed it turned into 2.5GB text file, so ~10x compression rate. Then the latest one 4.6GB uncompressed turned into 12GB so "only" 3x compression. Strange as it's still a plain text at the end. Anyways, thanks for looking into it!

[Ofer_Barzvi]

Hello Kaspars,

In case your setup is with R80.10, you can enter the Manage & Settings --> Revisions tab in SmartConsole and delete all the revisions up to the one selected by pressing the "Purge" button.

After the purge, the dump of the posgres database should be reduced and also the size of the mds_backup.

In addition, please use the CPM Doctor (as instructed in sk117219) before and after the purge and upload the tgz files to this thread or email them to barzvi@checkpoint.com.

If your setup is with R80, the recommendation is to upgrade it to R80.10 before running the purge command.

Regards,

Ofer

[Kaspars_Zibarts]

Awesome! We had revision control disabled as we have VSX and revisions didn't work that well in past with VSX.

Is that still a case? 

Is there a quick way to purge all revisions as we have tons of them and manual way will take too long!

Can we disable revisions completely?

[Gary_Lipets]

We had the same problem, actually purging the revisions was making things worse. We worked with support and eventually they deleted bunch of IPS updates (we weren't using IPS feature) 

/// from Checkpoint support

Development just finished an R80 script that can remove previous IPS versions from the database. Each one of these versions takes up about 11MB of space. In the replication, the backup size decreased by about 1.2GB.

As you are not using IPS, it may be a good idea to disable IPS automatic updates completely. Each IPS update will not be stored in your database and will keep the Gaia backup from growing as much.

///

[Kaspars_Zibarts]

Just a quick update that SR has been sitting with R&D so I'll update as soon as I hear more. Looks like we have some issues there.

[Kaspars_Zibarts]

We have a liftoff Houston!. The fix is complex and coming out in next jumbo - that's late December. As it turned out we were not the only customer at the end affected by it. In nutshell - some tables in the DB grow "very large" so R&D is implementing new way of handling those for backup.

[Kaspars_Zibarts]

Looks like our R80.10 MDS backup size growth has "stabilised" - it went up to nearly 14GB (from just under 4GB in R77.30) and has stopped growing noticeably. Even shrunk one week.. We still have not deployed latest jumbo on MDS so I'm not too sure if that would make things even better.