topic Re: MDS backup too big and slow in R80.10 in Firewall and Security Management

MDS backup too big and slow in R80.10

Kaspars_Zibarts — Thu, 23 Aug 2018 05:09:20 GMT

For those running MDS management solution. What's your take on backup after R80.10? In our case in R77.30 backup was approx 3GB in size and it took less than half an hour to restore MDS and have it up and running. With R80.10 backup has grown to 18GB(!) within a year and actual process takes well over an hour if not closer to two. As an engineer I might accept the argument that R80.10 brought in so many new features thus increasing backup size but from business and disaster recovery point of of view it is complete shumbles.

Ironically it makes even support process painfully slow as I was asked to upload MDS backup yesterday and considering that CP FTP servers are over 50ms away from us, it will take couple of hours to complete that.

I have been raising SRs trying to point out inefficiency of MDS backup process for years - same MDS TGZ being archived and compressed 4 times... Seriously. In order to restore backup now (offcial MDS GAIA backup) we would need nearly 100GB free disk-space. Not that it costs too much money but it makes it so slow.

I'm not expecting many votes as probably not that many run MDS but still would be good to hear opinions about the matter

Re: MDS backup too big and slow in R80.10

Tomer_Sole — Fri, 21 Jun 2019 09:10:29 GMT

Hi, please see https://community.checkpoint.com/thread/6312-how-can-i-control-the-size-of-my-r8010-security-management-server

Re: MDS backup too big and slow in R80.10

Kaspars_Zibarts — Thu, 23 Aug 2018 14:16:25 GMT

I have already gone through it and it made no difference Tomer staright after updating to Take 42. Postgres just keeps growing like nuts every month

Re: MDS backup too big and slow in R80.10

Tomer_Sole — Thu, 23 Aug 2018 14:33:37 GMT

how many revisions do you have & how many IPS updates? the difference is that we didn't take history in our backups prior to R80.

Re: MDS backup too big and slow in R80.10

Kaspars_Zibarts — Thu, 23 Aug 2018 14:47:17 GMT

When this info came out originally, I went and deleted every single revision on every single CMA that we had (we're talking hundreds) but hardly made a dent in a backup size. I already have asked that we world rather not have revisions at all and instead rely on good old backup restore. But got nowhere. Is it possible to disable revisions across whole MDS?

Re: MDS backup too big and slow in R80.10

Tomer_Sole — Thu, 23 Aug 2018 14:53:37 GMT

First of all, if you deleted all your previous revisions and size was still big, then even if you could disable revisions that wouldn't have solved your particular problem.

So now I'm thinking it's a support ticket worthy. Please send me privately your support ticket so that I can track its findings.

Re: MDS backup too big and slow in R80.10

Brian_Deutmeyer — Thu, 23 Aug 2018 19:32:33 GMT

We have also seen our backup size grow. I've been running into a memory consumption issue while purging and haven't been able to purge for a while, so I assumed that was our case. We are working on getting that straightened out first. The other thing we've noticed is while backups are occurring, SmartConsole becomes slow or unresponsive.

Re: MDS backup too big and slow in R80.10

Kaspars_Zibarts — Fri, 24 Aug 2018 06:14:19 GMT

We have always run backup out of hours as it needs to halt CMAs. memory consumption - i didn't notice yesterday when I purged over 10000 revisions. No problems there. But we have 128GB on that VM so I believe that should suffice.

Re: MDS backup too big and slow in R80.10

Kaspars_Zibarts — Fri, 24 Aug 2018 06:17:34 GMT

OK, morning update: after purging over 10000 revisions across all CMAs backup size remained the same

Before

-rw-r--r--. 1 netbackup1 netbackup 17G Aug 22 04:10 backup_mds01_22_Aug_2018_02_30.tgz

After

-rw-r--r--. 1 netbackup1 netbackup 17G Aug 24 04:08 backup_mds01_24_Aug_2018_02_30.tgz

SR it is then

Re: MDS backup too big and slow in R80.10

Sander_Zumbrink — Thu, 30 Aug 2018 07:04:25 GMT

Are you running the MDS on vmware or dedicated hardware?

I've also noticed the same, mostly caused by slow disk IO.

Running on dedicated hardware speeded up the backups.

And performance of MDS itself.

Re: MDS backup too big and slow in R80.10

Kaspars_Zibarts — Thu, 30 Aug 2018 08:08:38 GMT

It's a VM with proper storage. So IO is not the issue. How big is your backup? How much did it grow from R77 to R80?

Re: MDS backup too big and slow in R80.10

Sander_Zumbrink — Thu, 30 Aug 2018 09:55:07 GMT

Our backup is around 30 Gb. On dedicated hardware it takes 1:15.

Before our transition to dedicated hardware it took around 5 hours.

We also didn't expect disk IO issues, but we saw a lot of improvement.

The dump of the postgresql database was causing a lot of disk IO and took some time.

We've migrated a long time ago. I don't know the size of the backup from R77.30 anymore.

Re: MDS backup too big and slow in R80.10

Kaspars_Zibarts — Thu, 30 Aug 2018 10:05:27 GMT

Did you mean 1hr to create backup? That's normal. Ours is 17GB and it takes approximately half the time to create it. The problem is restore time for me. Takes way too long imo

Re: MDS backup too big and slow in R80.10

Sander_Zumbrink — Thu, 30 Aug 2018 13:01:23 GMT

The restore also made a big difference on the physical hardware. Restoring the 30 Gb and the reinitialize of the solr database to approx 14 hours in vmware. On physical hardware it was 2 hours I think. My opinion is that big MDS environments are not suitable for vmware.

Re: MDS backup too big and slow in R80.10

Kaspars_Zibarts — Thu, 30 Aug 2018 13:28:27 GMT

Hard to say about 30GB, but our 17GB restore in ~2hrs in the lab VM which is slightly under-powered. Still a long time compare what it used to be in R77. And it keeps growing quite fast (~1GB per month) I just think the whole Gaia backup process for MDS can be improved dramatically as the basic mds backup file that forms the biggest chunk in the archive is actually compressed and archived 4 times! seems an overkill. They could have used simple tar without compression after it's been compressed once..

Re: MDS backup too big and slow in R80.10

Sander_Zumbrink — Thu, 30 Aug 2018 13:39:06 GMT

The restore itself is 2 hours. But when you start the MDS, it takes up to several ours rebuilding the SOLR.

And yes... why the use the gzip archive multiple times... that takes time... and not necessary.

Re: MDS backup too big and slow in R80.10

Tomer_Sole — Thu, 30 Aug 2018 13:47:50 GMT

I just want to say I'm tracking your support request and hope this comes to a resolution soon that might fit for other customers as well.

Re: MDS backup too big and slow in R80.10

Mike_A — Thu, 30 Aug 2018 13:53:44 GMT

Are you just simply running an mds_backup?

1.) Are you by chance using the -l flag to not include your logs as well?

2.) What dir are you running the mds_backup from?

I recently upgraded from R77.30 MDS to R80.10 and my backups grew by maybe 300-400MB. Nothing crazy.

Re: MDS backup too big and slow in R80.10

Kaspars_Zibarts — Thu, 30 Aug 2018 13:55:30 GMT

That's my point - if we whinge enough here we might get some attention. Sorry Tomer - nothing against you personally, on contrary, you have always gone extra mile and it is really appreciated. More of a constructive feedback about the product and yes, we will have another remote session next week, had issues with my lab yesterday (backup restore took too long to restore so I was not ready when meeting time was up..)

Re: MDS backup too big and slow in R80.10

Kaspars_Zibarts — Thu, 30 Aug 2018 13:59:36 GMT

Hi, no this is box standard Gaia backup that will wrap in mds_backup. Logs are excluded. The biggest increase is postgres DB dump (part of mds_backup) that has grown from 3GB orginally to nearly 10GB in one year.. So keep eye on yours. Or it could be that ours is "broken" somewhere as we went from R77.30 to R80 and then R80.10. Who knows. We'll keep it posted here about my SR progress