Snapshot question

[the_rock]

Hey guys,

Im wondering if someone could confirm this for me. I know as per below doc, it states when you run snapshot, everything continues to run fine:

https://sc1.checkpoint.com/documents/R81/WebAdminGuides/EN/CP_R81_Gaia_AdminGuide/Topics-GAG/Snapsho...

Well, that does not seem to happen in my lab. I tested with R81.10 and R81.20 gateways and though I could access ssh, it was super slow and barely responsive and GUI was not loading. Now, weird thing is, policy did show the same, but in smart console, policy push failed and was complaining about sic. Obviously, when cpstop is issued, it removed the current policy from the firewall, so that tells me it was not done in my lab, but I still find it odd that access was so sluggish.

About 10 mins later, I manually deleted the snapshot from web UI and all came back green.

Thoughts?

[Danny · Accepted solution]

I see this as well on appliances with less than 4 CPU cores.
I recommend announcing a maintenance window for snapshot creation for appliances lower than the 6700 series.
Also make sure your snapshot partition is sized big enough.

[the_rock]

Hey @Danny 

Thanks kindly for your response. Just for context, R81.10 lab has 8 cores and R81.20 4 cores. Both are running on 16 GB ram and esxi 6.7 server. Not sure if that makes any difference compared to a physical appliance, but yes, I will let my colleague know to tell customer not to generate a snapshot during work hours. Better be safe than sorry.

[Bob_Zimmerman]

I have a bunch of systems running R80.40 and R81.10, both installed clean (with XFS) and upgraded from lower versions (so still running ext3). Most use spinning disks, so should be pretty much the worst case in terms of storage performance on dedicated hardware. I take snapshots live all the time with no issues.

A snapshot on GAiA seems to involve taking an LVM snapshot (to ensure filesystem consistency), creating a new LV in unallocated PV space, copying the data from the snapshot to the new LV, then deleting the snapshot. It's extremely disk-I/O-intensive, but shouldn't affect the processor or RAM load noticeably.

While the LVM snapshot is live, all reads and writes from the normal filesystem would be multiplied, since XFS and ext3 aren't actually aware of the snapshot. This is one of the reasons I like ZFS so much for my personal systems. Since ZFS is both the volume manager (handling arranging the data on the physical storage) and the filesystem (handling the mapping from file names to data blocks), and it's copy-on-write, volume-level snapshots are extremely fast to create (milliseconds) and don't have an ongoing performance impact while live.

Virtualization would also hurt disk I/O performance pretty seriously. Maybe the combination of the virtualization hit and the snapshot hit causes some disk operation during a policy push to time out? Are you using spinning disks or SSDs for the VM datastore?

[the_rock]

That makes sense @Bob_Zimmerman . These servers are NOT ssd, so that may explain slowness when connecting to ssh once I executed to create brand new snapshot, BUT, here is where my confusion comes in. So, since it was clear that services most likely did not stop, as policy did not change, I wonder why it was showing that SIC was broken?? And then after 10 mins or so, once I was able to log back into web UI and delete the snapshot, it all came back green, no issues and SIC was fine.

I know that indeed SIC was broken, since policy push failed while snapshot was being created.

Andy