topic Re: upgrade to R80.20 failed in Firewall and Security Management

upgrade to R80.20 failed

Marcel_Wildenbe — Fri, 16 Nov 2018 12:45:54 GMT

Hi CheckMates,

Last night, I have tried to upgrade our MDS from R80.10 to R80.20.

I have ran into a few issues, but the most aggravating was when the installer got stuck and I had to reboot in order to get any further, the snapshot that was made by the installer was not removed and a new attempt is telling me there is no free space enough.

CP support tells me to run MDS export, do a fresh install en import, but I would like to avoid the hassle and just remove the LV.

Can I remove this Logical Volume and if so, how do I do that?

It is GAIA running on VMware 5.5. So it is using LVM for Snapshots. "show snapshots" is showing no snapshots, but lvm_manager shows me lv_fcd_new of 300 GB, non configurable, containing: Factory defaults volume, which was not present prior to the upgrade.

Re: upgrade to R80.20 failed

JozkoMrkvicka — Fri, 16 Nov 2018 12:48:26 GMT

Try to check Snapshot Management from WebUI.

Re: upgrade to R80.20 failed

Marcel_Wildenbe — Fri, 16 Nov 2018 12:52:34 GMT

I have did that and it is the equivalent of "show snapshots" in CLISH.

No snapshots available and no free space available

Re: upgrade to R80.20 failed

JozkoMrkvicka — Fri, 21 Jun 2019 09:07:23 GMT

What about "delete snapshot <TAB>" from clish ? Nothing ?

Do you have some options in case "set snapshot <TAB>" ?

Some articles regarding Snapshot management:

Snapshot location on Gaia and SecurePlatform

Managing partition sizes via LVM manager on Gaia OS

How to add hardware resources, such as log storage, to a VMware Virtual Machine running Gaia OS

https://community.checkpoint.com/thread/6448-lvmmanager-successor-on-r8010

Re: upgrade to R80.20 failed

Marcel_Wildenbe — Fri, 16 Nov 2018 13:51:35 GMT

Hi Jozko,

No options available with "delete snapshot <TAB>"

Options "set snapshot <TAB>" import, export, revert. No options after export or revert, so no valid snapshot present.

In expert:

# lvs --all
LV VG Attr LSize Origin Snap% Move Log Copy%
lv_current vg_splat -wi-ao 300.00G
lv_fcd_new vg_splat -wi-a- 300.00G
lv_log vg_splat -wi-ao 3.44T

it is the second LV, attributed "-wi-a-", so without the "o", which stands for "Open"

Re: upgrade to R80.20 failed

PhoneBoy — Fri, 16 Nov 2018 18:51:58 GMT

While I've heard of customers doing an in-place CPUSE upgrade on their MDS, there can be issues in doing so, particularly if the MDSes also contain a fair amount of logs relative to the available free disk space.

I'm guessing that's what happened here.

I would expect that TAC would be able to help you manually remove the logical volume that got created here.

Note: one reason to do a migrate export/import in this case: by doing so, you can do a fresh install of R80.20 and leverage the new xfs filesystem, which offers better performance.

Re: upgrade to R80.20 failed

Boaz_Orshav — Sun, 18 Nov 2018 08:27:01 GMT

Hi Marcel

Sorry to hear you had problem during installation.

We have worked hard on reducing upgrade failures and I can say that on latest versions (80.10+80.20) we have over 99% success rate.

Notice that rebooting the machine while installer is "stuck" is very problematic, since you might get in a situation that the new partition was created but not defined as snapshot yet.

I guess this is what happened on your machine and that's why you could not run snapshot management commands to remove it.

In order to analyze and make sure it won't happen to others (and for sure not happen in next version) I would need some more information so I will contact you offline to further investigate.

The target is 100% success so if you encounter any problem please contact and we'll solve the issue.

Re: upgrade to R80.20 failed

Kaspars_Zibarts — Sun, 18 Nov 2018 19:22:48 GMT

That was very important point Dameon! Having xfs by doing export import! I did lab testing using CPUSE and realized only afterwards that I'm missing out on the new file system!

Re: upgrade to R80.20 failed

Marcel_Wildenbe — Mon, 19 Nov 2018 07:45:38 GMT

Thanks Dameon, for your reply. Valid point you mention, regarding the xfs filesystem.

The available disk space needed for a snapshot is not related to the amount of disk space occupied by the logs, imho. The logging is written to the lv_log Logical Volume, where the snapshot is creating its own Logical Volume. They live side-by-side on the physical disk, regardless of their content. Please correct me if I am wrong.

Anyway, I will get in touch with Boaz. Hopefully he can help me out.

Fun fact: I did run a "dry run" of the upgrade on a restored snapshot of the original MDS on a different VM and it worked like a charm... That made me confident to do the actual upgrade in production.

Re: upgrade to R80.20 failed

Marcel_Wildenbe — Mon, 19 Nov 2018 07:55:00 GMT

Hi Boaz,

Thanks for reaching out. I will reply to your mail a.s.a.p. I am aware of the fact that the reboot probably got me into this predicament, hopefully we can work this out.

Re: upgrade to R80.20 failed

JozkoMrkvicka — Mon, 19 Nov 2018 20:27:11 GMT

After how many minutes (hours) you did reboot ? What was the overal progress while it stucked ?

Re: upgrade to R80.20 failed

Kaspars_Zibarts — Mon, 19 Nov 2018 21:04:27 GMT

Mine didn't get stuck but it took good 4hrs in the lab (that's rather underpowered, 8 cores 24GB ram and RAID 5..)

Re: upgrade to R80.20 failed

Marcel_Wildenbe — Tue, 20 Nov 2018 08:06:39 GMT

I did a dry run of the upgrade on a snapshot of the production database and was much quicker than the real thing.

That is why I have stopped it (after 36 minutes).

Re: upgrade to R80.20 failed

Marcel_Wildenbe — Tue, 20 Nov 2018 08:08:32 GMT

I would like to go back to the initial question: "Can I remove this Logical Volume and if so, how do I do that?"

Re: upgrade to R80.20 failed

Marcel_Wildenbe — Tue, 20 Nov 2018 08:48:52 GMT

Boaz (from CheckPoint) has successfully supported me on removing the Logical Volume.

I am going to re-schedule a service window to upgrade.

Re: upgrade to R80.20 failed

Marcel_Wildenbe — Fri, 23 Nov 2018 08:18:22 GMT

Check Point User Center - Customer Portal for Licensing, Support and Account Management

Last night, we have performed a new attempt to upgrade the MDS, under supervision of Boaz Orshav (R^D) and colleagues.

The upgrade failed: after the upgrade was completed, one of the five Management Domains did not come up. MDSSTAT was showing all the Management Servers as up.

But the not in the SmartConsole (MDS) and the Management Server did not receive any logging.
So, we have reverted to the AutoSnapshot, created during the start of the procedure.

Re: upgrade to R80.20 failed

Kaspars_Zibarts — Fri, 23 Nov 2018 08:21:19 GMT

what did you end up doing? CPUSE or export and import on fresh filesystem?

Re: upgrade to R80.20 failed

Marcel_Wildenbe — Fri, 23 Nov 2018 08:26:48 GMT

CPUSE

Re: upgrade to R80.20 failed

Amir_Rehman — Sun, 28 Apr 2019 17:50:25 GMT

My CPUSE upgrade fails.

I am attempting to upgrade SM R80.10 to R80.20. However I get an error during DB import process.

DB exports runs successfully 100% and the sever gets rebooted and then it starts import process.

It reverts back to R80.10 automatically when DB import fails at 50%.

Any idea please ?

Re: upgrade to R80.20 failed

Maik — Mon, 29 Apr 2019 06:18:15 GMT

@Amir_Rehman, did you run the pre upgrade verifier before you started the actual upgrade? Especially errors during the database import could be related to issues that should get detected by the verifier on the first hand.