topic Re: Gaia partition misalignment in Firewall and Security Management

Gaia partition misalignment

nmelay — Fri, 28 Oct 2022 14:32:36 GMT

During Gaia installation (appliance/open server), partitions are NOT aligned on a 1MB boundary, but are instead cylinder-aligned, in a MS-DOS compatible way.

This alignment turns to a real performance problem with today's RAID, SSDs and AF HDDs.
Filesystem blocks being misaligned with storage blocks leads to read-before-write operations, which can incur a severe performance hit.
My own measurements showed storage performance being more than halved on some specific workload.
(The worst case scenario probably is heavy SmartEvent activity.)

This issue was fixed in WS 2008 and RHEL 6, when the performance hit first became obvious.
Gaia should have inherited the fix from RHEL, but this did not happen due to the use of a custom installer.
The packaged fdisk utility was fixed, the installer was not.

Fixing an installed Check Point system is almost impossible and requires LVM wizardry.

Please fix the installer and make sure partitions are 1MB-aligned at installation time.

Re: Gaia partition misalignment

nmelay — Fri, 28 Oct 2022 14:34:03 GMT

The above was posted as an RFE to https://usercenter.checkpoint.com/ucapps/rfe/ (reference number: lH5U9X43H), and I'm bringing it here to raise general awareness.

I know this has been reported before, and wrongly dismissed as an issue of the past.
The issue is very real, and very current on Check Point software.
I really want this to get ironed out now, there's no excuse for this 10-15 years after it was fixed by everyone else.
Sadly, this is probably too late for R81.20.

Re: Gaia partition misalignment

PhoneBoy — Mon, 31 Oct 2022 02:15:06 GMT

R81.20 has a newer installer—might be worth checking the Public EA to see if it has the same issue.

Re: Gaia partition misalignment

G_W_Albrecht — Mon, 31 Oct 2022 09:19:20 GMT

How to check for that in a simple way ?

Re: Gaia partition misalignment

_Val_ — Mon, 31 Oct 2022 12:07:54 GMT

I have asked R&D owners to comment here, please give them a bit of time to do that.

Spoiler alert: AFAIK, R81.20 clean install should resolve the issue.

Re: Gaia partition misalignment

nmelay — Mon, 31 Oct 2022 14:19:28 GMT

I did not get to play with the R81.20 ISO, only the upgrade package.

Re: Gaia partition misalignment

_Val_ — Mon, 31 Oct 2022 14:21:21 GMT

When you are upgrading, you get stuck with your "old" file system anyway

Re: Gaia partition misalignment

nmelay — Mon, 31 Oct 2022 14:21:56 GMT

fdisk -l /dev/sda

Every partition's first sector (Start) should be a multiple of 2048.
That's especially true for the LVM PVs.

Re: Gaia partition misalignment

nmelay — Mon, 31 Oct 2022 14:22:59 GMT

Thanks Val for the good news.
I'm delighted to hear this topic is getting the attention it deserves.

Re: Gaia partition misalignment

_Val_ — Mon, 31 Oct 2022 14:28:32 GMT

Sure thing. I can promise you, this particular issue is addressed very seriously.

Re: Gaia partition misalignment

nmelay — Mon, 31 Oct 2022 15:00:20 GMT

Indeed.

The "easiest" way to fix the alignment on an existing system is to create a new LVM PV, move LVs to the new PV, recreate the original PV correctly aligned, and move everything back.
That's something you can manage on a VM with enough available storage.
(Also, the misalignment/realignment will break deduplication on smart NAS/SAN storage, so you won't benefit from this.)

On a physical server or Smart-1 appliance, you need to shrink the existing PV, resize the hosting partition, create a new partition/PV, move LVs...

Anyway, you need to know what you're doing and make sure solid backups are not too far away.

Re: Gaia partition misalignment

itzhakd — Mon, 31 Oct 2022 15:43:27 GMT

Indeed as mentioned above, R81.20 release will have a new installer (with new fdisk of course) so clean installation will align the partitions to a 1MB boundary.

Re: Gaia partition misalignment

nmelay — Wed, 02 Nov 2022 11:15:13 GMT

OK, I did so, and verified that R81.20 EA correctly aligns partitions. 👍

Re: Gaia partition misalignment

nmelay — Wed, 02 Nov 2022 11:40:16 GMT

Thanks Itzhak for your confirmation.

It will be a while before R81.20 is widely adopted.
Do you know if there will there be an updated R81.10 ISO? And fix instructions for existing setups?

Re: Gaia partition misalignment

_Val_ — Wed, 02 Nov 2022 13:21:40 GMT

I hope you are happy now 🙂

Re: Gaia partition misalignment

nmelay — Wed, 02 Nov 2022 14:52:53 GMT

Yes I am!
Problem was fixed before I even got to report it, that's quite a feat!

Still I'd like to know where Check Point stands on this issue regarding current releases and upgrades.

Re: Gaia partition misalignment

_Val_ — Wed, 02 Nov 2022 14:58:31 GMT

I will let R&D answer the question about clean install, although I find it very unlike that we will be able to change that, too many moving parts for the released version: production lines, install tolls, Blink images, etc.

For the upgrades, you are stuck with your pre-existing file system, unless you perform a clean install.

Re: Gaia partition misalignment

_Val_ — Wed, 02 Nov 2022 15:28:41 GMT

Okay, after some internal chatter, it seems we will not be fixing the alignment with the previous versions.

Also, a personal note, in my 24 years with CP, I have never seen that being an actual issue. Any reason?

Re: Gaia partition misalignment

nmelay — Wed, 02 Nov 2022 18:23:40 GMT

Thanks for your time Val, I really appreciate this.

I take note that previous/existing versions won't be updated, I can understand this.
I'm really glad the fix is part of the R81.20 release, whether is was intentional or a side effect of the new installer. 😉

I wish the issue was clearly documented -- maybe this will happen once R81.20 is released.
Right now, Best practices for running SMS on VMware (sk104848) states you should "make sure the disk partitions within the guest are aligned"... but omits to say Gaia installer (up to R81.10) will forbid you from doing so.

What brought the issue back to my attention recently was a customer with abysmal SmartEvent performance.
A colleague of mine spent days relocating the log server to a new hosting infrastructure and reindexing everything, only to get a minor performance improvement.
When I had a look at it, the misalignment issue was obvious to me. With a few decades of systems/storage experience, misaligned partitions bring back old memories of poorly performing database workloads, wrongly designed/poorly documented storage vendor fixes (I'm looking at you, NetApp!) and so on.
I knew I saw it before on Check Point installs, but only then realized ALL of them are affected: old and new appliances, open servers, even CloudGuard IaaS Azure instances.
We did NOT spend more days fixing this for this specific customer -- we probably won't until he brings it back and we get an officially supported procedure for this. But I did run some tests on my own and it did not look good.

Re: Gaia partition misalignment

_Val_ — Wed, 02 Nov 2022 18:49:34 GMT

Thanks for the thorough write-up.

However, I will ask again, what were the issues you experienced with Check Point, related to the cylinder-aligned partitions?