Real life comparison of XFS and EXT3 file systems

[Kaspars_Zibarts]

To give a bit historic background - XFS was introduced in R80.20. The caveat was that if you were doing upgrades from R80.10 or earlier version using CPUSE, it would keep existing EXT3 file system and you would miss out on apparently faster XFS. The only way to change the file system to XFS would be by full fresh install from external disk, which for many of us meant a hard choice of being forced to rely on CPUSE upgrade and keeping EXT3 as boxes were in remote locations and cost of the travel and time of full box reinstall was too high.

I had a bit of lab time today and decided to test the difference between two file systems as we have many 5900 clusters that were introduced back in 2017 with R80.10 onboard = EXT3 file system. As mentioned above, we decided to perform upgrades to R80.30 and R80.40 using CPUSE, thus keeping original file system.

And the main reason for the test actually was the fact that snapshot creation on EXT3 boxes puts insane strain on CPU usage - they almost become unmanageable with multiple CPU cores running zero CPU idle time.

Results actually exceeded my expectations. In case you wanted to know 🙂

Test	XFS		EXT3
Snapshot creation	3 mins	Minimal CPU impact	15 mins	System becomes nearly unusable
HFA install	17 mins		21 mins	Fair impact on CPU
Reboot time	4 mins		6 mins
Write speed large file	100MB/s		27MB/s
Write speed small files	0.6-1.6GB/s	Varied a lot!	0.9GB/s	Fairly consistent

Just to add if you wanted to know how to check it:

[Daniel_]

Creation snapshot with BTRFS: less then a second 😏

[genisis__]

This is awesome information!

Are these results from a gateway or a Manager?  Everything I know about the XFS filesystem suggested there was no real benefit to do a clean build on a gateway, but certainly was worth doing it at the management layer, especially on a logging server.

[Kaspars_Zibarts]

Sorry - I should have been more clear - it's definitely gateways, not management. For management there was no question from the start - XFS was a MUST. And gateways I was doubtful as all they do is write a bit of local logging, but I was really wrong - there are very good reasons to update file system even on gateways, like taking snapshots during daytime which is impossible with EXT3 as it will affect traffic. And faster boot times. And faster HFA installations. For me it's all valid operational reasons.

[genisis__]

Awesome!

I think CP should be adding these figures into there documentation! 

Gives technical resources real figures to justify clean installation to the business.

[PhoneBoy]

Just to clarify, XFS was available on anything that has a Linux 3.10 kernel.
That meant initially on the management in R80.20 but didn’t become mainstream on gateways until R80.40.
And yes, it requires a fresh install from ISO to convert to the new filesystem since we use a different partitioning scheme as well.

In any cases this is exceptionally useful information you’ve provided here.

[Timothy_Hall]

Wow, what a difference!  Thanks for the post.

[Kaspars_Zibarts]

I know! Totally changed my mind about having XFS on gateways! 🙂 will be making plan of upgrading them all! 🙂

[HristoGrigorov]

XFS was developed by SGI to run on their Cray mainframes and later "donated" to Linux community.  As such it is targeted to work best with large files. On the down side it uses aggressive caching and in general eats a bit more memory. I was a great fan of it until BTRFS appeared. I wonder if CheckPoint is going to stick with XFS in the future or it is planning to support other file systems as well... 

[Bob_Zimmerman]

Eh. BTRFS's desirable features are way too immature for my taste. Needs another few years before I'll trust it on my management servers, let alone my firewalls. On the plus side, offering it would mean moving away from RHEL (which removed all support in RHEL 8), which could delay the systemd nonsense. I am absolutely never allowing systemd anywhere near my firewalls.

ZFS, on the other hand ...

draid2:1s means never having to say you're sorry.