Disk Recommendations for Logging

[RichUK]

Hi all,

We have a R81.10 management server and smartevent server that are both used for logging. They are virtual machines running on a dedicated host each. Both the management and smartevent VM guests are configured with two drives. I don't know why as the previous admin has left. It seems the /dev/mapper/vg_splat-lv_log folder was split across two drives with LVM. Last week our management server crashed and was rebuilt on a temporary esx host. I'm now looking to rebuild the original management server and don't know how to approach the drives configuration.

Would they have been split for a performance reason, is it best to have a different drive for the logging or do I reconfigure the guest to have one large drive?

I also don't know why both the management and smartevent are logging the same logs. The smartevent server is a higher spec host than the management and I'm wonder if it just best for the smartevent to do the logging.

 

Thanks in advance

Rich

[Chris_Atkinson]

sk94671 may help to explain the split if there were space issues historically.

Are logs being sent simultaneously to both, or only to the second in the event of a failure?

[Ruan_Kotze]

A lot of that would depend on the configuration of the underlying storage.  If both VMDKs sit on the same set of disks, then all read / writes will hit the same disks, so no improvement.

Conversely, if they're on a different set of spindles (or SSD's for that matter) then I can see potential for improvement.

[HeikoAnkenbrand]

Hi @RichUK,

Point 1)
I think that the /var/log/ partition was enlarged because the disk space was not enough in the past. From a performance point of view, it makes no sense to split the hard disks. The only thing that matters here is the performance of the storage system behind VMWare. I would take a look if you see I/O waits on the mouted hard drives. This could indicate performance problems with slow VMWare disks. 

You can view iowait with the following command:
# watch "iostat"

Point 2)
Separating the smart event and log server is a design and performance question. If you send the logs to two servers, you put more load on the gateway fwd and thus also on the network. But by doing this, you reduce the load on the log server and smart event server itself. It is important how busy the individual servers and gateways here!

[RichUK]

Hi all \ @HeikoAnkenbrand 

Speaking to a colleague, it seems the management server and the smart event server were built on VMWare because at the time the HP DL360 server was incompatible with Gaia. I still can't understand the reason to split the virtual disks as it was part of a RAID 5 set.

We have now rebuilt the management server directly to the server, so no ESX. At the time of the installation, we increased the log location to 5TB.

From a logging point of view, I would rather reduce load off the gateways and only send logs to one logging server. Both the management and smart event server have 5TB RAID sets for logs, therefore I guess I will have to pick one for the primary and use the other as a backup.

Thanks for your help

[genisis__]

I have a similar scenario where the previous admin for a logging server setup /var/log with just 40GB, being used for over 12 firewalls.  

In my case the log server is running R81 (upgraded from a previous version).

I've recommended snapshotting the VM, build a new VM with over 1TB disk running on highspeed storage.  That way I get the performance improvement of XFS and logging space which makes more sense.

I could of course opt to add more diskspace by adding another vdisk, but as you have rightly said, I see no performance gain from this and no XFS.