Fastest way to export to CSV

[Hugo_vd_Kooij]

HI,

I need to export 2 months of logging into csv format. Following https://support.checkpoint.com/results/sk/sk118521 I added the -n option. But an export of a 2 GB log file takes about 3 to 4 hours. And we got about 5 GB per day worth of logging.

Are there any other option I need to add to speed up the process?

My current syntax is 

fw log -n ${DOMAINLOGDIR}/${LOGFILE} > ${EXPORTDIR}/${LOGFILE}.txt

[Tal_Paz-Fridman]

Have you considered using the SmartView Web Application? it allows exporting up to one million logs. 

Perhaps you can filter out certain data and reduce the size and time it takes.

https://sc1.checkpoint.com/documents/R81.20/WebAdminGuides/EN/CP_R81.20_LoggingAndMonitoring_AdminGu...

 (exists in prior versions as well)

[the_rock]

That is true, BUT, worth mentioning this does not work if using S1C instance, its limited to 1k logs.

Andy

[Hugo_vd_Kooij]

Randsomware investigation which means they want ALL of it over the 2 months. And it's about 5 GB per day. So any filtering is out of the question.

[Joseph_Audet]

Tal's answer is your best path forward. I worked with a customer to do this in bulk once, and because CP stores the logs in an efficient binary format, when you decompress them to a CSV file they lose that efficiency. I was seeing up to a 5x expansion. It also takes a lot of time. We had to export 100GB of logs and ran a script to let it run, took awhile.

[Lloyd_Braun]

-p should speed it up, -n is the big one and you already have that.

from docs:

-p

Specifies to not to perform resolution of the port numbers in the log file (this is the default behavior).

This significantly speeds up the log processing.