Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
Hugo_vd_Kooij
Advisor

Fastest way to export to CSV

HI,

I need to export 2 months of logging into csv format. Following https://support.checkpoint.com/results/sk/sk118521 I added the -n option. But an export of a 2 GB log file takes about 3 to 4 hours. And we got about 5 GB per day worth of logging.

Are there any other option I need to add to speed up the process?

My current syntax is 

fw log -n ${DOMAINLOGDIR}/${LOGFILE} > ${EXPORTDIR}/${LOGFILE}.txt

 

<< We make miracles happen while you wait. The impossible jobs take just a wee bit longer. >>
0 Kudos
5 Replies
Tal_Paz-Fridman
Employee
Employee

Have you considered using the SmartView Web Application? it allows exporting up to one million logs. 

Perhaps you can filter out certain data and reduce the size and time it takes.

https://sc1.checkpoint.com/documents/R81.20/WebAdminGuides/EN/CP_R81.20_LoggingAndMonitoring_AdminGu...

 (exists in prior versions as well)

the_rock
Legend
Legend

That is true, BUT, worth mentioning this does not work if using S1C instance, its limited to 1k logs.

Andy

0 Kudos
Hugo_vd_Kooij
Advisor

Randsomware investigation which means they want ALL of it over the 2 months. And it's about 5 GB per day. So any filtering is out of the question.

<< We make miracles happen while you wait. The impossible jobs take just a wee bit longer. >>
0 Kudos
Joseph_Audet
Ambassador
Ambassador

Tal's answer is your best path forward. I worked with a customer to do this in bulk once, and because CP stores the logs in an efficient binary format, when you decompress them to a CSV file they lose that efficiency. I was seeing up to a 5x expansion. It also takes a lot of time. We had to export 100GB of logs and ran a script to let it run, took awhile.

0 Kudos
Lloyd_Braun
Collaborator

-p should speed it up, -n is the big one and you already have that.

 

from docs:

-p

Specifies to not to perform resolution of the port numbers in the log file (this is the default behavior).

This significantly speeds up the log processing.

Leaderboard

Epsum factorial non deposit quid pro quo hic escorol.

Upcoming Events

    CheckMates Events