Exporting Logs

[StevePearson]

I've had a lt of problems this week exporting logs to CSV, so I wanted to see if anyone has any suggestions as to what i'm doing wrong!

The first request was for all logs relating to a single user logging on/off the VPN going back as far as possible. I wrote a quick query to check I was getting the correct logs, which I was, then applied a date range. The first issue was it would only show me logs in the last 3 months, so I checked the log config and found it saves logs for 365 days but only saves indexes for 90 days. So, accepting that, I adjust the time frame to 90 days relative to today and get the logs. Then I ask it to export them to CSV, and I waited, and waited, and waited. After 20 minutes I gave up waiting, assuming it's gone wrong, and tried again but got an error saying a problem with the query and nothing else then worked on the logging side. I restarted the EV system and tried again, this time I got called away to do something else and came back a couple of hours later to find the message saying it was available to download!

So it's taking a significant amount of time to do what I would think is a relatively simple export. (management server is a VM with 8 cores and 16Gb Ram)

If anyone has a better way to get this info in a report i'd be very interested.

Now today, different customer, different criteria, but similar issue!

This time I want all logs for a 1 hour time period, simples, but it took nearly 20 minutes to create the export!

Is this right and to be expected, or am I missing something?

Any pointers greatly apriciated!

[the_rock]

I never had this issue when doing it from smartview, as you cant do csv export from smart console. Limit is 10000 lines, but not sure if maybe in R82 it will be million, no idea.

Andy

https://sc1.checkpoint.com/documents/Infinity_Portal/WebAdminGuides/EN/Check-Point-SmartCloud-Admin-...

[StevePearson]

Hi Andy,

I am doing it from Smartview, I have just found the file I exported earlier has 1 million lines! (R81.20). Filtered out the accept now, so just the drops, been waiting 9 minutes so far!

[the_rock]

That does not shock me at all...it may take some time. Never tried it in R81.20, but let me fire up my lab and test 🙂

Andy

[the_rock]

Just for the context, this is how you can tell if its done, without constantly checking. I also verified the file, shows exactly 1M logs, I guess thats MAXIMUM.

Andy

 

[StevePearson]

That's really useful to know thanks!

Looks like you can download from there too! I'll have a look at that in the morning

[the_rock]

yes sir! My colleague showed me that, I did not have any idea about it either...learn something new every day. After all, thats the life goal, hehe : - )

Cheers,

Andy

[the_rock]

Excample from my lab, for context. If you unzip the file, csv is actually about 600MBs and you can open it, look through it, NOTHING secretive, just my lab, so no one cares haha

Andy

[Lesley]

Do you use Smartevent server? This is a great way to make reports. Default there are already some reports setup in the default installation. You can also modify them a bit to request the needs you have. Or check here: https://community.checkpoint.com/t5/SmartEvent/bd-p/SmartEvent some ready to go templates that you can import and change if you like. I think the default one is called: Endpoint Security VPN Users Activity

Regarding the CSV, only smartview web can export to CSV. If the file is really big it will get stuck. I would limit the amount of lines and do not export colums without any data(this is an option). Normally exporting to CSV should not give any issues. So worth investigating, maybe bad log or performance issues. 

[AkosBakos]

Hi @StevePearson

My experience is that, mainly in large environment: If I have logging issues the first debug step is #evstop #evstart, on the MGMT or LOG server. Its restart the SOLR. This can  be really helpful in some cases.

The SmartView export is limitied to 1M records only since R80.10.

For housekeeping I suggest you to run the cpm_doctor script on your MGMT or LOG server, maybe it will find some interesting thing

/opt/CPsuite-R80.40/fw1/log/cpm_doctor (change the version to MGMT version)

I didn't remember clearly, when we had issues with SmartView, there was problem with the HEAP_size.

The solution was to increase the memory onf the VM.

This is a guide for the sizing but it is an internal SK. You can open a ticket at the TAC and thel will help you about the sizing:

https://wiki.checkpoint.com/confluence/display/CPPublic/Smartlog+and+Smartevent+-+Sizing+and+Perform... 

According the SMART-1 datasheet: https://www.checkpoint.com/downloads/products/smart-1-security-management-platform-datasheet.pdf

For 6 cores belong to at least 32 GB (this is only a approximate approach.

Cheers

Akos

[the_rock]

Thats very good point, and plus, really even rebooting the mgmt server does not cause any issues either, so can be done pretty much any time.

Andy