Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
Mangesh
Participant

Unable to export 10000+ records in SmartView in R81

Dear team, 

We have recently upgraded our management server into R81 and trying to export 1M logs from web , however it limits the logs from web 10k entries only, don't know why it is happening.

Please let us know any changes needs to be done.

 

Regards,

Mangesh Jadhav

19 Replies
_Val_
Admin
Admin

I believe it is by design. To export logs, you can also use a legacy SmartView Tracker client (silently installed with SmartConsole) or "fwm logexport" from CLI.

If you want to send logs to a SIEM, look into sk122323

0 Kudos
Mangesh
Participant

Thanks for your reply.

 

We tried both web as well as client and in which getting 10k records only.

Please let us know is there any registry changes something like that.

 

Regards,

Mangesh Jadhav

0 Kudos
_Val_
Admin
Admin

As I said, you cannot override this limitation with SmartView. Use other means, as advised above, such as SmartViewTracker or CLI tool

 

0 Kudos
Sony_James
Participant

Wasn't SmartView limitation at 1M and not at 10K?

0 Kudos
_Val_
Admin
Admin

Showing the records, yes. Different limit for exporting those, AFAIK

0 Kudos
lbalogh
Explorer

Hello, What exactly you mean under CLI tool? 

Is there any tool that can aggregate the fw logs together so it don't have to be opened one by one ?
Thanks!

0 Kudos
Tomer_Noy
Employee
Employee

You should be able to export up to 1M logs in SmartView.

I suggest checking 2 things:

1) Check the date filter in the view that you are exporting from. It's possible that it's set to 24 hours and those are the logs you are getting.

2) If you are trying to export shortly after your upgrade to R81, it's possible that older logs aren't indexed yet. R81 replaced the indexing engine, so we need to reindex in the background (last 24h by default) and it takes some time.
Check if there are logs that you can see (by querying them) that do not appear in the export.

Please let us know if it was one of these issues.

0 Kudos
Mangesh
Participant

Dear team,

 

Thanks for your reply,

 

this is also not working.

 

Any other suggestions????

0 Kudos
Paul_Hagyard
Collaborator

Log export has been nearly consistently broken since R80 landed. Under R7x we could reliably export from the main GUI, then it was moved to SmartView and we've logged support calls for basically every version since, often for multiple customers. Continues to be unreliable. In my home lab I can't even get an export to CSV working today for *1000* records today (it worked a few days ago when I couldn't get 10k to export). R81 jumbo take 42 (I hoped that would help). CPU is doing nothing, plenty of free memory, underlying disk storage is M.2 SSD.

Log export to CSV is one of the main things I use for quick analysis of issues. Maybe it can be fixed properly for R82?

D_TK
Contributor

R81 JHFA 42

Log server has been upgraded for about a month (from 80.40)

Using web smartview \ export will create a .csv with only 10000 rows.  In my environment, 10000 rows is roughly 2 minutes of logs.

This really needs to get fixed.

the_rock
Authority
Authority

Agree 100%. 

0 Kudos
the_rock
Authority
Authority

Definitely thats a limitation, I seen it in previous versions too. TAC gave me below sk, but I tried in R81.10 and cant do more than a 1000, so Im pretty sure article is wrong.

https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...

 

0 Kudos
Paul_Hagyard
Collaborator

I've definitely had 1M record exports work in R80.10-R80.40 in various environments, just not reliably.

0 Kudos
Tomer_Noy
Employee
Employee

I see that this is impacting multiple people, so there might be something here that I missed.

I will investigate this more deeply with my team and come back to you with a better understanding and hopefully a quick solution.

_Val_
Admin
Admin

Thanks @Tomer_Noy, the community appreciates that.

0 Kudos
Mangesh
Participant

Dear Tomer_Noy,

Please let us know if you have an update for this.

Regards,

Mangesh Jadhav

0 Kudos
Tomer_Noy
Employee
Employee

We doublechecked and did some additional testing and were not able to reproduce the issue with R81 or R81.10. We need you assistance with more information to pinpoint the issue and understand the specific conditions in which this isn't working.

First, it would help if you install the latest recommended JHF on the Management server (and log server if they are separate). That way we won't troubleshoot old issues that might have been resolved.

If you still have the issue (on R81 / R81.10) and cannot export more than 10K logs, please open a TAC ticket. They will work with you to gather the information and we will be able to further the investigation.

Once we have a solution, I'll also update back here on the post.

0 Kudos
Paul_Hagyard
Collaborator

Hi Tomer,

Home lab environment with no support here - so can't raise a SR. I've done some more digging (R81 jumbo take 42) and while SmartView does not show the export as completing in the Archive tag, a CSV file does get written under /opt/CPrt-R81/smartview/exported_files/ - but the files never exceed 5,000 lines (this is when selecting 1M records). No obvious failures in a cpm_debug (Search crud Solr), but would need TAC involved for more guidance as you said. Hopefully someone with a supported environment can get a SR raised.

Cheers,

Paul

0 Kudos
Paul_Hagyard
Collaborator

Further to this, the CSV file created is still being held open by smartview-jetty:

/opt/CPshrd-R81/jre_64/bin/java <lots of options> -jar start.jar OPTIONS=Server,resources,websocket /opt/CPrt-R81/conf/smartview-jetty.xml /opt/CPrt-R81/conf/smartview-service-jetty.xml

However, no more data is written.

This is the same behaviour I see with the log API. I can dump logs with an initial query, but going back for more with the existing query ID gives a hung session with no more data.

0 Kudos