Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
Mangesh
Participant

Unable to export 10000+ records in SmartView in R81

Dear team, 

We have recently upgraded our management server into R81 and trying to export 1M logs from web , however it limits the logs from web 10k entries only, don't know why it is happening.

Please let us know any changes needs to be done.

 

Regards,

Mangesh Jadhav

28 Replies
_Val_
Admin
Admin

I believe it is by design. To export logs, you can also use a legacy SmartView Tracker client (silently installed with SmartConsole) or "fwm logexport" from CLI.

If you want to send logs to a SIEM, look into sk122323

0 Kudos
Mangesh
Participant

Thanks for your reply.

 

We tried both web as well as client and in which getting 10k records only.

Please let us know is there any registry changes something like that.

 

Regards,

Mangesh Jadhav

0 Kudos
_Val_
Admin
Admin

As I said, you cannot override this limitation with SmartView. Use other means, as advised above, such as SmartViewTracker or CLI tool

 

0 Kudos
Sony_James
Participant

Wasn't SmartView limitation at 1M and not at 10K?

0 Kudos
_Val_
Admin
Admin

Showing the records, yes. Different limit for exporting those, AFAIK

0 Kudos
lbalogh
Participant

Hello, What exactly you mean under CLI tool? 

Is there any tool that can aggregate the fw logs together so it don't have to be opened one by one ?
Thanks!

0 Kudos
Tomer_Noy
Employee
Employee

You should be able to export up to 1M logs in SmartView.

I suggest checking 2 things:

1) Check the date filter in the view that you are exporting from. It's possible that it's set to 24 hours and those are the logs you are getting.

2) If you are trying to export shortly after your upgrade to R81, it's possible that older logs aren't indexed yet. R81 replaced the indexing engine, so we need to reindex in the background (last 24h by default) and it takes some time.
Check if there are logs that you can see (by querying them) that do not appear in the export.

Please let us know if it was one of these issues.

0 Kudos
Mangesh
Participant

Dear team,

 

Thanks for your reply,

 

this is also not working.

 

Any other suggestions????

0 Kudos
Paul_Hagyard
Collaborator

Log export has been nearly consistently broken since R80 landed. Under R7x we could reliably export from the main GUI, then it was moved to SmartView and we've logged support calls for basically every version since, often for multiple customers. Continues to be unreliable. In my home lab I can't even get an export to CSV working today for *1000* records today (it worked a few days ago when I couldn't get 10k to export). R81 jumbo take 42 (I hoped that would help). CPU is doing nothing, plenty of free memory, underlying disk storage is M.2 SSD.

Log export to CSV is one of the main things I use for quick analysis of issues. Maybe it can be fixed properly for R82?

D_TK
Collaborator

R81 JHFA 42

Log server has been upgraded for about a month (from 80.40)

Using web smartview \ export will create a .csv with only 10000 rows.  In my environment, 10000 rows is roughly 2 minutes of logs.

This really needs to get fixed.

the_rock
Champion
Champion

Agree 100%. 

0 Kudos
the_rock
Champion
Champion

Definitely thats a limitation, I seen it in previous versions too. TAC gave me below sk, but I tried in R81.10 and cant do more than a 1000, so Im pretty sure article is wrong.

https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...

 

0 Kudos
Paul_Hagyard
Collaborator

I've definitely had 1M record exports work in R80.10-R80.40 in various environments, just not reliably.

0 Kudos
Tomer_Noy
Employee
Employee

I see that this is impacting multiple people, so there might be something here that I missed.

I will investigate this more deeply with my team and come back to you with a better understanding and hopefully a quick solution.

_Val_
Admin
Admin

Thanks @Tomer_Noy, the community appreciates that.

0 Kudos
Mangesh
Participant

Dear Tomer_Noy,

Please let us know if you have an update for this.

Regards,

Mangesh Jadhav

0 Kudos
Tomer_Noy
Employee
Employee

We doublechecked and did some additional testing and were not able to reproduce the issue with R81 or R81.10. We need you assistance with more information to pinpoint the issue and understand the specific conditions in which this isn't working.

First, it would help if you install the latest recommended JHF on the Management server (and log server if they are separate). That way we won't troubleshoot old issues that might have been resolved.

If you still have the issue (on R81 / R81.10) and cannot export more than 10K logs, please open a TAC ticket. They will work with you to gather the information and we will be able to further the investigation.

Once we have a solution, I'll also update back here on the post.

0 Kudos
Paul_Hagyard
Collaborator

Hi Tomer,

Home lab environment with no support here - so can't raise a SR. I've done some more digging (R81 jumbo take 42) and while SmartView does not show the export as completing in the Archive tag, a CSV file does get written under /opt/CPrt-R81/smartview/exported_files/ - but the files never exceed 5,000 lines (this is when selecting 1M records). No obvious failures in a cpm_debug (Search crud Solr), but would need TAC involved for more guidance as you said. Hopefully someone with a supported environment can get a SR raised.

Cheers,

Paul

0 Kudos
Paul_Hagyard
Collaborator

Further to this, the CSV file created is still being held open by smartview-jetty:

/opt/CPshrd-R81/jre_64/bin/java <lots of options> -jar start.jar OPTIONS=Server,resources,websocket /opt/CPrt-R81/conf/smartview-jetty.xml /opt/CPrt-R81/conf/smartview-service-jetty.xml

However, no more data is written.

This is the same behaviour I see with the log API. I can dump logs with an initial query, but going back for more with the existing query ID gives a hung session with no more data.

0 Kudos
Miri_Ofir
Employee
Employee

Update here following further investigation.

We see that in some cases export to CSV from SmartView might fail also in R81, mainly when exporting huge amount of records (1 million), we will release a fix in the next JHF. If you need urgently, please approach TAC.

For all home lab users, feel free to contact me privately and I will share the fix.

Paul_Hagyard
Collaborator

Hi Miri,

Thanks. Do you have an issue ID number or similar that we can reference when talking to TAC? I've just built a new R81 SmartEvent/SmartLog host (jumbo take 44) for a customer and export to CSV is only giving 10k (exactly) lines for each export.

Cheers,

Paul

0 Kudos
Miri_Ofir
Employee
Employee

Sure, when you contact support, refer to Issue ID PRJ-30722 from  sk175545

0 Kudos
D_TK
Collaborator

exporting from smartview seems to be the only option for me.  when i try to export directly form r81 smartconsole, i receive this message:

0 Kudos
Miri_Ofir
Employee
Employee

Right, in R81 this is the only option, please get the hotfix from TAC

0 Kudos
Bonnie_Self
Participant

Does the R81 hotfix allow export from console?  I can see my historical logs within console, but they do not show in SmartView.  If I could export them from there I could at least have something.  Case opened for almost a week now.  Days to index set to 31 but SmartView only shows current and previous day logs, but will show statistics on the side for them.  Exporting logs is an important feature and all of the engineers that have had my case so far say 'we don't know why you can't export from console.  Console has always worked.'

0 Kudos
Miri_Ofir
Employee
Employee

In R81 we support export to CSV of 1 million records from SmartView web only and not SmartConsole, and all logs should be available not only current and previous day. 

I suggest to verify the time frame filter, because we recently changed the default time frame filter to last 24 hours to improve the performance of the system.

If everything is configured properly and still you don't see logs of 31 days, please share with me the SR number and I will work internally to ensure you have the full info from SmartView

0 Kudos
Bonnie_Self
Participant

SR#6-0003092442

Still puzzling that each engineer on my case has said that it should work from SmartConsole.  Apparently they are not trained that this changed.

Also, as I understand it, SmartView can only show logs that are indexed, but sometimes I have a need to pull logs from several months ago and I don't see a way to open the older flat log files there.  This feels like a step in the negative direction.

0 Kudos
Miri_Ofir
Employee
Employee

Hi Bonnie, 

Stacy also shared with me this case.

I discussed it with the case owner, he will have a session with you today, as far as I know, and I am sure there will be a progress. I am monitoring it with him.

0 Kudos