Unable to export 10000+ records in SmartView in R81

[Mangesh]

Dear team, 

We have recently upgraded our management server into R81 and trying to export 1M logs from web , however it limits the logs from web 10k entries only, don't know why it is happening.

Please let us know any changes needs to be done.

Regards,

Mangesh Jadhav

[_Val_]

I believe it is by design. To export logs, you can also use a legacy SmartView Tracker client (silently installed with SmartConsole) or "fwm logexport" from CLI.

If you want to send logs to a SIEM, look into sk122323

[Mangesh]

Thanks for your reply.

We tried both web as well as client and in which getting 10k records only.

Please let us know is there any registry changes something like that.

Regards,

Mangesh Jadhav

[_Val_]

As I said, you cannot override this limitation with SmartView. Use other means, as advised above, such as SmartViewTracker or CLI tool

[Sony_James]

Wasn't SmartView limitation at 1M and not at 10K?

[_Val_]

Showing the records, yes. Different limit for exporting those, AFAIK

[lbalogh]

Hello, What exactly you mean under CLI tool? 

Is there any tool that can aggregate the fw logs together so it don't have to be opened one by one ?
Thanks!

[Tomer_Noy]

You should be able to export up to 1M logs in SmartView.

I suggest checking 2 things:

1) Check the date filter in the view that you are exporting from. It's possible that it's set to 24 hours and those are the logs you are getting.

2) If you are trying to export shortly after your upgrade to R81, it's possible that older logs aren't indexed yet. R81 replaced the indexing engine, so we need to reindex in the background (last 24h by default) and it takes some time.
Check if there are logs that you can see (by querying them) that do not appear in the export.

Please let us know if it was one of these issues.

[Mangesh]

Dear team,

Thanks for your reply,

this is also not working.

Any other suggestions????

[Paul_Hagyard]

Log export has been nearly consistently broken since R80 landed. Under R7x we could reliably export from the main GUI, then it was moved to SmartView and we've logged support calls for basically every version since, often for multiple customers. Continues to be unreliable. In my home lab I can't even get an export to CSV working today for *1000* records today (it worked a few days ago when I couldn't get 10k to export). R81 jumbo take 42 (I hoped that would help). CPU is doing nothing, plenty of free memory, underlying disk storage is M.2 SSD.

Log export to CSV is one of the main things I use for quick analysis of issues. Maybe it can be fixed properly for R82?

[D_TK]

R81 JHFA 42

Log server has been upgraded for about a month (from 80.40)

Using web smartview \ export will create a .csv with only 10000 rows.  In my environment, 10000 rows is roughly 2 minutes of logs.

This really needs to get fixed.

[the_rock]

Agree 100%. 

[the_rock]

Definitely thats a limitation, I seen it in previous versions too. TAC gave me below sk, but I tried in R81.10 and cant do more than a 1000, so Im pretty sure article is wrong.

https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...

[Paul_Hagyard]

I've definitely had 1M record exports work in R80.10-R80.40 in various environments, just not reliably.

[Tomer_Noy]

I see that this is impacting multiple people, so there might be something here that I missed.

I will investigate this more deeply with my team and come back to you with a better understanding and hopefully a quick solution.

[_Val_]

Thanks @Tomer_Noy, the community appreciates that.

[Mangesh]

Dear Tomer_Noy,

Please let us know if you have an update for this.

Regards,

Mangesh Jadhav

[Tomer_Noy]

We doublechecked and did some additional testing and were not able to reproduce the issue with R81 or R81.10. We need you assistance with more information to pinpoint the issue and understand the specific conditions in which this isn't working.

First, it would help if you install the latest recommended JHF on the Management server (and log server if they are separate). That way we won't troubleshoot old issues that might have been resolved.

If you still have the issue (on R81 / R81.10) and cannot export more than 10K logs, please open a TAC ticket. They will work with you to gather the information and we will be able to further the investigation.

Once we have a solution, I'll also update back here on the post.

[Paul_Hagyard]

Hi Tomer,

Home lab environment with no support here - so can't raise a SR. I've done some more digging (R81 jumbo take 42) and while SmartView does not show the export as completing in the Archive tag, a CSV file does get written under /opt/CPrt-R81/smartview/exported_files/ - but the files never exceed 5,000 lines (this is when selecting 1M records). No obvious failures in a cpm_debug (Search crud Solr), but would need TAC involved for more guidance as you said. Hopefully someone with a supported environment can get a SR raised.

Cheers,

Paul

[Paul_Hagyard]

Further to this, the CSV file created is still being held open by smartview-jetty:

/opt/CPshrd-R81/jre_64/bin/java <lots of options> -jar start.jar OPTIONS=Server,resources,websocket /opt/CPrt-R81/conf/smartview-jetty.xml /opt/CPrt-R81/conf/smartview-service-jetty.xml

However, no more data is written.

This is the same behaviour I see with the log API. I can dump logs with an initial query, but going back for more with the existing query ID gives a hung session with no more data.

[Miri_Ofir]

Update here following further investigation.

We see that in some cases export to CSV from SmartView might fail also in R81, mainly when exporting huge amount of records (1 million), we will release a fix in the next JHF. If you need urgently, please approach TAC.

For all home lab users, feel free to contact me privately and I will share the fix.

[Paul_Hagyard]

Hi Miri,

Thanks. Do you have an issue ID number or similar that we can reference when talking to TAC? I've just built a new R81 SmartEvent/SmartLog host (jumbo take 44) for a customer and export to CSV is only giving 10k (exactly) lines for each export.

Cheers,

Paul

[Miri_Ofir]

Sure, when you contact support, refer to Issue ID PRJ-30722 from  sk175545

[D_TK]

exporting from smartview seems to be the only option for me.  when i try to export directly form r81 smartconsole, i receive this message:

[Miri_Ofir]

Right, in R81 this is the only option, please get the hotfix from TAC

[Bonnie_Self]

Does the R81 hotfix allow export from console?  I can see my historical logs within console, but they do not show in SmartView.  If I could export them from there I could at least have something.  Case opened for almost a week now.  Days to index set to 31 but SmartView only shows current and previous day logs, but will show statistics on the side for them.  Exporting logs is an important feature and all of the engineers that have had my case so far say 'we don't know why you can't export from console.  Console has always worked.'

[Miri_Ofir]

In R81 we support export to CSV of 1 million records from SmartView web only and not SmartConsole, and all logs should be available not only current and previous day. 

I suggest to verify the time frame filter, because we recently changed the default time frame filter to last 24 hours to improve the performance of the system.

If everything is configured properly and still you don't see logs of 31 days, please share with me the SR number and I will work internally to ensure you have the full info from SmartView

[Bonnie_Self]

SR#6-0003092442

Still puzzling that each engineer on my case has said that it should work from SmartConsole.  Apparently they are not trained that this changed.

Also, as I understand it, SmartView can only show logs that are indexed, but sometimes I have a need to pull logs from several months ago and I don't see a way to open the older flat log files there.  This feels like a step in the negative direction.

[Miri_Ofir]

Hi Bonnie, 

Stacy also shared with me this case.

I discussed it with the case owner, he will have a session with you today, as far as I know, and I am sure there will be a progress. I am monitoring it with him.

[Shiran_wang]

Hi All,

   please try down small file to test less than 100 line is better,  if Management server running many year it cause archive file more than 10000 line. please refer this sk179795,  you running attachment to test SMS log function and get report.