- CheckMates
- :
- Products
- :
- Quantum
- :
- Management
- :
- Re: Unable to export 10000+ records in SmartView i...
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Are you a member of CheckMates?
×- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Unable to export 10000+ records in SmartView in R81
Dear team,
We have recently upgraded our management server into R81 and trying to export 1M logs from web , however it limits the logs from web 10k entries only, don't know why it is happening.
Please let us know any changes needs to be done.
Regards,
Mangesh Jadhav
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I believe it is by design. To export logs, you can also use a legacy SmartView Tracker client (silently installed with SmartConsole) or "fwm logexport" from CLI.
If you want to send logs to a SIEM, look into sk122323
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks for your reply.
We tried both web as well as client and in which getting 10k records only.
Please let us know is there any registry changes something like that.
Regards,
Mangesh Jadhav
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
As I said, you cannot override this limitation with SmartView. Use other means, as advised above, such as SmartViewTracker or CLI tool
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Wasn't SmartView limitation at 1M and not at 10K?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Showing the records, yes. Different limit for exporting those, AFAIK
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello, What exactly you mean under CLI tool?
Is there any tool that can aggregate the fw logs together so it don't have to be opened one by one ?
Thanks!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
You should be able to export up to 1M logs in SmartView.
I suggest checking 2 things:
1) Check the date filter in the view that you are exporting from. It's possible that it's set to 24 hours and those are the logs you are getting.
2) If you are trying to export shortly after your upgrade to R81, it's possible that older logs aren't indexed yet. R81 replaced the indexing engine, so we need to reindex in the background (last 24h by default) and it takes some time.
Check if there are logs that you can see (by querying them) that do not appear in the export.
Please let us know if it was one of these issues.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Dear team,
Thanks for your reply,
this is also not working.
Any other suggestions????
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Log export has been nearly consistently broken since R80 landed. Under R7x we could reliably export from the main GUI, then it was moved to SmartView and we've logged support calls for basically every version since, often for multiple customers. Continues to be unreliable. In my home lab I can't even get an export to CSV working today for *1000* records today (it worked a few days ago when I couldn't get 10k to export). R81 jumbo take 42 (I hoped that would help). CPU is doing nothing, plenty of free memory, underlying disk storage is M.2 SSD.
Log export to CSV is one of the main things I use for quick analysis of issues. Maybe it can be fixed properly for R82?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
R81 JHFA 42
Log server has been upgraded for about a month (from 80.40)
Using web smartview \ export will create a .csv with only 10000 rows. In my environment, 10000 rows is roughly 2 minutes of logs.
This really needs to get fixed.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Agree 100%.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Definitely thats a limitation, I seen it in previous versions too. TAC gave me below sk, but I tried in R81.10 and cant do more than a 1000, so Im pretty sure article is wrong.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I've definitely had 1M record exports work in R80.10-R80.40 in various environments, just not reliably.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I see that this is impacting multiple people, so there might be something here that I missed.
I will investigate this more deeply with my team and come back to you with a better understanding and hopefully a quick solution.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks @Tomer_Noy, the community appreciates that.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Dear Tomer_Noy,
Please let us know if you have an update for this.
Regards,
Mangesh Jadhav
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
We doublechecked and did some additional testing and were not able to reproduce the issue with R81 or R81.10. We need you assistance with more information to pinpoint the issue and understand the specific conditions in which this isn't working.
First, it would help if you install the latest recommended JHF on the Management server (and log server if they are separate). That way we won't troubleshoot old issues that might have been resolved.
If you still have the issue (on R81 / R81.10) and cannot export more than 10K logs, please open a TAC ticket. They will work with you to gather the information and we will be able to further the investigation.
Once we have a solution, I'll also update back here on the post.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Tomer,
Home lab environment with no support here - so can't raise a SR. I've done some more digging (R81 jumbo take 42) and while SmartView does not show the export as completing in the Archive tag, a CSV file does get written under /opt/CPrt-R81/smartview/exported_files/ - but the files never exceed 5,000 lines (this is when selecting 1M records). No obvious failures in a cpm_debug (Search crud Solr), but would need TAC involved for more guidance as you said. Hopefully someone with a supported environment can get a SR raised.
Cheers,
Paul
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Further to this, the CSV file created is still being held open by smartview-jetty:
/opt/CPshrd-R81/jre_64/bin/java <lots of options> -jar start.jar OPTIONS=Server,resources,websocket /opt/CPrt-R81/conf/smartview-jetty.xml /opt/CPrt-R81/conf/smartview-service-jetty.xml
However, no more data is written.
This is the same behaviour I see with the log API. I can dump logs with an initial query, but going back for more with the existing query ID gives a hung session with no more data.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Update here following further investigation.
We see that in some cases export to CSV from SmartView might fail also in R81, mainly when exporting huge amount of records (1 million), we will release a fix in the next JHF. If you need urgently, please approach TAC.
For all home lab users, feel free to contact me privately and I will share the fix.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Miri,
Thanks. Do you have an issue ID number or similar that we can reference when talking to TAC? I've just built a new R81 SmartEvent/SmartLog host (jumbo take 44) for a customer and export to CSV is only giving 10k (exactly) lines for each export.
Cheers,
Paul
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Sure, when you contact support, refer to Issue ID PRJ-30722 from sk175545
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
exporting from smartview seems to be the only option for me. when i try to export directly form r81 smartconsole, i receive this message:
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Right, in R81 this is the only option, please get the hotfix from TAC
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Does the R81 hotfix allow export from console? I can see my historical logs within console, but they do not show in SmartView. If I could export them from there I could at least have something. Case opened for almost a week now. Days to index set to 31 but SmartView only shows current and previous day logs, but will show statistics on the side for them. Exporting logs is an important feature and all of the engineers that have had my case so far say 'we don't know why you can't export from console. Console has always worked.'
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
In R81 we support export to CSV of 1 million records from SmartView web only and not SmartConsole, and all logs should be available not only current and previous day.
I suggest to verify the time frame filter, because we recently changed the default time frame filter to last 24 hours to improve the performance of the system.
If everything is configured properly and still you don't see logs of 31 days, please share with me the SR number and I will work internally to ensure you have the full info from SmartView
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
SR#6-0003092442
Still puzzling that each engineer on my case has said that it should work from SmartConsole. Apparently they are not trained that this changed.
Also, as I understand it, SmartView can only show logs that are indexed, but sometimes I have a need to pull logs from several months ago and I don't see a way to open the older flat log files there. This feels like a step in the negative direction.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Bonnie,
Stacy also shared with me this case.
I discussed it with the case owner, he will have a session with you today, as far as I know, and I am sure there will be a progress. I am monitoring it with him.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
![](/skins/images/84DAB6BD358ECB13CE1094473F6E2961/responsive_peak/images/icon_anonymous_message.png)