cancel
Showing results for 
Search instead for 
Did you mean: 
Create a Post
Kim_Moberg
Silver

SmartLog only look back 14 days - how to reindex 90 days back?

Hi

I have a challenge with Mgmt log indexing in R80.10 take 56.

In my SmartLog I can only look 14 days back in time, but in folder /opt/CPsuite-R80/fw1/log/ there are logs back to August 2017.

How can I index the old logs so they are included in SmartLog?
I have found sk77640 (SmartLog does not index logs that existed prior to SmartLog installation) but does not include R80.10.

Any suggestions?

Thanks

Kim

Best Regards
Kim
12 Replies

Re: SmartLog only look back 14 days - how to reindex 90 days back?

Hi,

I am not sure about R80.10 but we have done following things in R77.30 for the same issue.

If you right click on Header, There are options like "sort by Log server arrival" or "Sort by log timestamp". You can select "sort by log timestamp"

Please check if same options are there in R80.10

 

0 Kudos
Admin
Admin

Re: SmartLog only look back 14 days - how to reindex 90 days back?

To index the older log files, follow the steps in the R80.10 docs for importing Offline Log Files:

Kim_Moberg
Silver

Re: SmartLog only look back 14 days - how to reindex 90 days back?

Dameon,

Where should I insert the extra line for let it index more than 90 days?

$INDEXERDIR/log_indexer_custom_settings.conf

(
:data ("/opt/CPrt-R80/log_indexer/data")
:server_port ("127.0.0.1:18244")
:dns_resolving (true)
:dns_backresolving (true)
:connections (
:domain (
:management (
:name (127.0.0.1)
:uuid ()
:log_files (all)
:is_local (true)
:read_mode (CPMI)
)
:log_servers (
: (
:name (127.0.0.1)
:uuid ()
:log_files (all)
:folder ("/opt/CPsuite-R80/fw1/log")
:is_local (true)
:read_mode (FILES)
)
)
)
)
:max_disk_space_usage (0)
)

I have tried to add :num_days_restriction_for_fetch_all_integrated (90) it before or after :max_disk_space_usage (0). But in the link to the guide, one have to remove two lines first which doesn't exist in my situation.

Any hints what do look after?

Thanks

Best Regards
Kim
0 Kudos
Admin
Admin

Re: SmartLog only look back 14 days - how to reindex 90 days back?

I think you can put it after dns_backresolving.

Or you can try what https://community.checkpoint.com/people/simone996b1d2-bee9-3af1-a14a-7f918695c76d‌ suggested above Smiley Happy

0 Kudos

Re: SmartLog only look back 14 days - how to reindex 90 days back?

Follow sk98894 - Run SmartEvent Offline Jobs for multiple log files"

FYI : The doctor-log script should be able to pick up any of the errors during the reindexing, it will also check the status of the other SmartEvent components, it may be worthwhile.

$RTDIR/scripts/doctor-log.sh -f

Kim_Moberg
Silver

Re: SmartLog only look back 14 days - how to reindex 90 days back?

Simon, 

I have tried to run the command, before and after changing the file $INDEXERDIR/log_indexer_custom_settings.conf but it keeps telling there is an error with my SmartEvent.

[Expert@gwmgmt:0]# $RTDIR/scripts/doctor-log.sh -f
Initializing...

*** Detailed Diagnostics Results ***

System Status : Attention (79% of disk in use)
Changes in Config Files : OK
Load Average : Medium(>2.0)
Check Processes : OK
Correlation Unit Status : OK
Correlation Units Config : OK
Connections Config : OK
GW's and Log Clients : Warning
Problems in Debug Log Files : Error
Rfl/Solr Memory Report : OK
Log Indexes : OK
Maintenance Configuration : OK
Smart View Status : Error
Total Logs Number : OK
Logging/Indexing Rates : OK
Indexing Status : OK
Query Solr Logs by Product : OK


System Info:
Machine type : VM
Version : R80.10
Branch : R80_10_jumbo_hf
Take : 421
Hotfix : HOTFIX_R80_10, HOTFIX_R80_10_JUMBO_HF take_56
Is Upgraded : Yes
Management : Smart Center + Smart Event
Pre R80 Dbsync : Yes


System Status:
OS Ver : 64-bit
CPUs : 4
Total Memory : 15917 Mb
Free Memory : 7146 Mb
Used Disk Mb : 312G
Used Disk % : 79%


Logging/Indexing Rates:
Rates metrics is logs per second
Logging Rate : 47
Indexing Rate : 55


Issues Found:
----------------------------


System Status:
WARNING : Used over 70% of disk space


Check Processes:
Attention : Found core dumps for CPSEAD
Attention : Found core dumps for CPSEMD
Attention : Found core dumps for CPD


GW's and Log Clients:
WARNING : Possible Monitoring issue:
gw1 Last Login Time is Wed Dec 13 13:56:22 2017
WARNING : Possible Monitoring issue:
gw2-de Last Login Time is Wed Dec 13 14:39:09 2017
WARNING : Possible Monitoring issue:
GW1-PL Last Login Time is Thu Jan 4 19:44:21 2018
WARNING : Possible Monitoring issue:
GW1-RO Last Login Time is Thu Jan 4 19:42:51 2018
WARNING : Possible Monitoring issue:
GW1-SE Last Login Time is Thu Jan 4 09:13:20 2018
WARNING : Possible Monitoring issue:
gw1-de Last Login Time is Wed Dec 13 14:15:29 2017
WARNING : Possible Monitoring issue:
gw2 Last Login Time is Wed Dec 13 13:58:48 2017


Problems in Debug Log Files:
WARNING : Found total of 6 occurrences of exception indicators in the last 1 hours
In "/opt/CPrt-R80/log/solr.log"
ERROR : [15 Jan 17:13:24] - Indexer failed to connect Solr. Solr process is down, or not listening for connections on local machine
WARNING : Found total of 108 occurrences of exception indicators in the last 1 hours
In "/opt/CPrt-R80/log_indexer/log/log_indexer.elg"


Smart View Status:
ERROR : Found a large number of exception indicators (54) in smartview
WARNING : Found total of 2 occurrences of exception indicators in the last 1 hours
In "/opt/CPrt-R80/log/smartview-service.log"


Summary:
Found 2 Errors, 11 Warnings in this running configuration.


Detailed report and more can be found under /tmp/sme-diag/results

*** Diagnostic Completed ***

I have restored the file  vi $INDEXERDIR/log_indexer_custom_settings.conf back to before I changed it.

cp $INDEXERDIR/log_indexer_custom_settings.conf_orig $INDEXERDIR/log_indexer_custom_settings.conf

Thanks

Kim

Best Regards
Kim
0 Kudos

Re: SmartLog only look back 14 days - how to reindex 90 days back?

Not too sure if you resolved it already, but check if indexer is not stuck at some specific log. Check sk112336 for details how. Or there is one-liner here https://community.checkpoint.com/message/11199-how-to-quickly-check-log-indexing-backlog  

I have seen odd behaviour when queue gets stuck, but doesn't sound like your case

0 Kudos
Kim_Moberg
Silver

Re: SmartLog only look back 14 days - how to reindex 90 days back?

I wasn't able to get it solve yet..

It is still an open issue, but I hope I will soon manage to get deeper into what the problem is.

Best Regards
Kim
0 Kudos
Tom_Cripps
Copper

Re: SmartLog only look back 14 days - how to reindex 90 days back?

Hi Kim,

I've just found this with in the general settings of our primary management server.

Hope this helps you?

Re: SmartLog only look back 14 days - how to reindex 90 days back?

That option is removed from MDS/CMA/CLM env Smiley Happy

0 Kudos
Tom_Cripps
Copper

Re: SmartLog only look back 14 days - how to reindex 90 days back?

Good to know! Our environment is just a single domain. 

Re: SmartLog only look back 14 days - how to reindex 90 days back?

Hello guys,

If any of you are still having trouble re-indexing logs, here is the solution for each version.

VersionR80, R80.10, R80.20

R80.x SmartLog/SmartEvent server doesn't index/show logs older than 1-14 days back 

VersionR76, R77, R77.10, R77.20, R77.30

SmartLog does not index logs that existed prior to SmartLog installation 

VersionR75.40, R75.40VS, R75.45, R75.46, R76

SmartLog does not index log files that were moved back to log directory - Specific Scenario

0 Kudos