How to verify logs and log indexing

[fjulianom]

Hi everyone,

My customer MDS suddenly started losing logs some months ago. It was a problem related to the disk space, it seems the MDS didn't delete old logs, and when the disk was full, it started to lose the current logs. We opened a TAC case, and for solving the problem in some way, customer created a script for maintaining the disk space low and not losing logs. Yesterday, customer told me the MDS suddenly started again to work fine about the logs again, the disk space was reduced, and indexing started to work fine again. We don't know the reason yet. But, apart from the root problem, how can I verify logs and log indexing is working fine? What files do I have to check? Sorry but I am kind of newbie in Check Point.

Regards,

Julián

[Lesley]

I am getting triggered by this:

"customer created a script for maintaining the disk space low and not losing logs."

What is this script and why? The system should automatically cleanup old logs if configured correctly.

I would recommend moving away from this customer script and check the settings here:

SmartConsole -> Relevant SmartCenter mgmt object -> Logs -> Local Storage

[PhoneBoy]

The Doctor Log script mentioned here might be useful: https://support.checkpoint.com/results/sk/sk181782 

[fjulianom]

Hi guys,

The problem is not easy. As said, I opened a case to TAC and didn’t find the root cause. They created a fix and it didn’t work, so my customer had to find a temporal solution. TAC continues investigating. But as said, I didn’t open this thread to investigate the root cause of the issue, because first, TAC is on it (I hope), and second, customer told me is solved (suddenly it appeared, and suddenly it disappeared). Then, apart from the root problem, do you know how can I verify logs and log indexing is working fine right know? What files do I have to check? I have no idea. Please your help.

Regards,

Julian

[Alex-]

Check $INDEXERDIR/log/log_indexer.elg

[fjulianom]

Hi,

What is the expected output of this file?

Regards,

Julian

[Lesley]

Review the output of "cpstat fw -f log_connection" on the Security Gateway

Review the output of "cpstat ls -f logging" on the Log Server

Is the active firewall log file fw.log growing on the Security Gateway

• On Gaia / SecurePlatform / Linux :

  # watch -d -n 2 "ls -l $FWDIR/log/fw.log"

Is the Security Management Server listening on TCP port 257

• On Gaia / SecurePlatform / Linux / IPSO OS:

  # netstat -anp | grep ":257"

[fjulianom]

Hi,

Many thanks, I will try tomorrow.

Regards,

Julian

[PhoneBoy]

The Doctor Log script outputs diagnostic information that should help you determine this.
At a high level, the script returns the following:

*** Diagnostics Results ***

System Status : OK
Changes in Config Files : OK
MGMT version : OK
Registry file Corruptions : OK
Load Average : OK
Check Processes : Error
Inconsistencies Detector : OK
Connections Config : OK
GW's and Log Clients : OK
Problems in Debug Log Files : OK
Rfl/Solr Memory Report : OK
Verify FetchedFiles : OK
Maintenance Configuration : OK
Smart View Status : Warning
SmartView performance : OK
Topology View : OK
Abnormal Long Requests : OK
RFL Status Query : OK
Current Logging Rates : OK
Daily Average Logging Rates : OK
Sizing status : OK
Detects Indexing Delay : OK
Top Consuming Processes : OK
dbsync status : OK

It will show some additional information, including logging rates, top consuming processes, and issues found, as shown below.
In this case, it's a standalone gateway running R81.20 and not passing traffic (thus the warning about log rates) 🙂

Issues Found:
----------------------------

System Info:
  Attention           : Some of the server's jars are from different versions 

Registry file Corruptions:
  Possible fix        : Please refer to: sk168472 

Check Processes:
  ERROR               : RFL_SOLR_MAX_HEAP is smaller then RFL_SOLR_MIN_HEAP. 'D_SOLR' process will fail to start. 
  Possible fix        : Please refer to: sk172385 

Smart View Status:
  WARNING             : Found no data in the file: [smartview] from the last 1 hours 
                              Check the debug logging level of smartview 
                              If debug level is correct this could indicate that the process is stuck 


Current Logging Rates:
  Hint                : Indexed mode is disabled (non-indexed mode) 

Summary:
Found 1 Errors, 1 Warnings in this running configuration.

Detailed report and more can be found under /tmp/sme-diag/results

*** Diagnostic Completed ***

[fjulianom]

Hi,

Many thanks. I remember TAC requested me this script output when I opened the case. I will run again tomorrow. No impact, right?

Regards,

Julian

[the_rock]

The sk Phoneboy gave is super useful.

Andy