How to verify logs and log indexing

[fjulianom]

Hi everyone,

My customer MDS suddenly started losing logs some months ago. It was a problem related to the disk space, it seems the MDS didn't delete old logs, and when the disk was full, it started to lose the current logs. We opened a TAC case, and for solving the problem in some way, customer created a script for maintaining the disk space low and not losing logs. Yesterday, customer told me the MDS suddenly started again to work fine about the logs again, the disk space was reduced, and indexing started to work fine again. We don't know the reason yet. But, apart from the root problem, how can I verify logs and log indexing is working fine? What files do I have to check? Sorry but I am kind of newbie in Check Point.

Regards,

Julián

[Lesley]

I am getting triggered by this:

"customer created a script for maintaining the disk space low and not losing logs."

What is this script and why? The system should automatically cleanup old logs if configured correctly.

I would recommend moving away from this customer script and check the settings here:

SmartConsole -> Relevant SmartCenter mgmt object -> Logs -> Local Storage

[PhoneBoy]

The Doctor Log script mentioned here might be useful: https://support.checkpoint.com/results/sk/sk181782 

[fjulianom]

Hi guys,

The problem is not easy. As said, I opened a case to TAC and didn’t find the root cause. They created a fix and it didn’t work, so my customer had to find a temporal solution. TAC continues investigating. But as said, I didn’t open this thread to investigate the root cause of the issue, because first, TAC is on it (I hope), and second, customer told me is solved (suddenly it appeared, and suddenly it disappeared). Then, apart from the root problem, do you know how can I verify logs and log indexing is working fine right know? What files do I have to check? I have no idea. Please your help.

Regards,

Julian

[Alex-]

Check $INDEXERDIR/log/log_indexer.elg

[Lesley]

Review the output of "cpstat fw -f log_connection" on the Security Gateway

Review the output of "cpstat ls -f logging" on the Log Server

Is the active firewall log file fw.log growing on the Security Gateway

• On Gaia / SecurePlatform / Linux :

  # watch -d -n 2 "ls -l $FWDIR/log/fw.log"

Is the Security Management Server listening on TCP port 257

• On Gaia / SecurePlatform / Linux / IPSO OS:

  # netstat -anp | grep ":257"

[fjulianom]

Hi,

Many thanks, I will try tomorrow.

Regards,

Julian

[PhoneBoy]

The Doctor Log script outputs diagnostic information that should help you determine this.
At a high level, the script returns the following:

*** Diagnostics Results ***

System Status : OK
Changes in Config Files : OK
MGMT version : OK
Registry file Corruptions : OK
Load Average : OK
Check Processes : Error
Inconsistencies Detector : OK
Connections Config : OK
GW's and Log Clients : OK
Problems in Debug Log Files : OK
Rfl/Solr Memory Report : OK
Verify FetchedFiles : OK
Maintenance Configuration : OK
Smart View Status : Warning
SmartView performance : OK
Topology View : OK
Abnormal Long Requests : OK
RFL Status Query : OK
Current Logging Rates : OK
Daily Average Logging Rates : OK
Sizing status : OK
Detects Indexing Delay : OK
Top Consuming Processes : OK
dbsync status : OK

It will show some additional information, including logging rates, top consuming processes, and issues found, as shown below.
In this case, it's a standalone gateway running R81.20 and not passing traffic (thus the warning about log rates) 🙂

Issues Found:
----------------------------

System Info:
  Attention           : Some of the server's jars are from different versions 

Registry file Corruptions:
  Possible fix        : Please refer to: sk168472 

Check Processes:
  ERROR               : RFL_SOLR_MAX_HEAP is smaller then RFL_SOLR_MIN_HEAP. 'D_SOLR' process will fail to start. 
  Possible fix        : Please refer to: sk172385 

Smart View Status:
  WARNING             : Found no data in the file: [smartview] from the last 1 hours 
                              Check the debug logging level of smartview 
                              If debug level is correct this could indicate that the process is stuck 


Current Logging Rates:
  Hint                : Indexed mode is disabled (non-indexed mode) 

Summary:
Found 1 Errors, 1 Warnings in this running configuration.

Detailed report and more can be found under /tmp/sme-diag/results

*** Diagnostic Completed ***

[fjulianom]

Hi,

Many thanks. I remember TAC requested me this script output when I opened the case. I will run again tomorrow. No impact, right?

Regards,

Julian

[just13pro]

There is no impact on running doctor log.

Remember to run on Management Server

[the_rock]

The sk Phoneboy gave is super useful.

Andy

[fjulianom]

Hi,

I run the doctor-log script and still many errors related to log and log indexing... so better to continue investigate the root cause of the issue.

Regards,

Julián

[the_rock]

Can you post those errors here?

Andy

[fjulianom]

Hi,

What exactly? There are many errors. I tried attaching the doctor_log_report.html and doctor_log.tar.gz files, and both extensions are not supported here.

Regards,

Julián

[the_rock]

If you message me directly, we can connect offline. I would love to have a look at the file and see if I can help further.

Andy

[Alex-]

$INDEXERDIR/log/log_indexer.elg logs the indexing status and if there are errors you could see them in there.

However your customer might have gone too far with their custom script.

Since the Doctor Log reported issues, your best approach is to continue with TAC.

[Amir_Senn]

Not solving the issues but the file "$RTDIR/log_indexer/data/FetchedFiles" keep tracks of what part of the log files are indexed.

The numbers in the end tell you how many logs were indexed. If I remember correctly, if it ends with 3 it means it's finished indexing the file.

Example: 2 9 127.0.0.1 21 2024-09-06_235900.log 1725570000 1 5819 0 0 3