- CheckMates
- :
- Products
- :
- Quantum
- :
- Management
- :
- Doubt about reindexing logs
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Are you a member of CheckMates?
×- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Doubt about reindexing logs
Hi guys,
I am doing a lab and an upgrade with migration from a R81.10 SMS to a R81.20 SMS. I have imported the database with only logs (-l option in command "migrate_server import"). The migration was fine, and as expected, I couldn't see the R81.10 SMS logs, since I used option -l which migrates the logs without logs indexes. Then I wanted to reindex those logs, and then I follow this post, which leads to sk111766. After following the sk, I was able to see the old R81.10 SMS logs. So far everything is fine. But my doubt is the following. I run the command "./log_indexer -days_to_index 30 -workingDir $INDEXERDIR/", and after doing the evstart, the Logs/Storage section was like this:
I thought that command would enable the checkbox "Apply the following logs retention policy" and put 30 days in "Keep indexed logs for no longer than". In fact, now the log_indexer_custom_settings.conf file has added the line ":days_to_index (30)".
Isn't there a relation between the Logs/Storage/Daily Logs Retention Configuration section and the log_indexer_custom_settings.conf file?
Am I missing something?
Regards,
Julián
Accepted Solutions
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
They're not the same. The SmartConsole configuration is about when to remove indexes after the logs have been indexed, the 'days to index' is about how long in the past the log instance should index back to, but does not have any affect on then removing those indexes. By default it is 1 day. If you enabled the log retention in Smart Console to delete indexes older than 14 days but you tell it to index back 30 days, they'll fight with each other. So don't do that.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Im pretty sure running that command does NOT actually change setting in smart console, at least thats what TAC informed me while back. This was back in R81.10, but I doubt its different in R81.20
Andy
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Andy,
Yes, you are right. I made a test and enabled the checkbox "Apply the following logs retention policy", and put 45 days in "Keep indexed logs for no longer than", and installed database. Then I checked the log_indexer_custom_settings.conf file and it kept the line ":days_to_index (30)", no change. Then, what is the relation between the Logs/Storage/Daily Logs Retention Configuration section and the log_indexer_custom_settings.conf file? I thought the "Keep indexed logs for no longer than" option was related to line ":days_to_index (30" in log_indexer_custom_settings.conf file. And, if they are not the same, what is the difference?
Regards,
Julián
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Can you send content of that file? I just checked mine and dont even see that line anywhere.
[Expert@CP-MANAGEMENT:0]# cd /opt/CPrt-R81.20/log_indexer
[Expert@CP-MANAGEMENT:0]# more log_indexer_custom_settings.conf
(
:data ("/opt/CPrt-R81.20/log_indexer//data")
:server_port ("127.0.0.1:18244")
:dns_resolving (true)
:dns_backresolving (true)
:connections (
:domain (
:management (
:name (127.0.0.1)
:uuid ()
:log_files (all)
:is_local (true)
:read_mode (CPMI)
)
:log_servers (
: (
:name (127.0.0.1)
:uuid ()
:log_files (all)
:folder ("/opt/CPsuite-R81.20/fw
1/log")
:is_local (true)
:read_mode (FILES)
)
)
)
)
:max_disk_space_usage (0)
)
Andy
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
Sure, this is the mine:
(
:data ("/opt/CPrt-R81.20/log_indexer//data")
:server_port ("127.0.0.1:18244")
:dns_resolving (true)
:dns_backresolving (true)
:connections (
:domain (
:management (
:name (127.0.0.1)
:uuid ()
:log_files (all)
:is_local (true)
:read_mode (CPMI)
)
:log_servers (
: (
:name (127.0.0.1)
:uuid ()
:log_files (all)
:folder ("/opt/CPsuite-R81.20/fw1/log")
:is_local (true)
:read_mode (FILES)
)
)
)
)
:max_disk_space_usage (0)
:days_to_index (30)
)
And this was after following sk111766, and it makes sense, the sk itself says:
Regards,
Julián
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Got it now, I dont have that line as I never made those changes. Might be worth TAC case to confirm the behavior, but I get the point you are making.
Andy
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
Yes, as I said, the migration with logs and without log indexes has gone pretty fine, and the reindexing the logs as well following the sk. The doubt I have after reindexing the logs is what is the relation between the Logs/Storage/Daily Logs Retention Configuration section and the log_indexer_custom_settings.conf file? I thought the "Keep indexed logs for no longer than" option was related to line ":days_to_index (30" in log_indexer_custom_settings.conf file. And, if they are not the same, what is the difference?
I will wait to see if anyone sheds some light on this, otherwise I will consult with TAC... Many thanks anyway 😉
Regards,
Julián
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
That would make sense to me they are related, but oddly enough, when you hit ? mark in smart console when you are looking at those settings, does NOT give explanation for that setting, wonder why lol
Andy
Storage
What can I do here?
In this window, you can configure how logs are stored and manage their size according to disk space.
Tell me about the fields...
- Measure free disk space in - Choose MBytes or Percentage.
- When disk space is below <number> Mbytes, issue alert- Get an alert when the available disk space for logs and log index files is below this threshold. This value must be at least 5 MB greater than the value of ...stop logging in the Additional logging options page.
- When disk space is below <number> Mbytes, start deleting old files- Delete the oldest logs and log index files when the available disk space is below this threshold. This value must be at least 5 MB greater than the ...issue alert value.
- Run the following script before deleting old files - Enter a path to the script.
- Apply the following log retention policy - set the number of days index logs are kept for or set an extra number of days to keep log files.
This option is for Gateways only
- Reserve ... for packet capturing - Some types of logs can also capture the packets that created the log event. Set the amount, in MBytes or Percent, that you want to use for captured packets.
These options and examples are for a Security Management Server or SmartEvent Server only:
- Index Files - Configure when the Security Management Server or SmartEvent Server automatically deletes old index files:
- Delete index files older than <number> Days - the Security Management Server or SmartEvent Server automatically deletes index files that are older than a specified number of days. Specify the number of days. The index files are deleted even if there is enough disk space for them.
Examples:
These examples show how these options work together
- When disk space is below <number> Mbytes, start deleting old files
- Delete index files older than <number> Days
For these examples, the administrator enables these thresholds:
- When disk space is below [5000] Mbytes, start deleting old files
- Delete index files older that [14] Days
Example 1:
The server has 3000 MBytes of free disk space, and 5 days of logs and index files.
The server deletes logs and index files, one day at a time, until there is 5000 Mbytes of free disk space.
Example 2:
The server has 10 GBytes of free disk space and 30 days of logs and index files.
The server deletes all index files older than 14 days.
Example 3:
The server has x days of index files and x+5 days of logs.
The server deletes log and index files, one day at a time, in this order:
- Deletes index files older than 14 days
- Deletes logs until only 14 days remain, or until the disk space threshold is reached.
- Deletes logs and index files until the disk space threshold is reached.
Example 4:
The server has x days of logs, and x+6 days of index files.
The server deletes log and index files, one day at a time, in this order:
- Deletes index files older than 14 days
- Delete Index files until only x days remain, or until the disk space threshold is reached.
- Delete logs and index files, until the disk space threshold is reached.
|
Getting Here - Network object properties > Logs > Storage or: Network object properties > Logs > Storage |
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Running log_indexer on the CLI won't change the SmartConsole configuration.
It is meant to manually reindex in certain instances, e.g. when you do certain upgrades or restoration of logs.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Glad you confirmed that. I sort of suspected that was the case from the sk, but was not 100% sure.
Best,
Andy
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @PhoneBoy ,
Then do you mean that "Keep indexed logs for no longer than" in SmartConsole do the equivalent than line ":days_to_index (xx)" in log_indexer_custom_settings.conf file, only that the latest is used to manually reindex?
For example, with Log Exporter happens similar thing, you can configure Log Exporter via CLI with the cp_log_export commands, but this configuration is not reflected in SmartConsole. Is something like this?
Regards,
Julián
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
They're not the same. The SmartConsole configuration is about when to remove indexes after the logs have been indexed, the 'days to index' is about how long in the past the log instance should index back to, but does not have any affect on then removing those indexes. By default it is 1 day. If you enabled the log retention in Smart Console to delete indexes older than 14 days but you tell it to index back 30 days, they'll fight with each other. So don't do that.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
Then in an SMS with no changes in the log_indexer_custom_settings.conf file, do you say the SmartLog should show logs for the last 1 day (the default)? I made a test, in that SMS I can see logs for many days back.
Regards,
Julián
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The indexing will start with logs from when the server was stood up. The logs that have been indexed will remain in the database until the disk is nearly full, unless you enable the log retention option to remove them after X days. So SmartLog will show logs from when the server was built to now.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
Then, I don't understand "the 'days to index' is about how long in the past the log instance should index back to"... Some example?
Regards,
Julián
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
If you spin up a new server and copy over some older log files to its log directory, it will not index them by default, as it will only index logs into the log database from the day before it was built and forwards. If you want the older logs to be indexed, you need to do the '$INDEXERDIR/log_indexer -days_to_index <days>' command to cover how far back the logs go that you imported. Once those logs are indexed, that piece of configuration you put in is basically vestigial.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi emmap,
Got it now 😁
One more doubt, the "Keep indexed logs for no longer than" option does remove old index files. For example, "Keep indexed logs for no longer than 30 days" will remove index files older than 30 days back. Will this remove the logs themselves (.log), or the index files (.logptr, .loginitial_ptr, etc.)?
Regards,
Julián
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
It only removes the indexes unless you also check (and configure) "Keep Log Files For An Extra X Days" which will remove the underlying log files X days after the indexes are removed.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
Then, what's the advantage of option "Keep indexed logs for no longer than"? Because log files can be big, but IMO indexes are small files...
Regards,
Julián
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
It cleans up disk space, as the logs exist in both the log files and also the log database. If you only keep one of them it's generally best to keep the log files as they can be reindexed or manually opened, or exported off the box.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi everyone,
Many thanks guys, I understand now 😀
By the way @the_rock , the mark ? in SmartConsole does give an explanation on that setting, what happens is the wording is different. The window that pops-up when you click on ? in SmartConsole says "Delete index files older than...", and the parameter we are talking about here is "Keep indexed logs for no longer than", but they are actually the same. They should change this in the documentation. The latest one is from R81.10 on, I think.
Regards,
Julián