Problem with displaying logs in SmartConsole

[Nadezhda]

Hi!

We are facing a rather unusual problem.
Logs in SmartConsole are displayed as empty lines, when clicking on this line we see the error "Could not open log file @A@@@B@1707955200 on position:819573. Possible reasons: log-files were deleted, moved or have been corrupted"

The magic is that if you refresh the page several times, the logs will display correctly on a random refresh, but after a new refresh they will disappear again.

Any guesses ? What can this be related to ?

[the_rock]

Hey,

What version is this? First thing I would try, as its management, so is very safe to do, would be either cpstop; cpstart and if no luck, you can easily reboot it.

Best,

Andy

[Nadezhda]

Hi!

This appears on both version 80.40 and 81.20.
cpstop/cpstart helps only temporarily. we assume that the problem has something to do with SmartEvent.

[the_rock]

You mean if mgmt is R80.40 and R81.20? Do you have dedicated se server? I ask, because we have customer with dedicated smart event and they had issue for longest time, we have TAC case about it, went to escalation team and guy we worked with told us after lots of investigation and checking there was a problem that stemmed from database issue and once client added more RAM, it did actually help a lot.

Best,

Andy

[PD_localuser]

Hello.

i have same problem, logs appear some times or if you build some custom filters.

i change log register file with command  FW logswitch last friday, and this action appear apparently resolve this problem because new logs appear fully and display correctly, but today we have same problem, in some moment this behaivor reappear

What is your version? we have R81.10

[the_rock]

This happens after you cpstop;cpstart or reboot?

Best,

Andy

[Nadezhda]

Hi,

This helps for a while, but then the problem occurs again.

[Lesley]

What jumbo takes are you running? What about the Smartconsole software itself? Is that updated?

[the_rock]

I was thinking when I saw your response that it used to happen in R80.40 as well, I would open support case and have this investigated. Is this option enabled?

Best,

Andy

 

[Nadezhda]

Yes, this feature is enabled

[the_rock]

One time client had this issue, TAC simply asked them to disable it, install database, re-enable, install database again and that fixed it.

Not sure if you attempted that or not.

Best,

Andy

[PD_localuser]

Yes, I ried also restart

[the_rock]

I would say TAC case would be best in that case, as it appears you tried most things people do.

Best,

Andy

[PD_localuser]

sure, if we'll find solution and procedure to solucion i'll share in this post

[the_rock]

Yes please, as this seems to be an issue that comes up frequently.

Best,

Andy

[Amir_Senn]

Good thing to check is how much free space left in the log partition.

[the_rock]

Always keep forgetting about it, but so important, for sure.

Best,

Andy

[PD_localuser]

thanks. we have 87% occupation in /var/log

tomorrow i´ll work to free some space, the disk is about 2TB and approx 230GB free, 230Gb appear are enough but i´m not sure if this 87% has priority

[the_rock]

I would do below from expert:

find /var/log -size +900M

see what files you can delete. You can also replace 9 with any other digit, so say 400M is 400Mbs

Best,

Andy

[PD_localuser]

Hi everybody

i update my case

Last week we delete several backup files, schedule backup was bad configuration and since 2021 every day this schedule was execute, after remove space in /var/log/ decrease to 40%  approx 

after, we execute comand fw logswitch, from thursday to now logs are visible without problems, now we need trie recover old logs before thursday.

Any tips for this?

[the_rock]

So if you do fw logswitch, that would technically rotate the existing log, so say its currently size 1 GB (just making that up), it would default it to 0 and start there, so then would rotate again by default at midnight or when it reaches 2 GB in size (whichever would come first).

Now, as far as older logs, can you navigate to $FWDIR/log directory and see if they are there?

Best,

Andy

you can do this...example from my lab.

[Expert@azurefw:0]# cd $FWDIR/log
[Expert@azurefw:0]# ls -lh *.log
-rw-rw---- 1 admin root 19K Jan 6 00:00 2024-01-06_000000.log
-rw-rw---- 1 admin root 9.5M Jan 8 09:30 2024-01-08_093043.log
-rw-rw---- 1 admin root 8.2K Jan 8 09:30 fw.log
-rw-rw---- 1 admin root 1.8M Feb 26 14:10 tracker.log
[Expert@azurefw:0]#

[PD_localuser]

unfortunately only have a pair .log files from after switch

i need investigate more but i this moment can see other files type or audit logs files since 2023

i hope wasn´t are delete or move

[the_rock]

If you dont see them there, not sure changing anything with indexing may help : - (. Maybe you can ask via TAC case, see what they say.

Best,

Andy

[Amir_Senn]

If they don't appear in the log directory they were probably deleted. Under some conditions audit log files will not be deleted - this is because they're insignificant in size in comparison to traffic log file.

You can check $FWDIR/log/fwd.elg , search for the following outputs:

CCyclicLogging::_moveFile: moving file: 2015-12-02_000000.log, for deletion dir: /opt/CPsuite-R80/fw1/log//cl_del
FWLOG_MAINTENANCE - RemoveFilesFromCLDir: removing file: 2015-12-02_000000.log from dir: /opt/CPsuite-R80/fw1/log//cl_del

[Nadezhda · Accepted solution]

Hello!

I'm sharing a solution that has helped us and hopefully will help someone else.

1. stop the logging and monitoring processes:
#evstop
2. Move FetchedFiles:
#mv $INDEXERDIR/data/FetchedFiles $INDEXERDIR/data/FetchedFiles.backup
3. Start the logging and monitoring processes::
#evstart

[the_rock]

Thats super useful, thank you!

Andy

[Amir_Senn]

FetchedFiles is the file that monitors what log files are indexed and status.

When you remove the file it will create it again. If you don't remove the indexes, what is the affect of this?

[the_rock]

Hey Amir,

Are you saying this process should not be followed or more along the lines backup everything before doing it?

Best,

Andy

[Amir_Senn]

IMO, more likely that restarting the logging processes helped more. I think we might have racing conditions in some scenarios that restarting the processes solves.

I played with it in my lab. Looks like this doesn't behave well:

I had no live traffic, only recorded traffic I inject. After injection firewallandvpn core showed directory size of 128Mb and other-smartlog core was 48Mb (stable). After evstop ; removing FetchedFiles ; evstart - after restart FetchedFiles doesn't have information about the log files so indexer will index backwards as many days as defined in definitions and the size increased on both cores directories:

firewallandvpn 128Mb -> 142Mb -> 176 Mb -> 218Mb -> 152Mb

other-smartlog 48Mb -> 74Mb

So we know that it creates more indexes and since we can also see a decrease it might consolidate some of them but either way it keeps more indexes than needed. IDK what this can cause, from nothing to not delete indexes properly to other behaviors.

Usually if we look on scenarios in which we delete the FetchedFiles we also delete the indexes - no double sets of indexes.

[the_rock]

K, fair enough, thank you.

Andy