Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
Nadezhda
Contributor
Jump to solution

Problem with displaying logs in SmartConsole

Hi!

We are facing a rather unusual problem.
Logs in SmartConsole are displayed as empty lines, when clicking on this line we see the error "Could not open log file @A@@@B@1707955200 on position:819573. Possible reasons: log-files were deleted, moved or have been corrupted"

The magic is that if you refresh the page several times, the logs will display correctly on a random refresh, but after a new refresh they will disappear again.

Any guesses ? What can this be related to ?

0 Kudos
1 Solution

Accepted Solutions
Nadezhda
Contributor

Hello!

I'm sharing a solution that has helped us and hopefully will help someone else.

1. stop the logging and monitoring processes:
#evstop
2. Move FetchedFiles:
#mv $INDEXERDIR/data/FetchedFiles $INDEXERDIR/data/FetchedFiles.backup
3. Start the logging and monitoring processes::
#evstart

View solution in original post

(1)
29 Replies
the_rock
Legend
Legend

Hey,

What version is this? First thing I would try, as its management, so is very safe to do, would be either cpstop; cpstart and if no luck, you can easily reboot it.

Best,

Andy

0 Kudos
Nadezhda
Contributor

Hi!

This appears on both version 80.40 and 81.20.
cpstop/cpstart helps only temporarily. we assume that the problem has something to do with SmartEvent.

0 Kudos
the_rock
Legend
Legend

You mean if mgmt is R80.40 and R81.20? Do you have dedicated se server? I ask, because we have customer with dedicated smart event and they had issue for longest time, we have TAC case about it, went to escalation team and guy we worked with told us after lots of investigation and checking there was a problem that stemmed from database issue and once client added more RAM, it did actually help a lot.

Best,

Andy

0 Kudos
PD_localuser
Explorer

Hello.

i have same problem, logs appear some times or if you build some custom filters.

i change log register file with command  FW logswitch last friday, and this action appear apparently resolve this problem because new logs appear fully and display correctly, but today we have same problem, in some moment this behaivor reappear

What is your version? we have R81.10

0 Kudos
the_rock
Legend
Legend

This happens after you cpstop;cpstart or reboot?

Best,

Andy

0 Kudos
Nadezhda
Contributor

Hi,

This helps for a while, but then the problem occurs again.

0 Kudos
Lesley
Leader Leader
Leader

What jumbo takes are you running? What about the Smartconsole software itself? Is that updated?

-------
If you like this post please give a thumbs up(kudo)! 🙂
0 Kudos
the_rock
Legend
Legend

I was thinking when I saw your response that it used to happen in R80.40 as well, I would open support case and have this investigated. Is this option enabled?

Best,

Andy

 

Screenshot_1.png

0 Kudos
Nadezhda
Contributor

Yes, this feature is enabled

0 Kudos
the_rock
Legend
Legend

One time client had this issue, TAC simply asked them to disable it, install database, re-enable, install database again and that fixed it.

Not sure if you attempted that or not.

Best,

Andy

0 Kudos
PD_localuser
Explorer

Yes, I ried also restart

0 Kudos
the_rock
Legend
Legend

I would say TAC case would be best in that case, as it appears you tried most things people do.

Best,

Andy

0 Kudos
PD_localuser
Explorer

sure, if we'll find solution and procedure to solucion i'll share in this post

0 Kudos
the_rock
Legend
Legend

Yes please, as this seems to be an issue that comes up frequently.

Best,

Andy

0 Kudos
Amir_Senn
Employee
Employee

Good thing to check is how much free space left in the log partition.

Kind regards, Amir Senn
the_rock
Legend
Legend

Always keep forgetting about it, but so important, for sure.

Best,

Andy

0 Kudos
PD_localuser
Explorer

thanks. we have 87% occupation in /var/log

tomorrow i´ll work to free some space, the disk is about 2TB and approx 230GB free, 230Gb appear are enough but i´m not sure if this 87% has priority

0 Kudos
the_rock
Legend
Legend

I would do below from expert:

find /var/log -size +900M

see what files you can delete. You can also replace 9 with any other digit, so say 400M is 400Mbs

Best,

Andy

0 Kudos
PD_localuser
Explorer

Hi everybody

i update my case

Last week we delete several backup files, schedule backup was bad configuration and since 2021 every day this schedule was execute, after remove space in /var/log/ decrease to 40%  approx 

after, we execute comand fw logswitch, from thursday to now logs are visible without problems, now we need trie recover old logs before thursday.

Any tips for this?

0 Kudos
the_rock
Legend
Legend

So if you do fw logswitch, that would technically rotate the existing log, so say its currently size 1 GB (just making that up), it would default it to 0 and start there, so then would rotate again by default at midnight or when it reaches 2 GB in size (whichever would come first).

Now, as far as older logs, can you navigate to $FWDIR/log directory and see if they are there?

Best,

Andy

you can do this...example from my lab.

 

[Expert@azurefw:0]# cd $FWDIR/log
[Expert@azurefw:0]# ls -lh *.log
-rw-rw---- 1 admin root 19K Jan 6 00:00 2024-01-06_000000.log
-rw-rw---- 1 admin root 9.5M Jan 8 09:30 2024-01-08_093043.log
-rw-rw---- 1 admin root 8.2K Jan 8 09:30 fw.log
-rw-rw---- 1 admin root 1.8M Feb 26 14:10 tracker.log
[Expert@azurefw:0]#

0 Kudos
PD_localuser
Explorer

unfortunately only have a pair .log files from after switch

i need investigate more but i this moment can see other files type or audit logs files since 2023

i hope wasn´t are delete or move

0 Kudos
the_rock
Legend
Legend

If you dont see them there, not sure changing anything with indexing may help : - (. Maybe you can ask via TAC case, see what they say.

Best,

Andy

0 Kudos
Amir_Senn
Employee
Employee

If they don't appear in the log directory they were probably deleted. Under some conditions audit log files will not be deleted - this is because they're insignificant in size in comparison to traffic log file.

You can check $FWDIR/log/fwd.elg , search for the following outputs:

CCyclicLogging::_moveFile: moving file: 2015-12-02_000000.log, for deletion dir: /opt/CPsuite-R80/fw1/log//cl_del
FWLOG_MAINTENANCE - RemoveFilesFromCLDir: removing file: 2015-12-02_000000.log from dir: /opt/CPsuite-R80/fw1/log//cl_del

Kind regards, Amir Senn
0 Kudos
Nadezhda
Contributor

Hello!

I'm sharing a solution that has helped us and hopefully will help someone else.

1. stop the logging and monitoring processes:
#evstop
2. Move FetchedFiles:
#mv $INDEXERDIR/data/FetchedFiles $INDEXERDIR/data/FetchedFiles.backup
3. Start the logging and monitoring processes::
#evstart

(1)
the_rock
Legend
Legend

Thats super useful, thank you!

Andy

0 Kudos
Amir_Senn
Employee
Employee

FetchedFiles is the file that monitors what log files are indexed and status.

When you remove the file it will create it again. If you don't remove the indexes, what is the affect of this?

Kind regards, Amir Senn
the_rock
Legend
Legend

Hey Amir,

Are you saying this process should not be followed or more along the lines backup everything before doing it?

Best,

Andy

0 Kudos
Amir_Senn
Employee
Employee

IMO, more likely that restarting the logging processes helped more. I think we might have racing conditions in some scenarios that restarting the processes solves.

I played with it in my lab. Looks like this doesn't behave well:

I had no live traffic, only recorded traffic I inject. After injection firewallandvpn core showed directory size of 128Mb and other-smartlog core was 48Mb (stable). After evstop ; removing FetchedFiles ; evstart - after restart FetchedFiles doesn't have information about the log files so indexer will index backwards as many days as defined in definitions and the size increased on both cores directories:

firewallandvpn 128Mb -> 142Mb -> 176 Mb -> 218Mb -> 152Mb

other-smartlog 48Mb -> 74Mb

So we know that it creates more indexes and since we can also see a decrease it might consolidate some of them but either way it keeps more indexes than needed. IDK what this can cause, from nothing to not delete indexes properly to other behaviors.

Usually if we look on scenarios in which we delete the FetchedFiles we also delete the indexes - no double sets of indexes.

Kind regards, Amir Senn
the_rock
Legend
Legend

K, fair enough, thank you.

Andy

0 Kudos

Leaderboard

Epsum factorial non deposit quid pro quo hic escorol.

Upcoming Events

    CheckMates Events