Cannot access logs

[Tomas_Hamrle]

Hello, I have a problem in Logs & Monitor on OpenServer R80.10. When I want to open logs, "Loading SmartView" shows and after a while "Error loading tab, Error: TimeOut, An operation timed out." I also cannot access the WebUI (This site can't be reached).

Is there any way to fix this?

Thank you

[Jerry]

have you confirmed you've got log under $FWDIR/log and $CPDIR/log folders?

what about if you open SmartView Tracker independently and point login towards same CP logging appliance/CLM - can you see logs loading or not at all? have you checked cpwd_admin list and CPD and FWD processes running?

[Jerry]

check content of /opt/CPsuite-R80/fw1/log folder - do you any files with .log/.ptr/.logptr/.log_stats extensions?

also, can you double check what is in $CPDIR/log folder? you should have there following files:

-rw-r--r-- 1 admin config 79188 Aug 22 16:46 cp_conf.elg
-rw-r--r-- 1 admin root 1173 Jul 11 16:20 cpapache_postinstall.elg
-rw-rw-r-- 1 admin root 690157 Aug 29 16:56 cpd.elg
-rw-rw---- 1 admin root 20971647 Aug 27 06:58 cpd.elg.0
-rw-r--r-- 1 admin root 83341 Aug 29 10:33 cprid.elg
-rw-r--r-- 1 admin root 880 Aug 22 16:34 cprid_wd.elg
-rw-rw---- 1 admin root 7269 Aug 22 16:36 cpstart.log
-rw-rw---- 1 admin users 3667 Aug 14 11:10 cpview_stats_live
-rw-rw---- 1 admin root 114570 Aug 29 16:55 cpwd.elg
-rw-rw---- 1 admin root 94 Jul 11 16:13 fw1_components.log
-rw-r--r-- 1 admin config 1177476 Aug 29 16:35 hservice.elg
drwxrwx--- 2 admin root 4096 Jul 11 16:12 log
-rw-rw---- 1 admin root 0 Jul 11 16:13 mpclient.elg
-rw-rw---- 1 admin root 0 Jul 11 16:13 mpdaemon.elg
-rw-rw-r-- 1 admin root 15817 Aug 29 10:33 postgresqlcmd.elg

*** if not all - most of them, but the most important are the BOLD one.

please confirm.

Jerry

[Kfir_Dadosh]

Hi,

Seems like a connectivity to the SmartView web server:

1. Can you access the Gaia settings portal?

2. Is it a standalone server (Mgmt + FW)?

3. Validate that SmartView is up and running using "cpwd_admin list"

4. Restart SmartView by running:
      $RTDIR/scripts/stopSmartView
      $RTDIR/scripts/startSmartView

5. Make sure SmartView is listening on port 8082

6. Make sure Apache is listening on port 443

Kfir Dadosh

[Tomas_Hamrle]

When I opened SmartView Tracker independently I could see all the logs. 

I checked the $FWDIR and $CPDIR and there are many files with *.log / *.logptr extensions.

Under $CPDIR I got:

-rw-r--r-- 1 admin config 16808 Aug 3 16:00 cp_conf.elg
-rw-r--r-- 1 admin root 1157 Aug 3 10:38 cpapache_postinstall.elg
-rw-rw---- 1 admin root 6773 Aug 11 12:29 cpconfig.elg
-rw-rw---- 1 admin root 8068814 Aug 30 08:00 cpd.elg
-rw-r--r-- 1 admin root 132650 Aug 28 14:21 cprid.elg
-rw-r--r-- 1 admin root 1176 Aug 11 12:30 cprid_wd.elg
-rw-rw---- 1 admin root 7274 Aug 11 12:32 cpstart.log
-rw-rw---- 1 admin root 3247 Aug 11 16:21 cpview_stats_history
-rw-rw---- 1 admin root 3243 Aug 11 16:18 cpview_stats_live
-rw-rw---- 1 admin root 226155 Aug 30 08:17 cpwd.elg
-rw-rw---- 1 admin root 94 Aug 3 10:36 fw1_components.log
-rw-r--r-- 1 admin root 1594 Aug 11 15:00 hservice.elg
drwxrwx--- 2 admin root 4096 Aug 3 10:36 log
-rw-rw---- 1 admin root 0 Aug 3 10:36 mpclient.elg
-rw-rw---- 1 admin root 0 Aug 3 10:36 mpdaemon.elg
-rw-rw-r-- 1 admin root 1026 Aug 3 13:17 pgsqlconfig.elg
-rw-r--r-- 1 admin root 18824 Aug 30 02:00 postgresqlcmd.elg

Today I tried to see logs from SmartConsole and now it's working. I can see all the logs. Just it is a little slow - when I open a new tab the red spinning circle appears and after a while the tab loads normally.

It is Standlalone installation for SecurityCheckup.

SmartView under cpwd_admin list is running.

I confirm that Apache is listening on 443 and SmartView is listening on 8082.

But still I cannot access WebUI

[Jerry]

what is the spec of that OpenServer Tomas? how many cores, memory and disk space?

looks like a resourcing problem to me but I might be wrong.

If I were you I would focus on reviewing the resource allocation for the Core's/Mem/HDD and NIC's.

Seem to me that the bottle neck you've got on the performance of the appliance (all-in-one I guess).

[Tomas_Hamrle]

It is HP ProLiant DL 20 Gen9, 4xCPU, 16GB RAM, 360 GB SSD (~290GB for logs, 75% used). I know that it is not the best HW, but it should be enough. When I installed it was fast and now, after ~20 days of collecting logs, it is very slowly. Now it is slow but it's working. I will do some more reviewing.

[Jerry]

can you make in expert mode following and paste it here?

uname -a

top -H (and press 1 when done) -> make a screen of it (with all the cores and mem usage)

I think we could get down to the root cause of your issues but let's make sure your devices is compliant with your hardware, pls. run in clish:

*> show installer packages 

*> show installer status

*> show version os build
*> show version os kernel
*> show version os edition

Cheers

[Tomas_Hamrle]

Hello,

uname -a
Linux checkpoint 2.6.18-92cpx86_64 #1 SMP Fri Apr 7 13:55:39 IDT 2017 x86_64 x86_64 x86_64 GNU/Linux

top -H

show installer packages
** ************************************************************************* **
** Hotfixes **
** ************************************************************************* **
Display name Status
R80.10 Jumbo Hotfix Accumulator General Availa... Installed
R80.10 Jumbo Hotfix Accumulator General Availa... Available for Download
R80.10 SmartConsole Build 005 Available for Download
HOTFIX_R80_10 Installed (Legacy)
** ************************************************************************* **
** HFAs **
** ************************************************************************* **
Display name Status
R80.10 SmartConsole for Windows Installed
R80.10 Jumbo Hotfix Accumulator SmartConsole B... Installed
** ************************************************************************* **
** Majors **
** ************************************************************************* **
Display name Status
R76 Fresh Install Available for Download
R77.10 Fresh Install and Upgrade from R76 Available for Download
R77.20 Fresh Install and Upgrade from R75.4X /... Available for Download
R77.30 Fresh Install and Upgrade from R75.4X /... Available for Download
R77 Fresh Install and Upgrade from R71.50 / R7... Available for Download

show installer status
Agent: enabled
Build number: 1298 (agent build is up to date)
Network connection: connected
Update from cloud: last updated on Thu Aug 31 06:39:10 2017
License: valid

show version os build
OS build 421

show version os kernel
OS kernel version 2.6.18-92cpx86_64

show version os edition
OS edition 64-bit

[Jerry]

Hi

seems you've got exactly the same kernel params like myself, see nothing wrong with it though but what is worrying me a little is how your SSD is engaged with 4 Core's (2 of which are on 14% on your screenshot) and if you say you've got 16GB or RAM --- how much swap is being utilized.

One question then - what is the current "uptime" of your machine and if possible can you show us again the "top -H" after it boots (like after first 5-10 minutes of work)?

To me it looks very suspicious how your GAIA behave knowing it's on SSD drive with 16GB of RAM and 4 Core CPU. I think that this deserve decent investigation from the kernel side unless you've already played yourself with SIM affinity and NIC/CORE/RAM associations did you?

I believe your issues are due to the performance not configuration issues therefore solving it would be quite complicated.

It happened to me that for example 16GB or RAM on my GAIA was 2 x 8 GB but RAM modules differed - this made GAIA sluggish like a hell

Also, 360GB SSD - what is that? Crossair? Samsung? or maybe Hitachi one? Just out of my curiosity as you know ALL SSD's are different and ALL behaving differently depending on speed factors and what NAND modules are in use.

I'd be very happy to help going forward but I guess we need a little bit more t-shooting here if you don't mind

Jerry

[Tomas_Hamrle]

Hello,

thanks for your interest 

There is 17GB of swap (4GB used). The current uptime is 20 days. Unfortunately I cannot reboot the system right now so I cannot provide the "after-boot top -H". I did not configure anything about kernel. I cannot say for sure what SSD is inside because I did not plug it in or unplug it.

For now it is working as it should be. Both SmartConsole and WebUI. I upgraded Java to the latest version on my computer, maybe it has something to do with that.

Tom

[Daniel_]

Hi,

in top there 33% Software Interrupts (per CPU). That's why the load is so high.

Can you post

cat /proc/softirqs

and also the HW interrupts

cat /proc/interrupts

?

Thanks!

[Tomas_Hamrle]

Hello,

[Daniel_]

Damn! I hate this old GAiA System. /proc/softirqs is available in old 3.6 kernel but not in the very old kernel from GAiA.

I have no ideas anymore. I think you should open a SR about this issue.....

[Jerry]

I thoughts that those CPU2 and CPU3 has something wrong they act about like this ... those 14.3% under the load like this seemed suspicious to me as well but indeed I've overlooked SI factor. Spot on Daniel!