topic Re: Management problem with GAIA OS in General Topics

Management problem with GAIA OS

Matlu — Tue, 22 Aug 2023 23:42:46 GMT

Hello, everyone.

We have a HA SMS environment, which is in version R81.10, both on Open Server.

Currently the HA SMS passive equipment, has very big problems SLOWNESS.
Every time we try to access by CLI to the equipment
(We enter by Putty, or some other terminal, the "prompt" to enter the username and password, takes an eternity to appear).

The active member of HA SMS has no problem.

When accessing after a long time to the passive equipment, the equipment fails to accept the commands you try to type, and/or if it accepts the command, and you try to have an output of the same, such as commands like (cpview, top, free -m), the equipment takes forever to display the output of the command.

We have noticed that the CPU seems to be "stressing out" and we believe this is the reason for the problem.

In this scenario, is it advisable to FRESH INSTALL the device, and reinstall the GAIA OS?

I have the impression that the OS has been damaged for some reason.

Do you have any opinion and/or experience similar to the above?

Thanks for your comments.

Re: Management problem with GAIA OS

the_rock — Wed, 23 Aug 2023 00:37:29 GMT

You can never go wrong with fresh install, that will always work, for sure. But, any idea whats causing the problem? Any process consuming high cpu/memory?

Andy

Re: Management problem with GAIA OS

Timothy_Hall — Wed, 23 Aug 2023 02:16:41 GMT

cpstat -f os sensors

Sounds like a classic case of a CPU fan failure, and the CPU downclocking itself to a fraction of the normal clock rate. Could also possibly be a slow/nonresponding DNS server issue, use nslookup or dig to perform a couple of website name lookups to IP addresses. Slow? Timing out? Does this system have a RAID array? Could also be a symptom of a degraded RAID array or a hard drive that has partially failed. Check /var/log/messages.

Re: Management problem with GAIA OS

Matlu — Wed, 23 Aug 2023 02:54:30 GMT

Buddy,

The system is so slow, it is difficult to detect the fault.

When we applied the "cpview" command, we waited a long time to get results, and observed that the CPUs were "too high".

Do you have any command similar to "top" that can catch the exact process that may be consuming all the resources.

We are seriously thinking of "Formatting" the Gaia OS.

😕

Re: Management problem with GAIA OS

the_rock — Wed, 23 Aug 2023 03:13:12 GMT

ps -auxw

top

free -g

Re: Management problem with GAIA OS

Daniel_3 — Wed, 23 Aug 2023 06:20:39 GMT

Hi,

I had similar issues on an open server and the problem was a faulty RAID battery or cache battery. After a battery replacement the system went back to normal state.

Re: Management problem with GAIA OS

Matlu — Wed, 23 Aug 2023 13:27:41 GMT

Hello,

Sorry, but this RAID has to do directly with the OPEN SERVER, right?

Our GAIA OS, is on a VMWARE ESXi

This RAID is something that should be checked at the server level as such?

Regards

Re: Management problem with GAIA OS

Timothy_Hall — Wed, 23 Aug 2023 13:34:32 GMT

It would have been helpful to know you are running in VMWare.

Run the top command. What do the wa (waiting for I/O) and st (steal) percentages look like? If they are constantly nonzero or even very high values, that indicates disk I/O contention (wa) with other virtual machines due to oversubscription of the disk path, or the VM being denied access to the CPU when it wants it due to oversubscription of CPU resources (st) by the hypervisor. Also see here for overall best practices:

sk104848: Best Practices - Performance Optimization of Security Management Server installed on VMware ESX Virtual Machine

Re: Management problem with GAIA OS

Matlu — Wed, 23 Aug 2023 13:53:08 GMT

I understand the above, but I have a question.

Reinstalling Gaia OS, can it be a definitive solution for our problem?

Or this that now we have as a problem, can remain, installing the GAIA OS again from 0?

Regards

Re: Management problem with GAIA OS

_Val_ — Wed, 23 Aug 2023 14:03:30 GMT

Why would you reinstall in the first place?

Re: Management problem with GAIA OS

Matlu — Wed, 23 Aug 2023 14:40:56 GMT

It is an extreme decision.

A case has been opened with CT, but they are unable to "find" a probable cause-root.

That is why, given the customer's need for the solution to work well, we are almost determined to reinstall the GAIA OS, believing that this will "correct" the problem.

Re: Management problem with GAIA OS

the_rock — Wed, 23 Aug 2023 14:48:06 GMT

Are you able to send us output of command @Timothy_Hall asked, as well as below:

top

ps -auxw

free -g

screencap of cpview output of memory/cpu

Re: Management problem with GAIA OS

the_rock — Wed, 23 Aug 2023 14:50:43 GMT

Also, send us output of cpstat os -f all

Re: Management problem with GAIA OS

Timothy_Hall — Wed, 23 Aug 2023 16:27:04 GMT

If the hypervisor environment is the source of the problem, reloading Gaia in the same VM won't help. Creating a brand new VM and reloading it might help if the hypervisor was denying/restricting resources to the old VM for some reason.

Re: Management problem with GAIA OS

Matlu — Wed, 23 Aug 2023 17:58:35 GMT

Hello, again.

This is information I have been able to collect from the SMS that is failing. It is so slow to enter the SMS prompt, that it has taken me a long time to collect the information.

With CPview, I see that the CPU is flying, obviously I can't capture how the value changes constantly, but at least the images will help to show the error I am referring to.

I hope you can give me some opinion on where you think the error might be.

Re: Management problem with GAIA OS

the_rock — Wed, 23 Aug 2023 18:18:25 GMT

Its way more clear now, thanks for providing those things. I found notes from 2 years ago where we had EXACT same issue with a customer and their CP mgmt server in Azure (NOT S1C) and after having TAC case for more than a month and them running who in the world knows how many debugs and reviewing files, we ended up restoring recent backup and that permanently fixed the issue.

You may ask how that fixed the problem? Answer is...I HAVE NO IDEA lol

At the end of the day, TAC could not tell us either and at that point, Im sure customer did not care either, as long as issue was fixed, which it was. Personally, I still dont understand why java was causing such high CPU, because there were no changes made, no new version or jumbo installed, so there was zero logic in any of it.

Also, worth pointing out that in their case, there was lots of free memory, same as with your management.

Re: Management problem with GAIA OS

Matlu — Wed, 23 Aug 2023 18:28:39 GMT

Heck, then it is almost 99% more likely that with a new installation from 0 on the VMWARE ESXi, with respect to GAIA OS, we will solve the problem.

Well, I think it is the most "healthy" for our scenario 😞

In our case, this SMS that has problems is the Standby (We have the environment of a SMS HA).

Is there a need to break from the SmartConsole, the SMS HA, in order to reinstall from 0 the damaged MGMT?

Thanks for your comments.

Re: Management problem with GAIA OS

Timothy_Hall — Wed, 23 Aug 2023 18:40:46 GMT

Your CPUs are not busy at all, but they are blocked 70-80% of the time waiting for access to the disk (I/O Wait). You have plenty of memory, so either the disk path is somehow bad/corrupt or postgres is going bonkers hitting the database due to corruption, hard to tell from the list of processes. Please provide output of iotop. Talk to your VM person and ask them how utilized the disk path is that is assigned to this VM.

Re: Management problem with GAIA OS

the_rock — Wed, 23 Aug 2023 18:41:42 GMT

Bro, command is cpstat os -f all, can you send that as well?

Andy

Re: Management problem with GAIA OS

Matlu — Wed, 23 Aug 2023 20:49:39 GMT

Cannot obtain information with the command "cpstat -f os all".

I applied the command "iotop", but you will understand, that the values of this command, change continuously, I hope that what I capture, serves.

Thanks for your comments and help