Maestro Orchestrator MHO-140 & MHO-175 - /var/log/gaia_api_server.log and multiple Hotfix Takes

[moritz_r]

Hello There,
I have several customers where Maestro is used. The current Version is R81.10 and multiple Hotfix Takes are installed on MHO.
Now I got an error message on the Orchestrator when I want to import a new Jumbo Hoftix via CPUSE >> "Error: not enough space. The session will be terminated. Press OK to proceed."

When I check the "show asset all" command on the MHO Appliance, it displays a 120GB SSD.
The "df -h" shows me, for /var/log/ partition we got only 20GB and around 2GB are left!

So next I checked all big Files on the /var/log/ partition to find out the root cause.
I was then wondering about the /var/log/gaia_api_server.log File, which is continuously growing and takes nearly 8GB of the 20GB Capacity from the Partition.
Unfortunately, I can't find any information about that log-file. Can someone explain the background of the "gaia_api_server.log" and if it can be deleted?

Is it normal that this log is growing up to 8GB? why there is no log rotation for this log or something like a scheduled job that cleans it up? What will happen with the MHO, if the /var/log/ is completely full because of the continuously growing logfile?

In general, I was wondering about how to clean up the Gaia /var/log/ Partition on a Maestro Orchestrator and how it will differ from a non-Maestro Gaia.

Best regards

Moe

EDIT: I just found out that the Log rotation does not work anymore. I will open a TAC Case:

[Expert@MHO-140_1:0]# ls -alh /var/log/gaia_api_server.log*
-rw-r--r-- 1 admin config 7.5G Apr 19 16:57 /var/log/gaia_api_server.log
-rw-rw---- 1 admin root 1.1M Feb 22 16:39 /var/log/gaia_api_server.log.1
-rw-rw---- 1 admin root 1.1M Feb 22 14:39 /var/log/gaia_api_server.log.10
-rw-rw---- 1 admin root 1.1M Feb 22 16:25 /var/log/gaia_api_server.log.2
-rw-rw---- 1 admin root 1.2M Feb 22 16:12 /var/log/gaia_api_server.log.3
-rw-rw---- 1 admin root 1.2M Feb 22 15:59 /var/log/gaia_api_server.log.4
-rw-rw---- 1 admin root 1.2M Feb 22 15:45 /var/log/gaia_api_server.log.5
-rw-rw---- 1 admin root 1.1M Feb 22 15:32 /var/log/gaia_api_server.log.6
-rw-rw---- 1 admin root 1.1M Feb 22 15:19 /var/log/gaia_api_server.log.7
-rw-rw---- 1 admin root 1.1M Feb 22 15:05 /var/log/gaia_api_server.log.8
-rw-rw---- 1 admin root 1.2M Feb 22 14:52 /var/log/gaia_api_server.log.9

[Majd_Sharkia · Accepted solution]

Hi, you can safely remove these log files, we will consider adding gaia_api logs to the rotation infra to avoid this in the future.

Thanks.

[moritz_r · Accepted solution]

Hey @Majd_Sharkia 
Thanks for your response.
Because yesterday I wasn't even able to uninstall any Hotfix Take anymore, I have performed the following Workaround:

Create a system-backup and snapshot in the first place.
After this, I removed the 8GB gaia_api_server.log.
Then I created a new clean/empty gaia_api_server.log.
Because it didn't take effect on the "df -h" output, I restarted the MHO.
After MHO reboot, more disk space was available again on the 120GB SSD and I was able to delete the older Patches.

Best Regards

[moritz_r]

Hello There,

I tracked this a few weeks since now after I have uninstalled Jumbo Hotfix Take 66 & 81, and installed Take 95 only. After MHO rebooting the issue was solved for some reason. So I guess it's fixed but not listed in the release notes.

Cheers

[Bulat · Accepted solution]

I found out that this log file was removed from rotation list

take94
[Expert@MHO-01:0]# log_start list | grep gaia
   57) gaiaRest.log                             10485760         10
   97) gaia_api_server.log                       1048576         10
   99) gaia_api_taskmanager.log                  1048576         10

take95
[Expert@MHO-01:0]# log_start list | grep gaia
   57) gaiaRest.log                             10485760         10

I added new line to file  /etc/cpshell/log_rotation according to sk36798 

[Enliven89]

sk113241 is better suited

[Jones]

Hi,

In jumbo take 110 the gaia_api entries are back again!