Unable to login to Smartconsole

[medtiti92]

Hi, 

After the disk was full in SmartConsole, we delete some files to make free espaces, but sinces, we are unable to login to the management via Smart Console : 

when i make api status, i have this ouput : 

[Management]# api status
2023-Dec-11 11:48:20 - ERROR: Log directory not found at ${NGM_WEB_API_LOG_DIR}.
Exiting...
Could not write to api_sh.elg: [Errno 2] No such file or directory: '/opt/CPsuite-R81.10/fw1/log/api_sh.elg'

Please help 

[Amir_Senn]

Same output for "api start"?

Which partition was full? Other partitions still have unused space?

Usually the partition that gets filled up is /var/log/ but it should have a maintenance mechanism to preserve free space, so it's worth while to check.

Please add version+JHF information as well as df -kh output.

[medtiti92]

Hi Amir,

Thanks for your reply. Yes it is same output "api start", the /var/log was very full, after deleting some folders and files, the size decrease but i am unable to login to the smartconsole

[Management]# df -kh
Filesystem                                                      Size                 Used                    Avail                   Use%                 Mounted on
/dev/mapper/vg_splat-lv_current              32G                   26G                     6.9G                     79%                             /
/dev/sda1                                                       291M                43M                     233M                   16%                            /boot
tmpfs                                                              7.8G                  28K                     7.8G                       1%                           /dev/shm
/dev/mapper/vg_splat-lv_log                     100G                 52G                     49G                       52%                          /var/log
cgroup                                                            7.8G 0                7.8G                                                 0%                           /sys/fs/cgroup

[the_rock]

Sounds like you deleted some sort of database files, which would 100% cause this sort of issue. I reclal few years ago this exact thing happened with a customer and once we restored a backup, all worked, we just had to reinstall the jumbo hotfix.

Do you happen to have working backup from the appliance?

Andy

[the_rock]

Just to confirm, I assume you tried cpstop; cpstart and reboot?

Andy

[Tal_Paz-Fridman]

Does $FWDIR/log exist? The error suggests this directory is missing.

As the_rock suggested does reboot help?

[the_rock]

Thats an excellent point @Tal_Paz-Fridman . After examining the api status output, it would appear to be the case.

Andy

[the_rock]

Here is another suggestion...IF indeed $FWDIR/log dir is missing, and you dont have a workinmg backup (i hope you do though), one thing to try might be copy the content from working mgmt with the same version to $FWDIR/log. It might be a bit risky doing that, but at this point, things dont look great, but lets be positive.

Andy

[medtiti92]

hello guys,

i did cpstop cpstart, i made reboot, nothiing changed. I don't have backup, but i think that i have another management but it isn't same version(current version : R81.10, other version : R80.30)

[Tal_Paz-Fridman]

Did you check the existence of files in $FWDIR/log ?

If the workarounds we suggested do not work you will need to contact TAC. 

[medtiti92]

it exist, but no files log insides $FWDIR/log ... see below :

[Tal_Paz-Fridman]

That's the $FGDIR/log folder (for QoS)

The one you need to look at is with fw1 -->  /opt/CPsuite-R81.10/fw1 ... if that is also missing it can explain a lot of the problems you are having.

[the_rock]

As Tal said, not the same folder. I would honestly work with TAC on this, because if reboot does not help and error is still the same for api status, it needs further investigation. My gut feeling tells me you most inadvertently deleted the database file from root dir...

Andy

[medtiti92]

Sorry, please see below the content of /opt/CPsuite-R81.10/fw1 : 

[Management]# cd /opt/CPsuite-R81.10/fw1/
[Management]# ls
ICS cyrus_sasl_install sclient
Python database scripts
SU default_cpm_db spool
Solr dlp sql_scripts
Solr7 doc sql_scripts_R80_20
amw fast_policy srpkg
amw_kss hash state
api ips sup
appi lib te
aspam_engine_install lib64 teCurrentPack
av log teDefaultPack
bin modules temp
boot nac tmp
c-icap nacportal tmpCKP_mutex::SIC_DB
cipher_util ngm tmpcpm.lock
cisco ngm_upgrade uf
conf oracle_oi webconsole
cpdiag orig well
cpextensions parsers wylis_tmp
cpm-server policy
cpm_client_stats postfix_install

[the_rock]

This is my working R81.20 lab mgmt...you are definitely missing some stuff.

Andy

[Expert@CP-management:0]# cd $FWDIR
[Expert@CP-management:0]# pwd
/opt/CPsuite-R81.20/fw1
[Expert@CP-management:0]# ls
ICS api c-icap cpm_client_stats hash nacportal postfix_install state tmpcpm.lock
Python appi cipher_util cyrus_sasl_install ips ndr sclient sup uf
REVERT_BACKUP_TMP aspam_engine_install cisco database lib ngm scripts te webconsole
SU av conf default_cpm_db lib64 oracle_oi spool teCurrentPack well
Solr7 bin cpdiag dlp log orig sql_scripts teDefaultPack wylis_tmp
amw bkp-jars cpextensions doc modules parsers sql_scripts_R80_20 tmp zp.js
amw_kss boot cpm-server fast_policy nac policy srpkg tmpCKP_mutex::SIC_DB

[medtiti92]

Yeah i see well, i don't know what doing, i will chearch someone who has same version and copy his content 

[the_rock]

Just be careful with that as I mentioned, but I dont see that making it worse than what it is, as long as you backup existing content...its Hail Mary attempt as they say, but personally, I would go for it, but, thats just me.

Andy

[_Val_]

Do you have any backup to restore your management from?

[medtiti92]

No unfortunately i don't have any backup.

[the_rock]

I hate to say this, but without the backup, chances this can be fixed are not that great. We had literally same issue few years ago and case even went to escalation team and luckily, we found a working backup that helped us solve the issue once it was restored.

Without it, I really dont know how it can be solved easily...

Andy

[medtiti92]

Thanks for your reply @the_rock, i really appreciate. i will look for this backup and make some simulation to find à solution for this issues. back to you if i get some solution guys

[the_rock]

Of course, any time! Be free to message me directly, we can set up remote session, happy to have a look.

Andy

[medtiti92]

hi rock, i find a backup in /var/log/CPbackup/backups, can i restore it using clish 

[the_rock]

CP-TEST-FW> restore backup local
Backup of the system configuration to a folder in the local appliance
CP-TEST-FW> restore backup local

Just make sure you input right path there and off you go.

Andy

[medtiti92]

Good, let me make this and back to you soon

[the_rock]

Make sure obviously to include exact file name, so it knows what to restore from...so something like restore backup local /var/log/CPbackup/backups/backupfilename.tgz

Kind regards,

Andy

[medtiti92]

i did. it restored. but i have issues, i can't able to login to smartconsole, because the api is down :

[medtiti92]

[Expert@Smartcenter:0]# api start
2023-Dec-13 18:16:36 - ERROR: Cannot start API because CPM is not ready
Exiting...

[medtiti92]

[Expert@Smartcenter:0]# api status

API Settings:
---------------------
Accessibility: Require local
Automatic Start: Unknown

Processes:

Name State PID More Information
-------------------------------------------------
API Stopped 0
CPM Stopped 0
FWM Stopped 0
APACHE Started 7362

Port Details:
-------------------
JETTY Internal Port: 0
JETTY Documentation Internal Port: 0
APACHE Gaia Port: 4434 (a non-default port)
When running mgmt_cli commands add '--port 4434'
When using web-services, add port 4434 to the URL

Profile:
-------------------
Machine profile: 15800-24800 with SME or Dedicated Log Server
CPM heap size: 1536m

--------------------------------------------
Overall API Status: The API Server Is Not Running!

[medtiti92]

[Expert@Smartcenter:0]# api status

API Settings:
---------------------
Accessibility: Require local
Automatic Start: Unknown

Processes:

Name State PID More Information
-------------------------------------------------
API Stopped 0
CPM Stopped 0
FWM Stopped 0
APACHE Started 7362