SmartLog Issue

[Raj_Khatri]

We are facing an issue with logs not working when connected to our MDS.  SmartLog no longer works and displays the message “SmartLog is Initializing…”  The issue started while we were running Take 87 which was stable for quite some time.  We can see logs when connected directly to the MLM via SmartConsole.

We have rebooted, performed smartlogstop/smartlogstart several times, without success.  We have an open case with TAC who suggested upgrading to Take 118 which didn’t help.  They are reviewing other logs in the meantime.

Environment

• MDS – R80.20 Take 118 (Smart1-3150)
• MLM – R80.20 Take 118 (HP Open Server)
• SME – R80.20 Take 118 (HP Open Server)

Looking to see if anyone has run into this issue and other possible suggestions to look at and try.

Thanks

[PhoneBoy]

Have you tried accessing using a web browser e.g. https://CMA-IP/smartview

[Raj_Khatri]

I just tried and get "Query Failed" message when viewing any of the data.  

[PhoneBoy]

This would most likely have to be debugged by the TAC as some process is clearly wedged/not starting.

[Raj_Khatri]

TAC wasn't much help.  Luckily I was able to work with someone directly in R&D to fix the issue.  There was a corrupt file which was affecting RFL.

[Teet]

Hello, what file was corrupt? We might have similar or even same issue, would be of help 🙂

[Raj_Khatri]

Sorry for the late reply, here was the RCA sent to us - 

The RFL process was not working with the correct port (18218).

When we checked netstat over port 8218 we did not see the RFL:

============

[Expert@harry]# netstat -nap | grep 8218

tcp 0 0 127.0.0.1:8218 127.0.0.1:40392 ESTABLISHED 17289/java

tcp 0 0 127.0.0.1:8218 127.0.0.1:37118 ESTABLISHED 17289/java

tcp 0 0 127.0.0.1:37118 127.0.0.1:8218 ESTABLISHED 17375/log_indexer

tcp 0 0 127.0.0.1:40392 127.0.0.1:8218 ESTABLISHED 17375/log_indexer

============

When the issue was resolved we were able to see below output:

============

[Expert@harry]# netstat -nap | grep 8218

tcp 0 0 127.0.0.1:8218 0.0.0.0😘 LISTEN 17289/java

tcp 0 0 127.0.0.1:8218 127.0.0.1:40392 ESTABLISHED 17289/java

tcp 0 0 127.0.0.1:8218 127.0.0.1:37118 ESTABLISHED 17289/java

tcp 0 0 127.0.0.1:37118 127.0.0.1:8218 ESTABLISHED 17375/log_indexer

tcp 0 0 127.0.0.1:40392 127.0.0.1:8218 ESTABLISHED 17375/log_indexer

============ 

When we checked RFL.log file we could see below information:

============ 

2020-02-05 15:07:08,896 ERROR [main] org.apache.activemq.broker.BrokerService.start:571 - Failed to start Apache ActiveMQ([RaphaelBroker, null], org.apache.activemq.protobuf.InvalidProtocolBufferException: Protocol message contained an invalid tag (zero).)

2020-02-05 15:07:08,902 INFO [main] org.apache.activemq.broker.BrokerService.stop:714 - Apache ActiveMQ 5.9.0 (RaphaelBroker, null) is shutting down

 2020-02-05 15:07:09,257 ERROR [main] 2020-02-05 15:07:09,257 ERROR [main] com.checkpoint.core.LogCore.runJmsServer:48 - JMS server execution failed!

org.apache.activemq.protobuf.InvalidProtocolBufferException: Protocol message contained an invalid tag (zero).

               at org.apache.activemq.protobuf.InvalidProtocolBufferException.invalidTag(InvalidProtocolBufferException.java:48)

               at org.apache.activemq.protobuf.CodedInputStream.readTag(CodedInputStream.java:75)

 ============ 

We could see that Apache ActiveMQ was not able to start.

The cause of this issue was corruption at /opt/CPrt-R80.20/tmp/activemq-data/localhost/KahaDB/  directory.

As soon as we removed the files from the above directory and restarted the services, the smart log started working.