Logging not properly working after Management Server Upgrade from R77.30 to R80.40

[ChrisG]

Dear Checkmates,

we have the following situation:

- Management Server with R77.30 and a VSX Cluster with R77.10 on open servers.

We are planning to upgrade the management server to R80.40 first with migration and moving to a new server and then, of course, upgrade the VSX modules to R80.30 in a later step.

The migration and upgrade procedure is running fine and installing policy and monitoring the R77.10 VSX Gateway and its VS is also ok.

We have the following issue with the logging:
The VSX VS modules are logging to the new management, but the log format seems to be out of order for some fields like "Information", this is viewable with the R80.40 SmartView Tracker. As a result the log indexer is not able to parse the ongoing received logs from the VSX and writes tons of logs into log_indexer.elg with high CPU load. The SmartLog is without function, but this is just a result of the failures before. It seems to me as if the management server is using the wrong log parsing for the VSX R77.10 modules log format it receives.

Did anybody came across this problem too?

I did not found anything in the support database yet.

Afaik the log format has been changed between R77.10 and R77.30.

log_indexer.elg:
....
[9 Feb 20:16:07] JavaBinCodec::unmarshall Invalid version (expected ^B, but <) or the data in not in 'javabin' format
[9 Feb 20:16:07] SolrClient::Implementation::ParseJavaBinReply Bad Java bin format
[9 Feb 20:16:07] RFLIndexDoer::sendToSolr: Send error occurred(775,775). Tried 1 out of max 2
[9 Feb 20:16:07] JavaBinCodec::unmarshall Invalid version (expected ^B, but <) or the data in not in 'javabin' format
[9 Feb 20:16:07] SolrClient::Implementation::ParseJavaBinReply Bad Java bin format
[9 Feb 20:16:07] RFLIndexDoer::sendToSolr: Send error occurred(775,775). Tried 2 out of max 2
[9 Feb 20:16:07] RFLIndexDoer::Partial_Buf_Process: The following log is not accepted by solr (file = @A@@B@1612889292 position = 2456423) :<doc>
....

[PhoneBoy]

Recommend a TAC case here.

[Ido_Shoshana]

Hi Chris,

In order to understand the issue better, we need some more details.

Can you elaborate regarding "log format seems to be out of order for some fields"?

Can you say if you have delay/backlog issues (we suspect the log_indexer doesn't keep pace with the log rate, but this is just an assumption)?

Thanks,

Ido.

[ChrisG]

Hello thanks for the reply, I will retry the upgrade procedure once again tomorrow to see if the error is coming up again. If this  issue persists I will add some detais about the log behaviour asap.

[Ido_Shoshana]

Hi Chris 🙂

Any update?

[ChrisG]

Hello,

yes I did the export/import again with a fresh installed system followed by the latest Jumbo HFA installation R80.40.Ju91. Now the logging is working smoothly.

I assume the error before occured since I did too much changes at once, like renaming the MGMT server and perhaps there have been some files from the import tests before that were conflicting with the ones from the final import.

Nevertheless when using the SmartTracker R80.40 to view the binary logs there are entries (VPN traffic always has "encryption failure:" although there is no failure) in the information field, but this seems to be a problem of the binary SmartView Tracker in R80.40. The fields are now correctly filled in the SmartLog for the live logs and the ones being imported from the previous version.

I'm struggeling now with importing the audit log (Error: Failed to read record at position), but this is not a problem in daily administative tasks.

Thanks a lot!

[Ido_Shoshana]

Thanks, Chris.

Is there anything else I can assist with? (we can take it offline if necessary).