Checkpoint R81.20 Logging Issue

[SuchitSrivastav]

Hi CheckMates,

I have recently upgraded my Checkpoint Management Server from R80.30 to R81.20. So far, things looks good but I've started facing issue with Logging. When I open Logs&Monitor tab, in Logs I'm not able to see the Logs itself. To see the logs I've to open the specific log file. I've enabled log indexing etc. and my logging partition has more than 2TB space that should not be something with space.

I've detached & attached the log server from multiple gateways, just for testing purpose but no luck.

Any leads can suggest urgently if you've seen such issues? I've logged a TAC case but they told me this is something new issue for them too with R81.20.

PS: I'm aware that old log index wouldn't be upgraded in R81.x. I'm referring here about new log files that is being created post upgrade.

Thank you!!

[Chris_Atkinson]

Do you see any logs or report data at all post upgrade?

Does same issue continue if you do evstop; evstart on the node with smartevent enabled?

[SuchitSrivastav]

No, I didn't see any logs post upgrade. Ran these commands but no luck. When I try in SmartView, it just says "Query Failed" for running any queries or reports.

[Chris_Atkinson]

Is the machine still under considerable load from the indexing, how much resources are allocated to the system?

Suspect this will be something that you will need to continue working on with TAC who will raised a task with R&D where required.

[SuchitSrivastav]

My machine has 32 GB RAM with 8 CPU and 3TB  Hard Disk. Resources are enough I think. I had logged a case with TAC but this is something new for them as well.

[Chris_Atkinson]

CPU utilisation isn't high?

Please update us as the TAC case progresses and report back here with the final resolution.

[SuchitSrivastav]

CPU is fine. Will keep posted here as soon as I've any update from them.

[Liat_Cihan]

Hi,

Can you please share the SR number?

[the_rock]

@SuchitSrivastav , you are covered now, @Liat_Cihan is great!

[SuchitSrivastav]

SR#6-0003428171, here you go.

[Liat_Cihan]

Thanks. 
we are discussing your issue internally. 
I hope we will be able to update tomorrow with a solution. 

[the_rock]

Hey Liat,

Hope you are well :). Im just on remote with Suchit, lets see how far we get.

Andy

[Liat_Cihan]

Andy, you never rest 🙂
Suchit, we will be in touch tomorrow

[the_rock]

Thats cause Im eastern European, we are different ; - )

[the_rock]

Just did remote with Suchit and below are my notes. By the way, @SuchitSrivastav , yes, test log object CAN be created with R77 version in the host properties, though I would not do so, as version should be same or higher than gateways.

Notes:

-upgraded mgmt server from R80.30 to R81.20
-so took migrate_server from R80.30 and then imported to R81.20
-when checking logs from logs and monitor, we dont see anything, but opening log files individually, we see the logs
-/var/log has 1.8 T free space
-cpwd_shows fwd process as E 1, meaning its established
-we cant find logs for any of the clusters (there are many of them) when in "logs and monitor"
-Suchit also created CP host with same IP as mgmt server
-none of 23 clusters that exist, we cant see logs just by refreshing for last 24 hours in "logs and monitor"
-will try set up one cluster with new test_log_server (same IP as mgmt), push policy and test

[the_rock]

Just as a workaround for now...sorry, Im typing this on my Iphone, as I dont have access to my laptop, but what you can do is go to c/program files x86/check point/smart console R81.20 and then look for cplvg i believe (its a green-ish) icon and that will open old fashioned tracker.

See if logs are there. If yes, then could be indexing issue. If not, could be way bigger issue...

[genisis__]

Question:

When you say you do not see any logs after upgrading, did you confirm the following:

- Policy push to GWs was done after upgrade, and clearly confirm if SIC is working (it should be).

- Install database was done to all Management Layer devices.

- If above is done, fire up SmartView Tracker (yes Tracker), and confirm if you see any logs in this.  If you do then the issue is not logging from GWs to the SMS, but the SMS taking the raw logs and then indexing them.

[SuchitSrivastav]

Yes, as I said in my first post, I can see the logs when selecting the specific log files. Verfied in SmartConsole's log&monitor as well as old fashioned SmartView Tracker. 

SIC etc. are fine, I've checked this post upgrade and installed the policies on all gateways what I've configured with SMS.

Indexing is also configured and tested. It works well and same has been verified by TAC too. 

Please suggest if we've better way to fix this? I'm afraid as this is my production environment. 😞

[Ilya_Yusupov]

Hi @SuchitSrivastav ,

Just to clarify the case here, the issue is post upgrade you do see new logs but it's shown like empty list in the view itself and when you click on one raw the log details are seen.

Is the above correct?

Thanks,

Ilya

[SuchitSrivastav]

Yes that's right.

[the_rock]

You are in good hands now, when @Ilya_Yusupov comes to the rescue, trust me, stuff gets FIXED 100%!

[SuchitSrivastav]

Thank you Rock.. Hope we can fix this on urgent basis as my production is getting hamppered. 

PS: TAC is asking to grab some debug logs but I don't think that should be needed actually. I can see in tcpdump that port 257 is listening and logs are perfectly coming on management server from all of configured gateways.

[Ilya_Yusupov]

@SuchitSrivastav - thank you, let me clarify couple of things tomorrow with my colleagues before me suggesting a solution.

I will update here tomorrow.

[SuchitSrivastav]

Thank you so much. I'll be back tomorrow about 8 PM IST.

[the_rock]

People who know me know that I do NOT say things like that about anyone, unless I mean them 100%. @Ilya_Yusupov fixed ISP redundancy issue for my colleague and I and let me tell you, it was very DIFFICULT problem, so I have full confidence he will do same for you as well. Just give him little bit of time, he is super smart and very responsive.

[SuchitSrivastav]

I appreciate your valueable suggestion and support. Thanks again to both of you.

[the_rock]

Personally, I would not bother with debugs, unless fwd process is not working (maybe you can confirm that for us). Though, Im fairly positive it is based on your description. Have you ever checked if old fashioned tracker shows the logs? Hey, Im off today, but happy to do remote if you like, maybe I can help you out.

Let me know.

[SuchitSrivastav]

Yes, I can see logs in SmartView Tracker. Happy to the remote session if you can share a link to me. 

[the_rock]

Yup, just sent you private message with the details. Lets do my personal zoom, I use my own gmail account for it, so it has 40 mins limit, but does not matter, if we need more, will "fire up" a new one haha

Cheers.

[the_rock]

Hi Suchit,

Thanks for your time on remote today, appreciated! By the way, @Ilya_Yusupov and I connected via email after the call on this issue and we are both pretty sure that what we discussed on the call would work, ie set the CP host object with logging option selected to be the main logging server for say 1 gateway, push policy and test. If that works, just revert back to original option and see if it still works. However, if you have no luck with that procedure, I would suggest to get in touch with @Liat_Cihan and @Ilya_Yusupov for further help. They are both EXCELLENT and if they cant solve it for you, they will definitely find someone that can.

Please keep us posted.