Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
SuchitSrivastav
Contributor
Jump to solution

Checkpoint R81.20 Logging Issue

Hi CheckMates,

I have recently upgraded my Checkpoint Management Server from R80.30 to R81.20. So far, things looks good but I've started facing issue with Logging. When I open Logs&Monitor tab, in Logs I'm not able to see the Logs itself. To see the logs I've to open the specific log file. I've enabled log indexing etc. and my logging partition has more than 2TB space that should not be something with space.

I've detached & attached the log server from multiple gateways, just for testing purpose but no luck.

Any leads can suggest urgently if you've seen such issues? I've logged a TAC case but they told me this is something new issue for them too with R81.20.

PS: I'm aware that old log index wouldn't be upgraded in R81.x. I'm referring here about new log files that is being created post upgrade.

Thank you!!

0 Kudos
1 Solution

Accepted Solutions
SuchitSrivastav
Contributor

Hi Guys,

Apologies, I'm posting late here. Issue has been resolved now with help of TAC. 

Yep, it was IP confliction between two log servers. Since each log has a field named irg_log_server that is the uuid of the LS that indexed the log. When RFL get a log it verifies that a LS with the same UUID exists in the file $RTDIR/conf/logServerConfig.xml/ which in this case included only a single LS. In such case, RFL discards the logs with that orig_log_server value since it cannot find in it configuration files any LS with such UUID. 

 

After restarting log_indexer the configuration files were rebuilt and logs are seen.

Thank you everyone for your support and suggestions. 

Happy New Year to all in advacne! 

View solution in original post

35 Replies
Chris_Atkinson
Employee Employee
Employee

Do you see any logs or report data at all post upgrade?

Does same issue continue if you do evstop; evstart on the node with smartevent enabled?

CCSM R77/R80/ELITE
0 Kudos
SuchitSrivastav
Contributor

No, I didn't see any logs post upgrade. Ran these commands but no luck. When I try in SmartView, it just says "Query Failed" for running any queries or reports.

0 Kudos
Chris_Atkinson
Employee Employee
Employee

Is the machine still under considerable load from the indexing, how much resources are allocated to the system?

 

Suspect this will be something that you will need to continue working on with TAC who will raised a task with R&D where required.

CCSM R77/R80/ELITE
0 Kudos
SuchitSrivastav
Contributor

My machine has 32 GB RAM with 8 CPU and 3TB  Hard Disk. Resources are enough I think. I had logged a case with TAC but this is something new for them as well.

0 Kudos
Chris_Atkinson
Employee Employee
Employee

CPU utilisation isn't high?

Please update us as the TAC case progresses and report back here with the final resolution.

CCSM R77/R80/ELITE
0 Kudos
SuchitSrivastav
Contributor

CPU is fine. Will keep posted here as soon as I've any update from them.

Liat_Cihan
Employee
Employee

Hi,

Can you please share the SR number?

the_rock
Legend
Legend

@SuchitSrivastav , you are covered now, @Liat_Cihan is great!

0 Kudos
SuchitSrivastav
Contributor

SR#6-0003428171, here you go.

0 Kudos
Liat_Cihan
Employee
Employee

Thanks. 
we are discussing your issue internally. 
I hope we will be able to update tomorrow with a solution. 

0 Kudos
the_rock
Legend
Legend

Hey Liat,

Hope you are well :). Im just on remote with Suchit, lets see how far we get.

Andy

0 Kudos
Liat_Cihan
Employee
Employee

Andy, you never rest 🙂
Suchit, we will be in touch tomorrow

0 Kudos
the_rock
Legend
Legend

Thats cause Im eastern European, we are different ; - )

0 Kudos
the_rock
Legend
Legend

Just did remote with Suchit and below are my notes. By the way, @SuchitSrivastav , yes, test log object CAN be created with R77 version in the host properties, though I would not do so, as version should be same or higher than gateways.

Notes:

-upgraded mgmt server from R80.30 to R81.20
-so took migrate_server from R80.30 and then imported to R81.20
-when checking logs from logs and monitor, we dont see anything, but opening log files individually, we see the logs
-/var/log has 1.8 T free space
-cpwd_shows fwd process as E 1, meaning its established
-we cant find logs for any of the clusters (there are many of them) when in "logs and monitor"
-Suchit also created CP host with same IP as mgmt server
-none of 23 clusters that exist, we cant see logs just by refreshing for last 24 hours in "logs and monitor"
-will try set up one cluster with new test_log_server (same IP as mgmt), push policy and test

0 Kudos
the_rock
Legend
Legend

Just as a workaround for now...sorry, Im typing this on my Iphone, as I dont have access to my laptop, but what you can do is go to c/program files x86/check point/smart console R81.20 and then look for cplvg i believe (its a green-ish) icon and that will open old fashioned tracker.

See if logs are there. If yes, then could be indexing issue. If not, could be way bigger issue...

0 Kudos
genisis__
Leader Leader
Leader

Question:

When you say you do not see any logs after upgrading, did you confirm the following:

- Policy push to GWs was done after upgrade, and clearly confirm if SIC is working (it should be).

- Install database was done to all Management Layer devices.

- If above is done, fire up SmartView Tracker (yes Tracker), and confirm if you see any logs in this.  If you do then the issue is not logging from GWs to the SMS, but the SMS taking the raw logs and then indexing them.

0 Kudos
SuchitSrivastav
Contributor

Yes, as I said in my first post, I can see the logs when selecting the specific log files. Verfied in SmartConsole's log&monitor as well as old fashioned SmartView Tracker. 

SIC etc. are fine, I've checked this post upgrade and installed the policies on all gateways what I've configured with SMS.

Indexing is also configured and tested. It works well and same has been verified by TAC too. 

Please suggest if we've better way to fix this? I'm afraid as this is my production environment. 😞

0 Kudos
Ilya_Yusupov
Employee
Employee

Hi @SuchitSrivastav ,

 

Just to clarify the case here, the issue is post upgrade you do see new logs but it's shown like empty list in the view itself and when you click on one raw the log details are seen.

 

Is the above correct?

 

Thanks,

Ilya

0 Kudos
SuchitSrivastav
Contributor

Yes that's right.

0 Kudos
the_rock
Legend
Legend

You are in good hands now, when @Ilya_Yusupov comes to the rescue, trust me, stuff gets FIXED 100%!

0 Kudos
SuchitSrivastav
Contributor

Thank you Rock.. Hope we can fix this on urgent basis as my production is getting hamppered. 

PS: TAC is asking to grab some debug logs but I don't think that should be needed actually. I can see in tcpdump that port 257 is listening and logs are perfectly coming on management server from all of configured gateways.

0 Kudos
Ilya_Yusupov
Employee
Employee

@SuchitSrivastav - thank you, let me clarify couple of things tomorrow with my colleagues before me suggesting a solution.

I will update here tomorrow.

(1)
SuchitSrivastav
Contributor

Thank you so much. I'll be back tomorrow about 8 PM IST.

0 Kudos
the_rock
Legend
Legend

People who know me know that I do NOT say things like that about anyone, unless I mean them 100%. @Ilya_Yusupov fixed ISP redundancy issue for my colleague and I and let me tell you, it was very DIFFICULT problem, so I have full confidence he will do same for you as well. Just give him little bit of time, he is super smart and very responsive.

0 Kudos
SuchitSrivastav
Contributor

I appreciate your valueable suggestion and support. Thanks again to both of you.

0 Kudos
the_rock
Legend
Legend

Personally, I would not bother with debugs, unless fwd process is not working (maybe you can confirm that for us). Though, Im fairly positive it is based on your description. Have you ever checked if old fashioned tracker shows the logs? Hey, Im off today, but happy to do remote if you like, maybe I can help you out.

Let me know.

0 Kudos
SuchitSrivastav
Contributor

Yes, I can see logs in SmartView Tracker. Happy to the remote session if you can share a link to me. 

0 Kudos
the_rock
Legend
Legend

Yup, just sent you private message with the details. Lets do my personal zoom, I use my own gmail account for it, so it has 40 mins limit, but does not matter, if we need more, will "fire up" a new one haha

Cheers.

0 Kudos
the_rock
Legend
Legend

Hi Suchit,

Thanks for your time on remote today, appreciated! By the way, @Ilya_Yusupov and I connected via email after the call on this issue and we are both pretty sure that what we discussed on the call would work, ie set the CP host object with logging option selected to be the main logging server for say 1 gateway, push policy and test. If that works, just revert back to original option and see if it still works. However, if you have no luck with that procedure, I would suggest to get in touch with @Liat_Cihan and @Ilya_Yusupov for further help. They are both EXCELLENT and if they cant solve it for you, they will definitely find someone that can.

Please keep us posted.

0 Kudos

Leaderboard

Epsum factorial non deposit quid pro quo hic escorol.

Upcoming Events

    CheckMates Events