SMP logging issues

[nmelay2]

Many of the gateways I manage on SMP are missing their logs at the moment.
Logs show up locally, just not on SMP.
The issue started occurring 4 days ago on 4 gateways, and seems to be spreading.
Now 2/3 of them only show logs up until a few days ago.
Also, the same issue can be seen in different account IDs.
I'd be curious to know if any of you are also affected.

Case was raised with TAC, but got nowhere so far as we initially thought that was a 1 gateway issue, and focused on local debugging.
We'll see where this goes tomorrow.

[the_rock]

Centrally managed?

[nmelay2]

Nope, SMP.

[the_rock]

So thats cloud portal, meaning it would be centrally managed...no?

[nmelay2]

Well, in SMB parlance, "centrally managed" means managed by SMS, not SMP.

Confusing I know...

[the_rock]

Even in my 40s, some days, I feel I can only remember LOL meaning 🤣🤣

Anywho, what I would try is restart the cloud instance, since its multiple gateways having the same issue.

[the_rock]

Here is what I was referring to as far as restarting cloud instance.

 

[nmelay2]

Yeah but that's not SMP, that's S1C (centrally managed by the way). 😂

That's SMP : https://portal.checkpoint.com/dashboard/sparkmanagement

[the_rock]

Ah, yes, right 😂

Well, for that, you may have to open TAC case and ask them to restart it. Btw, I just created trial instance for it, how do you even install policy there? Looks way different than regular S1C 🙂

[Tom_Hinoue]

@nmelay2 
We have similar case in SMP since a few days ago and have a case raised to TAC.
I think its issue with log ingestion in SMP log server side and not the gateway.

[the_rock]

Good to know, Tom. It also sounds to me like this would be an issue on management side, for sure. Let us know what TAC says.

[nmelay2]

Thanks @Tom_Hinoue for the confirmation.

Indeed seeing how widespread it is, that's probably a SMP server issue, though we can't really rule out a problem with the data being sent out by the gateways.

Please keep me posted on your TAC case status, and I'll do the same.
My own case number is 6-0004447395.

[the_rock]

@nmelay2 Just curious though, might be worth asking TAC if they can restart that instance. I remember S1C days when customers could not do so up until 2-3 years ago.

[nmelay2]

Oh well. The Check Point SMP admin team doesn't need me to tell them how to manage their service.
And again, this is not S1C, totally different service, no per-customer instance.

[the_rock]

Fair enough 🙂

[yahavb · Accepted solution]

Hi, thanks for reporting this issue.

We’ve identified that the problem was caused by a server-side issue in our EU region, where approximately 30% of gateways experienced failures when attempting to store logs in the cloud.

The issue has now been fully resolved.
We are also implementing additional monitoring and alerting to ensure this type of incident is detected immediately and does not recur.

We are currently working on recovering any missing log data.

We sincerely apologize for the inconvenience, and we appreciate your patience.

[the_rock]

Thanks for the update @yahavb 

[nmelay2]

Hello Yahav,

I confirm we can see new logs coming in since yesterday morning.
It seems like we were more severely impacted than others
In our case, missing old logs is probably not going to be an issue, but it's still good to know you're working at recovering them.

Thanks to you and the team.