MDS HA Global Reassignments Issues on R81.10 (before upgrade to .20)

[Jerry]

more details once you got interested folks, personally those errors prevent operations on MDS and they're super annoying, nobody knows why revisions DB could get corrupted but it seems it is the case. anyone experienced similar bugs?

little debug below (with some details "unmarked" obviously).:

MDS:

18/07/23 10:26:22,801 ERROR coresvc.internal.GlobalDomainAssignmentSvcImpl [taskExecutor-3669]: GlobalDomainAssignmentTask: caught exception.

CpmGeneralException{base='com.checkpoint.management.is.exceptions.CpmGeneralException: An internal error has occurred.', errorCode='CP_ERR_UNSPECIFIED', errorFamily='null', messageForUser='null', message='An internal error has occurred.'}

CMA:

17/07/23 11:58:13,602  INFO management.threat.IpsUpdateManager [taskExecutor-2092]: Reassign IPS update operation failed for domain: 8828bb85-097c-45ef-b61b-b2cff20b1ad1

17/07/23 11:58:13,719  INFO dleserver.utils.LogSaverForFailedTasks [taskExecutor-2092]: failed task logs will be saved to $MDS_FWDIR/log/failed_tasks/IPS_Update

17/07/23 11:58:13,719  INFO dleserver.utils.LogSaverForFailedTasks [taskExecutor-2092]: running command: $MDS_FWDIR/scripts/save_logs_when_task_fails.sh 10 IPS_Management_Update_IPS_Update failed_on_assign true CMA-X_Management_Server IPS_Update

Running command: [$MDS_FWDIR/scripts/save_logs_when_task_fails.sh 10 IPS_Management_Update_IPS_Update_failed_on_assign true CMA-X_Management_Server IPS_Update]

17/07/23 11:58:15,575 ERROR coresvc.internal.GlobalDomainAssignmentSvcImpl [taskExecutor-2092]: AGP failed. domain session ab862f38e0ed2cd38d1a19c016013eb2 has domain work session 4e85fdd6-5713-431e-a5f7-b8934a7242e4. discarding WorkSession '16d50d89-5353-4bd3-913a-47fd729919de'

PS. should you wish to help my Customer with that errors please have a quick look at the following SR: 6-0003674786

[the_rock]

Hey mate,

Any movement on that case? Let us know the outcome.

Andy

[Jerry]

hi mate, thanks for your reply, was hoping you will 🙂

so far just the "assignment" from the TAC, I'll let you know as soon as Jay will finally have a call with me reg. that MDS.

my customer is struggling with Global Assignments towards multiply CMA's and quite frankly if TAC won't address those issues then maybe R&D will. R81.10 have "unknown" errors all over the place on that MDS despite that this MDS HA has all CMA's logging with no MLM's yet. Logging is killing that MDS's and we know it but Global (shared) Domains reassignment should work like a charm anyway, so far logs shows errors telling you NOTHING reg. what is going on.

awaiting TAC actioning but not sure if I can ask for the speed up ... :(* 

Cheers Andy.

[_Val_]

I assume you mean R81.10 and not R80.10. If this is the case, could you please PM me with the TAC case number?

[Jerry]

hi Val, yes it is R81.10 sorry, typo

<removed by admin>

[Jerry]

hi guys, Val

I've had an extensive cpm debug done yesterday with the T2 TAC eng. Jay but I believe this was not only not enough but far too simple. My customer require proper escalation and deserve deep-dive t-shooting what is going on by their primary MDS.

Is there anyone here who could/can make the call with me over the mentioned SR and try (at least attempt) to solve below issues? Few CMA's are simply IPS-Update failing, Reassignments broken etc. They just cannot operate like that.

Anything what R&D or MG Team from TAC could do would be highly appreciate as my hands are tight really.

ps. Take 109 could solve some of the mentioned issues but we need to be told that this is the 1st step resolution by official email. So far I'm still waiting for the TAC eng. to provide such hence my Customer is getting really impatient reg. the resolution times.

[_Val_]

@Jerry 

Several points:

1. I asked to send me a PM message, not to share your TAC case number in a public forum. 
2. CheckMates is not a proper escalation channel. You can escalate your ticket via TAC tools or through your local Check Point office. I actually see in the case, that you did escalate it earlier today. 

[Jerry]

sorry Val, misunderstood. All clear.

Yes TAC has the escalation and Yan is taking care of it as we speak.

my apologize for the SR number here, should you wish to remove it please do so.

[_Val_]

No worries, removed

[the_rock]

I missed that part, I thought I saw R81.10 as well, but mate, R80.10, thats has been unsupported for some time now : - )

Andy

[Jerry]

so sorry Andy you totally spot on, my typo. it is ALL about the R81.10 MDS HA setup.

cheers!

[the_rock]

IT person making spelling mistakes and typos? What a SHAME ; - )

Just kidding...any update from TAC mate?

Andy

[Jerry]

LOL 🙂 thanks mate, point taken 😛

reg. the MDS - well ,as several posts here state R81.10 struggle with the revision DB purge, we've got the same, you cannot purge the DB with several errors like this:

[Expert@mdsX:0]# mgmt_cli set automatic-purge enabled true keep-sessions-by-count false number-of-days-to-keep "30" scheduling.check-interval "5" scheduling.time-units "days" scheduling.start-date "2023-07-26T13:08:00"
Username: ............
Password:
code: "err_validation_failed"
message: "Operation is not allowed - can't create automatic purge from global or system domains!"

another point is that Reassignment are not happening and IP Management updates stuck as shown earlier.

So far I believe only the best of the bests from R&D or Diamond Support could solve that problems (my Customer does not have Diamond by Premium unfortunately hence this may take time till TAC realizes what's really wrong with that MDS.

other than that all good but certain domains are really in a poor shape, maybe due to the revision DB, as following

then each CMA with 3685 revisions ... just one with 3500 only 😞