R80.20 MDS restore missing over a month worth of data

[Kaspars_Zibarts]

This is a bit of SOS call if anyone else has seen this.

Was forced to restore our production MDS this morning. So not a biggie. Backup was taken yesterday and restore worked just fine.

But then we noticed weird things that a lot of rules are missing and some topology push failed due to missing interfaces or routes on VSX.

Then we realised that "newest" data we have on MDS is from 5th November! Ouch. Audit logs still show all the changes from yesterday but rule are gone.

Quite a pickle we are in now as I don't believe backups from day before would be any better. We will keep trying  but if anyone has seen/knows something would be great!

[Maarten_Sjouw]

Wow, that is really some bug, I feel your pain, currently I'm on R80.30 so cannot be sure that if we restore the data will be better than yours.

Good luck with this mess, cannot call it anything else.

[Kaspars_Zibarts]

What else! FGS!

Managed to restore MDS2 backup from yesterday and that seems OK. Limitation is that you cannot manage VSX from secondary MDS ( sk62502)

And sync back to  primary does not work because of SK145972

I'm about  to give up on CP after this... arghhhhh!

[Maarten_Sjouw]

And there is no way to promote the secondary?

[Kaspars_Zibarts]

Doesn't look like:

After fail-over between Provider-1 MDS servers, it is not possible to edit the VSX object on Secondary CMA, until the status of the Main CMA is changed correctly - Active on Secondary MDS, Standby on Primary MDS.

Since my Primary MDS was not available I don't it would have worked.

But most likely if I had spent more time on it.. I just had to get out from the deadlock in most efficient way and at the end restoring couple of days old backup on primary kicked in sync from the secondary / active so we are back in business!

[Maarten_Sjouw]

ok great that you have it resolved, it's always a bummer when things like these happen, they tend to cost you a lot of work and frustration.

[Kaspars_Zibarts]

Just heads up - yesterday I finally had time to play with my dodgy backup in the lab and I was able to reproduce it twice again! Basically MDS backup taken on 18th December, restore is successful  but over 6 weeks worth of data is missing. We only see changes done till 5th Nov. I will be re-opening my TAC case but thought to let know here to watch your MDS restore in R80.20! Might be just us (hopefully) but it can mess up things if you overlook it

[Kaspars_Zibarts]

Ok i have worked out the culprit: we run solr cure script on 4th Nov to compress DB to reduce MDS backup (that I have been complaining for years) and it seems to have done something on primary MDS, whilst secondary works OK. So if you have not run it, you should be OK 🙂

$MDS_FWDIR/scripts/solr_cure.sh –j