Management VM Upgrade with Database Migration

[KVC]

Hello All,

We were in the midst of upgrading the Management VM from version 80.40 to 81.20, and encountered an error while verifying the image, which led us to reference sk180769. It appears that due to incorrect partitioning in the Gaia OS, both disks are currently flagged for boot, prompting CP support to recommend installing a new Management VM with a fresh OS and migrating the database as the only viable solution.

Before proceeding, I have a query regarding the exported database file's size with log selection. The current size of /var/log/, as indicated by 'df -h', stands at 1.5TB, utilizing 70% of the total space. Does this imply that the exported file, including logs, will exceed 1.5TB?

Moreover, assuming a 1.5TB exported file size, is there a method to estimate the time required for its creation? Referring to sk133312, exporting a management database of 30GB typically takes 3 to 4 hours. Extrapolating this to a database plus log file size of approximately 1.5TB, the process could potentially span 150-200 hours, equivalent to 6-9 days for completion, with a similar timeframe anticipated for the import process.

Could someone please provide insights on this matter and confirm if my line of reasoning makes sense?

Regards,
K

[Amir_Senn]

Hi,

Log servers (Management in your case) will preserve logs and indexes until storage meets the threshold in the disk space management (usually ~5 GB by default). If this is too much you can apply the daily retention and it will delete older bugs.

/var/log has much more data than just logs. Actual log files are located here: $FWDIR/log/ but also other elg files.

In addition to that, when doing export you usually export indexes as well (also located on /var/log partition). Since from R80.40 to R81+ there's a service upgrade, it needs to re-index existing logs, so no point in exporting indexes as well, only logs. This alone will also save a significant size of the file.

[KVC]

Hello Amir,

Appreciate your prompt response.

I'll dive into checking the size of the $FWDIR/log/ to get a better grasp on estimating the export file size.

Regarding my concern about the export duration for the management database and log files, considering it's a hefty 1.5TB, do you think the timeframe mentioned in my initial post still holds true?

Looking forward to your insights.

Regards,
K

[Amir_Senn]

a. I truly believe that the actual log files size on disk is less than half of that.

b. Time could vary due to server resources.

c. Either way you can't compare compressing MGMT DB with collecting and compressing logs (which will be more efficient for migrating to a new server due to less disk space).

d. The number of logs in a log server is not set and I believe that in the future servers will keep even more logs. In the future, maybe such large migration time will happen. Best way to avoid it is to set daily retention to range you want to retain logs. This way you won't keep logs that you no longer need and migrate extra data.

[KVC]

Hello @Amir_Senn,

Thank you for your reply.

Upon reviewing the size of the $FWDIR\log directory, it appears to be a substantial 1.4TB in size, as evidenced by the attached screenshot. This directory contains logs dating back a year, with regular exports done by them to manage space. Many of the *.log files within are sizable, some reaching 2GB each, contributing significantly to the overall space consumption.

Despite the daily log retention being set to 3605 days, the logs remain relatively recent due to regular exports. Considering this, would adjusting the retention period to 365 days for example, aligning it with the actual practice, alleviate the space issue in any way?

Your insights on this matter would be greatly appreciated.

Thanks,
K

[Amir_Senn · Accepted solution]

@KVC Very good that you have both policies. Good practice.

I really can't tell if 365 is enough logs for you, that depends on your needs. I know admins that only saved 60 because they what they needed to save and those who have to save for 2 years. IMO, it's always good to keep data as long as it's not making life harder than it should.

Little things about log files:

-Maximum size is 2GB

-Every log file has complementary files that keep pointers, special properties etc. This means number of files per log file may vary according to your detail level chose for the log and when matched by traffic.

-Log file has auto switch in 00:00

-The name of the log is the time stamp when the log was closed

To see you oldest logs, just follow the latest date that appears.

You probably don't use logs older than 45 days since the indexes are deleted (which makes them not to appear on logs view unless opening the log file manually).

Recommendation from my POV:

-Figure out how many days you need to use logs and how many days you need to retain them - make sure the days to keep indexes is larger/equal than usage and for log retention the value to keep should be larger/equal to the number of days to retain. Myself, I would increase post upgrade the number to retain indexes (which will make you lower the amount in the retention value since the value is added (additional to the indexes) and could cross the max size of 3650) on the account of retaining really old logs.

-If you think you might need them in the future but not sure, you can copy the older logs to another server (make sure to copy all files connected to a specific log file), after that you can shorten the number of days to retain logs. Post cleaning up (0:00 a processes begins that is very quick) you'll less logs to migrate and some copy of them.

[the_rock]

I agree with all Amir said and as far as time frame you indicated, I would say that estimate seems fairly accurate to me. But again, it could depend on various things.

Best,

Andy

[emmap]

I would recommend doing the migration without logs, and manually copying however many logs over that you want to retain from the old VM to the new one, then reindexing them. 

[Amir_Senn]

I would not recommend that. We have a proper way of doing this with exporting which is tested and certified.

[the_rock]

@Amir_Senn Is there a good document to follow when you do this with log export? I see what @emmap is saying, even TAC recommended doing it the same way to me multiple times in the past.

Best,

Andy

[Amir_Senn]

No document/SK that I'm aware of. This is probably because we have a certified tool to do it and no matter how much TAC recommend migrating logs in this way, it's not certified and I wouldn't recommend it because of that.

People don't always know all the files that needs to be migrated in order to do a successful migration and things might go wrong.

And not related this branch since it's moving between SOLR versions, but migrating logs in this manner will require indexing, which will be more costly in resources than migrating them in most cases.

Also, it's not proven that it will save any amount of time (maybe on some setups, maybe not at all). If all the migration tools do with logs is compressing them and adding it to the exported DB it could be faster than migrating uncompressed logs.

[the_rock]

K, so thats different story then : - ). If there is certified tool, but customers dont know or never heard of it, it would not be sadly of any use to us. Once it becomes available, Im sure people would be happy to give it a try.

Best,

Andy

[Amir_Senn]

The tool is the same tool as upgrade. You migrate with no logs/logs/logs + indexes.

If you don't want to migrate the server itself, just the logs, you can migrate it to other servers with log forwarding. This is not the case though.

[the_rock]

Ah, k, I see what you mean. 

Best,

Andy

[emmap]

The problem is that if the log partition is 90% full, there is not enough space to export with logs as there's not (yet) a way to only partially include some logs in the export. Hence the manual approach. It's not uncertified to copy logs manually, it's in the admin guide under Importing Offline Log Files. I understand that it's easier to do it all in one, but file sizes can render that unfeasible. 

[the_rock]

I agree with you 100%. That sounds totally logical to me.

Best,

Andy