Export file name change in R82

[Hugo_vd_Kooij]

HI,

As we are testing R82 against our automation we found that the `$FWDIR/bin/upgrade_tools/migrate export`command has changed it's output file which broke our automation.

Previously the -n option name would be used in full and we used the .tgz at the and as that is effectively the used format.

But in R82 the export name in the -n option gets the .txz added after the full name.

Is this a bug or a feature not clearly enough documented as changed behaviour?

[_Val_]

I hope you actually mean "migrate_server" and not "migrate", which is legacy and should not be used.

[Hugo_vd_Kooij]

Got the wrong section there in the example That is our legacy command for older versions.

This is what I just tested as export:

# Use this on R82
$FWDIR/scripts/migrate_server export -v $FWVER \
-skip_upgrade_tools_check \
--include-uepm-msi-files \
-n $BACKUPDIR/export-$HOSTNAME-$DATE

[_Val_]

Thanks for clarifying, @Hugo_vd_Kooij 

[genisis__]

What sort of automation are you doing?  Just dipping my foot in the automation world.  I suspect my starting point would be backup and restore, via Ansible, however want to have some intelligence in there (did get someone to write a script for me ages ago to schedule a MDS backup which emailed me if it worked with a filename and md5sum, and if it did not work email me the log and why it failed ie. diskspace).

[Hugo_vd_Kooij]

At the moment we do:

• CIS Benchmark checks.
• GAIA backups
• SmartCenter backups
• MDS Backups

And more stuff is under development.

All backups are fetched to our central backup server.

[genisis__]

Like the quote!

On the backups side have you managed to test restores? Or is the design manual restore?

[Duane_Toler]

If you're interested in doing automation with the management API and Ansible, check my YouTube series I have going. I cover setting up the inventory for each type of management server, running Ansible with a Docker container (highly recommended), using git for tracking, and the recent episodes introduce Jinja templating and Ansible loops.  (links in my signature line below)  I do have a Substack where I also post a written version of each episode for those who prefer long-form content instead.

I'm in the process of moving to a new home, so the next episode is going to be a bit delayed. Meanwhile, any and all feedback is appreciated!

Thanks!

[genisis__]

That would be good!  I'm very new to ansible but will certainly watch the channel,  may ping you privately with questions, as I face challenges, if you don't mind.  

[Duane_Toler]

Sure thing.  The Substack page also has a chat feature that may work for that, but PMs here are fine as are direct emails (also linked in the channel info).

Hopefully, I got all the info conveyed that can get you going. 😊  Let me know if I missed something, tho.  I kept each episode about 15-20 min long, so they're digestible in "lunch time" sized chunks.  Each episode is time-indexed, too, so you can easily skip around to the section(s) you need most.

If you happen to be visually color-deficient, I also adjusted the color palette for everything so it's visually accessible for all types of color-blindness (i had no idea there were FOUR types of color-blindness!).

I have an itinerary for future episodes based on the results of a survey I posted a while back, but if you need something in particular, let me know and I'll see what I can do.

[_Val_ · Accepted solution]

I checked with R&D.

In R82, we use a different zipping tool, hence the change of the extension. We will update the documentation, the task is already open.

Regardless, you should use migrate_server and not migrate, as already mentioned.

[biskit]

Question - because I don't have a system easily available to test on...  Does migrate_server still stop services when it runs on R82?  
This was a problem on R81.x, especially on standalone systems where you needed downtime to run the command.  The old migrate export was nicer as no downtime was needed.

[Duane_Toler]

FWIW - I did a migrate export and migrate_server export on a standalone R81.20 firewall (mgmt+gw integrate) this weekend and neither of them stopped services or caused downtime.  I also did not use the "-n" for non-interactive mode either.

I used the "clean install/upgrade" image and ran it with "installer upgrade ...".  It did everything the Blink image does except include the latest Jumbo HFA.

It did its own migrate_server export (even tho I did my own earlier), install to a new LVM volume, copied the Gaia configuration and /home directories to the new volume, rebooted into the original Gaia OS configuration, resumed the installer, and ran its own migrate_server import to bring up the configuration.  I had to SSH into it and do install-database and install-policy manually (which always needs to be done).  Afterwards, I downloaded and installed the latest R82.00 JHF Recommended take (39?).  All went well.

Edit: All told, this process took about 90 minutes (including some individual housekeeping items of my own).  This was an HP Proliant DL360 Gen9 system, 64 GB RAM.

[PhoneBoy]

I recall that we had requests on CheckMates to move away from using gzip for the migration tools.
Looks like we did in R82, using XZ instead of gzip.
Aside from the change in extension (txz versus tgz), XZ should generate smaller archive files.
I vaguely recall our gzip implementation being single core, something I assume is now fixed with the move to XZ.

[Amir_Senn]

I actually watched this on YouTube 3 weeks ago: Explaining File Compression Formats

Interesting watch.

[Bob_Zimmerman]

Seems a little odd to move away from gzip to xz, of all compressors. xz has better compression than gzip, but it's significantly slower, both when compressing and when decompressing. The default xz implementation (which is what R82 seems to have) only uses one core, and while parallel alternatives exist, they're still far slower than single-threaded gzip, let alone parallel gzip (pigz, which Gaia has had for a long time).

Meanwhile, zstd achieves a similar compression ratio to xz, faster compression than xz (or a compression ratio a little better than gzip with compression speed faster than gzip), and much faster decompression than xz (about equivalent to gzip's decompression performance).