topic Re: R82.10 snapshot revert error in General Topics

R82.10 snapshot revert error

Duane_Toler — Fri, 29 May 2026 02:08:40 GMT

So this is new...

gateway01> set snapshot-onetime revert target lvm name Blink_R82.10_402 NMSNAP0042 Bad snapshot name. Snapshot name may only contain up to 15 digits, letters and underscores (_)

3950 appliance, R82.10, take 271

Had an in-place Blink upgrade to take 467 fail, but now I can't revert back to try it again. This worked on 2 other 3900s (one 3950 and one 3980). TAC case is open for the upgrade failure issue, tho.

Re: R82.10 snapshot revert error

the_rock — Fri, 29 May 2026 02:49:59 GMT

Interesting, never seen that before. Hope they can sort it out...but, is that actual snapshot name?

Re: R82.10 snapshot revert error

Duane_Toler — Fri, 29 May 2026 02:51:36 GMT

yep!

gateway01> show snapshots Restore points: --------------- Blink_R82.10_402

Re: R82.10 snapshot revert error

emmap — Fri, 29 May 2026 05:37:40 GMT

It doesn't like the full stop/decimal point in the name, I'd guess. Though really it shouldn't have let you take it in the first place if that's the case.

Re: R82.10 snapshot revert error

Duane_Toler — Fri, 29 May 2026 13:03:44 GMT

This was the snapshot created by the Blink upgrade, automatically. The manual command indeed refuses to make one with that name pattern.

On an R82.10 Take 467 gateway:

gateway02> add snapshot-onetime name Foo_R82.10 description FooSnap NMSNAP9999 Snapshot name may only contain digits, letters and underscores. Maximum 255 characters

Looks like the Left Thumb didn't talk to the Left Pinky about this. Oops. TAC case again?

Re: R82.10 snapshot revert error

Bob_Zimmerman — Fri, 29 May 2026 14:31:11 GMT

Maybe try reverting in GRUB? It may have fewer restrictions.

That said, I've had major snapshot problems on two separate systems, now. I've taken a snapshot on R82, upgraded the system to R82.10, reverted to the snapshot, and wound up with a totally hosed system. Both would hang at different points during boot. Went into single-user, and lv_current was 100% used.

Re: R82.10 snapshot revert error

Duane_Toler — Fri, 29 May 2026 16:23:21 GMT

Another bonus, that I didn't check first, is that the snapshot name from a Blink upgrade is 16 characters. Oops x2. The TAC person said no one over there knows how to recover from such a situation as an "interrupted" upgrade like this.

Deployment Agent as a whole is angry because /mnt/fcd is still present, and has a path opt/ which contains the various Deployment Agent logs. These were copied over by the installer from /opt/CPInstLog, according to the last line of the log:

[2026-05-28 - 20:15:24][294723 295414]:Copying CPInstLog dir [2026-05-28 - 20:15:24][294723 295414]:about to copy dir /opt/CPInstLog/ to /mnt/fcd//opt [2026-05-28 - 20:15:24][294723 295414]:About to execute command: /bin/cp -af /opt/CPInstLog/ /mnt/fcd//opt

The appliance rebooted, but since the process died, it can't recover.

I'm going to schedule a plain reboot tonight with the customer; perhaps it'll pick up where it left off and try Stage 2 again...? maybe?

Re: R82.10 snapshot revert error

the_rock — Fri, 29 May 2026 17:08:32 GMT

Hope all goes well.

Re: R82.10 snapshot revert error

emmap — Wed, 03 Jun 2026 02:29:04 GMT

That sounds very much like TAC needs to be escalating that information to the Blink owners internally.

Re: R82.10 snapshot revert error

Duane_Toler — Thu, 11 Jun 2026 14:26:53 GMT

Yep, they're waiting on me for some follow-up info which I was able to collect today. Short version:

1. The gateway actually never even rebooted, but I thought it did! It looked like it did, tho. When we pulled the power cable to hard cycle, it booted up and continued onwards to finish the upgrade as expected. Everything seems relatively fine after that now.

2. Reboot and shutdown scripts on R82.10 appear to be nothing but a NO-OP. The scripts shutdown FWD, shutdown the interfaces, then....exit. I'm not sure how reboot signals are expected to be handled, but I don't see anywhere this is happening. We couldn't reboot it from the 'reboot' or 'shutdown -r now' commands. Something is amiss here. This is why we had to do the hard power cycle above.

3. "This is not the systemctl you're looking for". /bin/systemctl is just a wrapper Bash script that calls the SysV init scripts. Um.. what? Systemd appears to be running things from the places we expect, which is how it picked up the post-boot script.

I also had a Jumbo HFA install fail tonight on a second gateway. It was already Take 467 with Jumbo HFA 6; I did the initial re-install/upgrade before putting the gateway into service. This one clearly rebooted successfully at the time. Tonight, I was installing Jumbo HFA 19 (for a VPN degradation issue they're having, that seems to be addressed in JHF 19).

It seems to have cranked the CPU and some system processes, then ... died. It was at the 85% level of "copying files..." when it just died. No console output when the customer connected... dunno if it even rebooted. The grub2 password we set also didn't work, so no snapshot revert option, no maintenance mode, nothing. Had no choice but to do factory reset.

~~In short...this has been a very painful experience with the 3900s. Nothing positive to say about these yet.~~

EDIT: I'm leaving the above, as a public confession of my sins and guilt. 🙂 I found the issue here and it's... sigh... embarrassingly awful. The systemctl script issue wasn't with the JHF or the Blink image after all. I'm gonna update the TAC SR as well. We had a deployment playbook that mis-categorized the 3900s and applied the wrong package to them.

I'm gonna update the other messages here as well.

Thanks @Bob_Zimmerman for your verification info as well! That was helpful to see!

Re: R82.10 snapshot revert error

emmap — Wed, 03 Jun 2026 03:18:06 GMT

What was your upgrade path here? Was it raw R82.10 build 271 that you then upgraded with the specific 3900 blink package?

Re: R82.10 snapshot revert error

Duane_Toler — Thu, 11 Jun 2026 14:28:51 GMT

Out-of-box, it was the factory original Take 271 (GA1). I then installed the Quantum Force 3900 Take 22 hotfix; (GA2 wasn't available then). Then I did Blink in-place upgrade to Take 467 (GA2) when it came available with Jumbo HFA Take 6 (new LVM volume anyway, with LV snapshot of 271). That process went just fine originally, just as expected, and the gateway was operating for a few weeks successfully. Very unremarkable, as it should be.

Now, I went to install the Jumbo HFA 19 package like any other Jumbo HFA would. I always run verifier before doing anything, too, and verifier succeeded with "install.applicable: true" and no warnings.

It went through the routine, then got to the meaningless 85% with message "Copying files". Sat there for a long time, which is fine... I always see it sit here with no updates. I played a card game of "Go Fish" with my daughter over dinner to pass the time. Eventually, the system became unresponsive... then my SSH session was killed and that was the end. No remote access, so customer traveled to the office. After about an hour, customer connected console cable but got no output. We let it sit for several more minutes but still nothing.

Had no meaningful option but to power cycle it. I hated to do it, but I knew we could revert to some prior snapshot. Then that turned out to be a bad choice. None of our input choices worked, even tho the CLISH config showed "set grub2-password-hash

grub.pbkdf2.sha512.10000.<string>". I know what I set that to when I did that on the original Gaia FTW of the appliance initial install. I copied it from another CLISH config where I set it manually at the CLI, and typed in the password string.

I'll update my write-up for TAC with this info, too.

EDIT: This message is still relevant, because we still had the Grub boot menu issue despite the above systemctl script issue.

Re: R82.10 snapshot revert error

Duane_Toler — Wed, 03 Jun 2026 21:14:56 GMT

I'm also gonna ask the customer to hang back one day at the office to see if we can do our own reboot tests and sanity-check the grub password mess. Too bad I don't have one of these on hand to test myself. With all these things, I'd bang on it and beat it to death with tests. I would've imagined R&D had done similar, but maybe they were pressured to get them shipped out the door... "we'll fix it in post! HFA-4-evah!".

Re: R82.10 snapshot revert error

Duane_Toler — Thu, 11 Jun 2026 14:32:00 GMT

EDIT: the below script info is not relevant now, but I'm leaving it here because the diagnostic path itself is still useful; even if the end result was bogus.

~~Did the confirmation test last night. reboot and shutdown are broken on R82.10.~~

[Expert@gateway01:0]# which reboot alias reboot='/bin/system_reboot' /bin/system_reboot [Expert@gateway01:0]# ls -l /bin/system_reboot -rwxr-xr-x 1 admin root 3996 May 4 08:39 /bin/system_reboot [Expert@gateway01:0]# file /bin/system_reboot /bin/system_reboot: POSIX shell script, ASCII text executable

~~So "reboot" is just a link to a shell script. Interesting.~~

~~(omitting the ceremonial parts)~~

[Expert@gateway01:0]# cat /bin/system_reboot #!/bin/sh # # reboot wrapper # # ...skip... # function kill_fwd_gracefully { # skip ... # cpwd_admin stop -name FWD -path "$FWDIR/bin/fw" -command "fw kill fwd" & > /dev/null } function down_and_reboot { cluster_down sleep 3 if $isVSX; then ensure_context_is_VS0 fi logger "Set interfaces down before reboot" for x in $(get_phys_ifs) ; do $IP link set dev $x down ## INTERFACES GO DOWN; NO REMOTE ACCESS done if $isVSX; then down_vs_interfaces fi kill_fwd_gracefully /sbin/reboot ### <<<< HERE } ### # skip ### ## prompt Are you sure?(y/n) askreally || { echo "Reboot aborted." 2>&1 exit 1 } down_and_reboot # call above function exit 0 ##<<< HERE 2

~~The "HERE" comments show where the sequence goes wrong. So the script calls another script, which is a link to... what?! a fake systemctl script in Bash!?~~

[Expert@gateway01:0]# file /sbin/reboot /sbin/reboot: symbolic link to ../bin/systemctl [Expert@gateway01:0]# file /bin/systemctl /bin/systemctl: Bourne-Again shell script, ASCII text executable [Expert@gateway01:0]# cat /bin/systemctl #!/bin/bash # service_name=${BASH_ARGV[0]} ## Processes arguments in reverse order service_name=${service_name%%.service} ## creates a SysV-style system service for opt in ${BASH_ARGV[@]} do case $opt in enable) systemd_script=$(grep ExecStart= /usr/lib/systemd/system/${service_name}.service|cut -f 2 -d '='|cut -f 1 -d ' ') init_script=${systemd_script%%.sh} init_script=${init_script/*\//} cp $systemd_script /etc/rc.d/init.d/${init_script} chkconfig --add ${service_name} chkconfig ${service_name} on ;; disable) systemd_script=$(grep ExecStart= /usr/lib/systemd/system/${service_name}.service|cut -f 2 -d '='|cut -f 1 -d ' ') init_script=${systemd_script%%.sh} init_script=${init_script/*\//} chkconfig ${service_name} off chkconfig --del ${service_name} rm /etc/rc.d/init.d/${init_script} ;; start) service ${service_name} start ;; stop) service ${service_name} stop ;; *) ### DEFAULT HANDLER for no arguments ;; ### Does nothing esac done ### END OF SCRIPT ...Nothing happens here?

Uh... this script does nothing. It does a head-fake for a "systemctl" command that just twists itself back to a SysV init script. For "reboot", it reaches the default handler, does nothing, then the end of the script exits, returns back to the caller script, which also does an "exit 0", but only after it happily shuts down the interfaces, locking you out remotely! Now you call the customer, or make the drive yourself, to either pull the power or connect a console cable!

[EDIT: The below is still correct; the Gaia Deployment Agent does use its own reboot handler; I did discover that.]

You can login at console and give PID 1 the "kill -SIGINT 1" command and that forces a reboot. PID 1 is now systemd, not init, and SIGINT (signal 2) causes systemd to run its own default reboot handler, which does work successfully. Meanwhile, all of the expected systemctl commands are fake and just exit 0. I sent this to my TAC case last night.

~~This scenario exists on both GA1 and GA2. Can someone escalate this to R&D?~~

As for the Deployment Agent, this must use its own reboot handler after packages are installed; ~~that has to be the only way I was able to install GA2 on 2 of these gateways.~~

Re: R82.10 snapshot revert error

Bob_Zimmerman — Fri, 05 Jun 2026 15:29:00 GMT

I just checked a 3920 and /bin/systemctl is an ELF for aarch64:

[Expert@SomeFirewall:0]# clish -c "show asset system" | egrep "^[PM]" Platform: BT-91-00 Model: CheckPoint 3920 [Expert@SomeFirewall:0]# file /sbin/{halt,init,poweroff,reboot,shutdown} /sbin/halt: symbolic link to ../bin/systemctl /sbin/init: symbolic link to ../lib/systemd/systemd /sbin/poweroff: symbolic link to ../bin/systemctl /sbin/reboot: symbolic link to ../bin/systemctl /sbin/shutdown: symbolic link to ../bin/systemctl [Expert@SomeFirewall:0]# file /bin/systemctl /bin/systemctl: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=7d40af13734da760dc7294191cb8e75d21f6159f, stripped [Expert@SomeFirewall:0]# file /lib/systemd/systemd /lib/systemd/systemd: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=a60ba79171277846207d64cbed8378406ca6e955, stripped

Same on a 3200 running R82.10 (though obviously x86-64 instead of aarch64). Not doubting what you see, but providing a data point that what you see doesn't seem to always be the case. Not sure what the difference might be between our environments.

'init 6' should give you a clean reboot from systemd directly without needing systemctl. Tested on the 3200.

You should always be able to force a reboot with 'echo b >/proc/sysrq-trigger'. This doesn't stop services or sync filesystems, so could result in data loss (same as pulling the power cord).

Re: R82.10 snapshot revert error

Duane_Toler — Thu, 11 Jun 2026 14:32:54 GMT

These gateways are 3950 and 3980, but still aarch64:

[Expert@gateway01:0]# clish -c "show asset system" Platform: BT-92-00 Model: CheckPoint 3950 Serial Number: WA25804857 CPU Model: Arm..Neoverse..(N2) CPU Frequency: 2100 MHz Number of Cores: 8 CPU Hyperthreading: Disabled [Expert@gateway02:0]# clish -c "show asset system" Platform: BT-93-00 Model: CheckPoint 3980 Serial Number: WA25400346 CPU Model: Arm..Neoverse..(N2) CPU Frequency: 2700 MHz Number of Cores: 8 CPU Hyperthreading: Disabled

That's wild that your systemctl is an ELF binary!! I don't have a 3920 to compare, tho. How in the world did my 3 gateways turn out differently?!? Very wild!

"init 6" doesn't work because PID 1 is now "systemd" not "init" anymore. I definitely tried that, tho. 🙂

~~R82.00 has "'/sbin/reboot" linked to an ELF binary for "/sbin/halt":~~

[Expert@mgmt01:0]# fw ver This is Check Point's software version R82 - Build 019 [Expert@mgmt01:0]# file /sbin/reboot /sbin/reboot: symbolic link to `halt' [Expert@mgmt01:0]# file /sbin/halt /sbin/halt: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), for GNU/Linux 2.6.16, stripped [Expert@mgmt01:0]#

Edit: Yes I did send updated CPINFO to the TAC SR, and the person said they'd file a request with R&D. This is going to be an "interesting" case...

Re: R82.10 snapshot revert error

Bob_Zimmerman — Fri, 05 Jun 2026 16:36:48 GMT

'init 6' definitely works on the R82.10 systems I've tried (two now, including a 3920). It doesn't give any feedback in the terminal where you run it, the system just reboots after 30 seconds or so.

systemd is a nondeterministic trainwreck, so it wouldn't exactly surprise me if it simply failed to reboot on the systems you tried. 'echo b >/proc/sysrq-trigger' only depends on the kernel, so if it's running well enough to let you type commands in a shell, it should work.

Re: R82.10 snapshot revert error

Duane_Toler — Fri, 05 Jun 2026 16:43:07 GMT

Odd. What is your pid 1?

[Expert@gw01:0]# ps -p 1 PID TTY TIME CMD 1 ? 00:00:00 systemd [Expert@gw01:0]#

Re: R82.10 snapshot revert error

Bob_Zimmerman — Fri, 05 Jun 2026 17:22:26 GMT

Connected via a serial server, it gives more output:

[Expert@DallasSC:0]# clish -c "show asset system" | egrep "^[PM]" Platform: PB-10-00 Model: Check Point 3200 [Expert@DallasSC:0]# fwm ver This is Check Point Security Management Server R82.10 - Build 003 [Expert@DallasSC:0]# cpinfo -y all 2>&1 | grep "JUMBO_HF_MAIN" | uniq HOTFIX_R82_10_JUMBO_HF_MAIN Take: 19 BUNDLE_R82_10_JUMBO_HF_MAIN Take: 19 [Expert@DallasSC:0]# ps -p 1 PID TTY TIME CMD 1 ? 00:00:03 systemd [Expert@DallasSC:0]# which init /sbin/init [Expert@DallasSC:0]# file $(which init) /sbin/init: symbolic link to ../lib/systemd/systemd [Expert@DallasSC:0]# init 6 [Expert@DallasSC:0]# [ 6391.698495] bootsplashd[701]: looking up window text geometry [ 6391.698619] bootsplashd[701]: window is now 80x25 text cells [ 6391.878995] wrp.sh[173080]: Removing wrp module [ 6392.032987] blkdeactivate[173063]: Deactivating block devices: [ 6392.065274] blkdeactivate[173063]: [SKIP]: unmount of vg_splat-lv_log (dm-1) mounted on /var/log [ 6392.065641] blkdeactivate[173063]: [SKIP]: unmount of vg_splat-lv_current (dm-0) mounted on / [ 6392.352824] cpboot[173209]: cpwd_admin: [ 6392.353036] cpboot[173209]: Process PROBEMOND isn't monitored by cpWatchDog. Stop request aborts ...

Re: R82.10 snapshot revert error

Duane_Toler — Fri, 05 Jun 2026 17:27:34 GMT

hah! *THAT* we agree on:

[Expert@gw01:0]# which init /sbin/init [Expert@ gw01:0]# ls -l /sbin/init lrwxrwxrwx 1 admin root 22 Jun 2 21:11 /sbin/init -> ../lib/systemd/systemd [Expert@ gw01:0]# file /usr/lib/systemd/systemd /usr/lib/systemd/systemd: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=a60ba79171277846207d64cbed8378406ca6e955, stripped

Ok, so this *might* have worked on that one gateway where I tried it, but this was in a state where the Blink installer failed reboot following the stage 1 process, and I believe Deployment Agent (or something) was eating the reboot signal.

So my "that didn't work" statement needed a "*" on it.