R82.10 snapshot revert error

[Duane_Toler]

So this is new...

gateway01> set snapshot-onetime revert target lvm name Blink_R82.10_402
NMSNAP0042  Bad snapshot name. Snapshot name may only contain up to 15 digits, letters and underscores (_)

3950 appliance, R82.10, take 271

Had an in-place Blink upgrade to take 467 fail, but now I can't revert back to try it again.  This worked on 2 other 3900s (one 3950 and one 3980).  TAC case is open for the upgrade failure issue, tho.

[the_rock]

Interesting, never seen that before. Hope they can sort it out...but, is that actual snapshot name?

[Duane_Toler]

yep!

gateway01> show snapshots
Restore points:
---------------
Blink_R82.10_402

[emmap]

It doesn't like the full stop/decimal point in the name, I'd guess. Though really it shouldn't have let you take it in the first place if that's the case. 

[Duane_Toler]

This was the snapshot created by the Blink upgrade, automatically.  The manual command indeed refuses to make one with that name pattern.

On an R82.10 Take 467 gateway:

gateway02> add snapshot-onetime name Foo_R82.10 description FooSnap 
NMSNAP9999  Snapshot name may only contain digits, letters and underscores. Maximum 255 characters

Looks like the Left Thumb didn't talk to the Left Pinky about this.  Oops.  TAC case again?

[Duane_Toler]

Another bonus, that I didn't check first, is that the snapshot name from a Blink upgrade is 16 characters.  Oops x2.  The TAC person said no one over there knows how to recover from such a situation as an "interrupted" upgrade like this.

Deployment Agent as a whole is angry because /mnt/fcd is still present, and has a path opt/ which contains the various Deployment Agent logs.  These were copied over by the installer from /opt/CPInstLog, according to the last line of the log:

[2026-05-28 - 20:15:24][294723 295414]:Copying CPInstLog dir
[2026-05-28 - 20:15:24][294723 295414]:about to copy dir /opt/CPInstLog/ to /mnt/fcd//opt
[2026-05-28 - 20:15:24][294723 295414]:About to execute command: /bin/cp -af /opt/CPInstLog/ /mnt/fcd//opt

 The appliance rebooted, but since the process died, it can't recover.

I'm going to schedule a plain reboot tonight with the customer; perhaps it'll pick up where it left off and try Stage 2 again...? maybe?

[the_rock]

Hope all goes well.

[emmap]

That sounds very much like TAC needs to be escalating that information to the Blink owners internally.

[Duane_Toler]

Yep, they're waiting on me for some follow-up info which I was able to collect today.  Short version:

1. The gateway actually  never even rebooted, but I thought it did!  It looked like it did, tho.  When we pulled the power cable to hard cycle, it booted up and continued onwards to finish the upgrade as expected.  Everything seems relatively fine after that now.

2. Reboot and shutdown scripts on R82.10 appear to be nothing but a NO-OP.  The scripts shutdown FWD, shutdown the interfaces, then....exit.  I'm not sure how reboot signals are expected to be handled, but I don't see anywhere this is happening.  We couldn't reboot it from the 'reboot' or 'shutdown -r now' commands.  Something is amiss here.  This is why we had to do the hard power cycle above.

3. "This is not the systemctl you're looking for".  /bin/systemctl is just a wrapper Bash script that calls the SysV init scripts.  Um.. what?  Systemd appears to be running things from the places we expect, which is how it picked up the post-boot script.

I also had a Jumbo HFA install fail tonight on a second gateway.  It was already Take 467 with Jumbo HFA 6; I did the initial re-install/upgrade before putting the gateway into service.  This one clearly rebooted successfully at the time.  Tonight, I was installing Jumbo HFA 19 (for a VPN degradation issue they're having, that seems to be addressed in JHF 19). 

It seems to have cranked the CPU and some system processes, then ... died.  It was at the 85% level of "copying files..." when it just died.  No console output when the customer connected... dunno if it even rebooted.  The grub2 password we set also didn't work, so no snapshot revert option, no maintenance mode, nothing.  Had no choice but to do factory reset.

In short...this has been a very painful experience with the 3900s.  Nothing positive to say about these yet.

EDIT:  I'm leaving the above, as a public confession of my sins and guilt. 🙂  I found the issue here and it's... sigh... embarrassingly awful. The systemctl script issue wasn't with the JHF or the Blink image after all.  I'm gonna update the TAC SR as well. We had a deployment playbook that mis-categorized the 3900s and applied the wrong package to them.

I'm gonna update the other messages here as well.

Thanks @Bob_Zimmerman  for your verification info as well! That was helpful to see!

[emmap]

What was your upgrade path here? Was it raw R82.10 build 271 that you then upgraded with the specific 3900 blink package? 

[Duane_Toler]

Out-of-box, it was the factory original Take 271 (GA1). I then installed the Quantum Force 3900 Take 22 hotfix; (GA2 wasn't available then).  Then I did Blink in-place upgrade to Take 467 (GA2) when it came available with Jumbo HFA Take 6 (new LVM volume anyway, with LV snapshot of 271).  That process went just fine originally, just as expected, and the gateway was operating for a few weeks successfully.  Very unremarkable, as it should be.

Now, I went to install the Jumbo HFA 19 package like any other Jumbo HFA would.  I always run verifier before doing anything, too, and verifier succeeded with "install.applicable: true" and no warnings.

It went through the routine, then got to the meaningless 85% with message "Copying files".  Sat there for a long time, which is fine... I always see it sit here with no updates.  I played a card game of "Go Fish" with my daughter over dinner to pass the time.  Eventually, the system became unresponsive... then my SSH session was killed and that was the end.  No remote access, so customer traveled to the office.  After about an hour, customer connected console cable but got no output. We let it sit for several more minutes but still nothing.

Had no meaningful option but to power cycle it.  I hated to do it, but I knew we could revert to some prior snapshot.  Then that turned out to be a bad choice.  None of our input choices worked, even tho the CLISH config showed "set grub2-password-hash 

grub.pbkdf2.sha512.10000.<string>".  I know what I set that to when I did that on the original Gaia FTW of the appliance initial install.  I copied it from another CLISH config where I set it manually at the CLI, and typed in the password string.

I'll update my write-up for TAC with this info, too.

EDIT: This message is still relevant, because we still had the Grub boot menu issue despite the above systemctl script issue.

[Duane_Toler]

I'm also gonna ask the customer to hang back one day at the office to see if we can do our own reboot tests and sanity-check the grub password mess.  Too bad I don't have one of these on hand to test myself.  With all these things, I'd bang on it and beat it to death with tests.  I would've imagined R&D had done similar, but maybe they were pressured to get them shipped out the door... "we'll fix it in post! HFA-4-evah!".

[Duane_Toler]

EDIT: the below script info is not relevant now, but I'm leaving it here because the diagnostic path itself is still useful; even if the end result was bogus.

Did the confirmation test last night.  reboot and shutdown are broken on R82.10.

[Expert@gateway01:0]# which reboot
alias reboot='/bin/system_reboot'
/bin/system_reboot

[Expert@gateway01:0]# ls -l /bin/system_reboot 
-rwxr-xr-x 1 admin root 3996 May  4 08:39 /bin/system_reboot

[Expert@gateway01:0]# file /bin/system_reboot 
/bin/system_reboot: POSIX shell script, ASCII text executable

So "reboot" is just a link  to a shell script.  Interesting. 

(omitting the ceremonial parts)

[Expert@gateway01:0]# cat /bin/system_reboot 
#!/bin/sh
#
# reboot wrapper

#
# ...skip...
#

function kill_fwd_gracefully {
# skip ...
#
cpwd_admin stop -name FWD -path "$FWDIR/bin/fw" -command "fw kill fwd" & > /dev/null 

}

function down_and_reboot {

cluster_down
sleep 3
if $isVSX; then
ensure_context_is_VS0
fi
logger "Set interfaces down before reboot"

for x in $(get_phys_ifs) ; do
$IP link set dev $x down  ## INTERFACES GO DOWN; NO REMOTE ACCESS
done

if $isVSX; then
down_vs_interfaces
fi

kill_fwd_gracefully
/sbin/reboot  ### <<<< HERE

}

###
# skip
###

## prompt Are you sure?(y/n)
askreally || {
  echo "Reboot aborted." 2>&1
  exit 1
}

down_and_reboot  # call above function

exit 0   ##<<< HERE 2

The "HERE" comments show where the sequence goes wrong.  So the script calls another script, which is a link to... what?! a fake systemctl script in Bash!?

[Expert@gateway01:0]# file /sbin/reboot 
/sbin/reboot: symbolic link to ../bin/systemctl

[Expert@gateway01:0]# file /bin/systemctl 
/bin/systemctl: Bourne-Again shell script, ASCII text executable


[Expert@gateway01:0]# cat /bin/systemctl 
#!/bin/bash
#

service_name=${BASH_ARGV[0]}            ## Processes arguments in reverse order
service_name=${service_name%%.service}  ## creates a SysV-style system service

for opt in ${BASH_ARGV[@]}
do
  case $opt in
    enable)
      systemd_script=$(grep ExecStart= /usr/lib/systemd/system/${service_name}.service|cut -f 2 -d '='|cut -f 1 -d ' ')
      init_script=${systemd_script%%.sh}
      init_script=${init_script/*\//}
      cp $systemd_script /etc/rc.d/init.d/${init_script}

      chkconfig --add ${service_name}
      chkconfig ${service_name} on
    ;;

    disable)
      systemd_script=$(grep ExecStart= /usr/lib/systemd/system/${service_name}.service|cut -f 2 -d '='|cut -f 1 -d ' ')
      init_script=${systemd_script%%.sh}
      init_script=${init_script/*\//}
      chkconfig ${service_name} off
      chkconfig --del ${service_name}

      rm /etc/rc.d/init.d/${init_script}
    ;;

    start)
      service ${service_name} start
    ;;

    stop)
      service ${service_name} stop
    ;;

    *)   ### DEFAULT HANDLER for no arguments
    ;;   ### Does nothing
  esac
done

### END OF SCRIPT ...Nothing happens here?

Uh... this script does nothing.  It does a head-fake for a "systemctl" command that just twists itself back to a SysV init script.  For "reboot", it reaches the default handler, does nothing, then the end of the script exits,  returns back to the caller script, which also does an "exit 0", but only after it happily shuts down the interfaces, locking you out remotely!  Now you call the customer, or make the drive yourself, to either pull the power or connect a console cable!

[EDIT: The below is still correct; the Gaia Deployment Agent does use its own reboot handler; I did discover that.]

You can login at console and give PID 1 the "kill -SIGINT 1" command and that forces a reboot.  PID 1 is now systemd, not init, and SIGINT (signal 2) causes systemd to run its own default reboot handler, which does work successfully.  Meanwhile, all of the expected systemctl commands are fake and just exit 0.  I sent this to my TAC case last night.

This scenario exists on both GA1 and GA2.  Can someone escalate this to R&D?

As for the Deployment Agent, this must use its own reboot handler after packages are installed; that has to be the only way I was able to install GA2 on 2 of these gateways.

[Bob_Zimmerman]

I just checked a 3920 and /bin/systemctl is an ELF for aarch64:

[Expert@SomeFirewall:0]# clish -c "show asset system" | egrep "^[PM]"
Platform: BT-91-00
Model: CheckPoint 3920

[Expert@SomeFirewall:0]# file /sbin/{halt,init,poweroff,reboot,shutdown}
/sbin/halt:     symbolic link to ../bin/systemctl
/sbin/init:     symbolic link to ../lib/systemd/systemd
/sbin/poweroff: symbolic link to ../bin/systemctl
/sbin/reboot:   symbolic link to ../bin/systemctl
/sbin/shutdown: symbolic link to ../bin/systemctl

[Expert@SomeFirewall:0]# file /bin/systemctl 
/bin/systemctl: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=7d40af13734da760dc7294191cb8e75d21f6159f, stripped

[Expert@SomeFirewall:0]# file /lib/systemd/systemd
/lib/systemd/systemd: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=a60ba79171277846207d64cbed8378406ca6e955, stripped

Same on a 3200 running R82.10 (though obviously x86-64 instead of aarch64). Not doubting what you see, but providing a data point that what you see doesn't seem to always be the case. Not sure what the difference might be between our environments.

'init 6' should give you a clean reboot from systemd directly without needing systemctl. Tested on the 3200.

You should always be able to force a reboot with 'echo b >/proc/sysrq-trigger'. This doesn't stop services or sync filesystems, so could result in data loss (same as pulling the power cord).

[Duane_Toler]

These gateways are 3950 and 3980, but still aarch64:

[Expert@gateway01:0]# clish -c "show asset system"
Platform: BT-92-00
Model: CheckPoint 3950
Serial Number: WA25804857
CPU Model: Arm..Neoverse..(N2)
CPU Frequency: 2100 MHz
Number of Cores: 8
CPU Hyperthreading: Disabled

[Expert@gateway02:0]# clish -c "show asset system"
Platform: BT-93-00
Model: CheckPoint 3980
Serial Number: WA25400346
CPU Model: Arm..Neoverse..(N2)
CPU Frequency: 2700 MHz
Number of Cores: 8
CPU Hyperthreading: Disabled

That's wild that your systemctl is an ELF binary!! I don't have a 3920 to compare, tho.  How in the world did my 3 gateways turn out differently?!? Very wild!

"init 6" doesn't work because PID 1 is now "systemd" not "init" anymore.  I definitely tried that, tho. 🙂

R82.00 has "'/sbin/reboot" linked to an ELF binary for "/sbin/halt":

[Expert@mgmt01:0]# fw ver
This is Check Point's software version R82 - Build 019
[Expert@mgmt01:0]# file /sbin/reboot 
/sbin/reboot: symbolic link to `halt'
[Expert@mgmt01:0]# file /sbin/halt 
/sbin/halt: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), for GNU/Linux 2.6.16, stripped
[Expert@mgmt01:0]# 

Edit: Yes I did send updated CPINFO to the TAC SR, and the person said they'd file a request with R&D.  This is going to be an "interesting" case...

[Bob_Zimmerman]

'init 6' definitely works on the R82.10 systems I've tried (two now, including a 3920). It doesn't give any feedback in the terminal where you run it, the system just reboots after 30 seconds or so.

systemd is a nondeterministic trainwreck, so it wouldn't exactly surprise me if it simply failed to reboot on the systems you tried. 'echo b >/proc/sysrq-trigger' only depends on the kernel, so if it's running well enough to let you type commands in a shell, it should work.

[Duane_Toler]

Odd.  What is your pid 1? 

[Expert@gw01:0]# ps -p 1
    PID TTY          TIME CMD
      1 ?        00:00:00 systemd
[Expert@gw01:0]# 

[Bob_Zimmerman]

Connected via a serial server, it gives more output:

[Expert@DallasSC:0]# clish -c "show asset system" | egrep "^[PM]"
Platform: PB-10-00
Model: Check Point 3200

[Expert@DallasSC:0]# fwm ver
This is Check Point Security Management Server R82.10 - Build 003

[Expert@DallasSC:0]# cpinfo -y all 2>&1 | grep "JUMBO_HF_MAIN" | uniq
	HOTFIX_R82_10_JUMBO_HF_MAIN	Take:  19
	BUNDLE_R82_10_JUMBO_HF_MAIN	Take:  19

[Expert@DallasSC:0]# ps -p 1
    PID TTY          TIME CMD
      1 ?        00:00:03 systemd

[Expert@DallasSC:0]# which init
/sbin/init

[Expert@DallasSC:0]# file $(which init)
/sbin/init: symbolic link to ../lib/systemd/systemd

[Expert@DallasSC:0]# init 6

[Expert@DallasSC:0]# [ 6391.698495] bootsplashd[701]: looking up window text geometry
[ 6391.698619] bootsplashd[701]: window is now 80x25 text cells
[ 6391.878995] wrp.sh[173080]: Removing wrp module
[ 6392.032987] blkdeactivate[173063]: Deactivating block devices:
[ 6392.065274] blkdeactivate[173063]:   [SKIP]: unmount of vg_splat-lv_log (dm-1) mounted on /var/log
[ 6392.065641] blkdeactivate[173063]:   [SKIP]: unmount of vg_splat-lv_current (dm-0) mounted on /
[ 6392.352824] cpboot[173209]: cpwd_admin:
[ 6392.353036] cpboot[173209]: Process PROBEMOND isn't monitored by cpWatchDog. Stop request aborts
...

[Duane_Toler]

hah!  *THAT* we agree on:

[Expert@gw01:0]# which init
/sbin/init

[Expert@ gw01:0]# ls -l /sbin/init
lrwxrwxrwx 1 admin root 22 Jun  2 21:11 /sbin/init -> ../lib/systemd/systemd

[Expert@ gw01:0]# file /usr/lib/systemd/systemd
/usr/lib/systemd/systemd: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=a60ba79171277846207d64cbed8378406ca6e955, stripped

Ok, so this *might* have worked on that one gateway where I tried it, but this was in a state where the Blink installer failed reboot following the stage 1 process, and I believe Deployment Agent (or something) was eating the reboot signal.

So my "that didn't work" statement needed a "*" on it.

[Duane_Toler]

I found it.

Let me guess:  You did your 3900 gateway with Blink + JHF 19?  All of this is now fixed, but *ONLY* in the Blink images.

I took my customer's factory default-reset Take 271 gateway back to Take 467 with Blink+JHF19 package tonight, and... ta-da!  Now it's all "magically fixed".  It looks just like yours.  I also skipped the Take22 for 3900 interim package; I saw no point in it since I was doing the Blink upgrade anyway.

[EDIT: the above is incorrect now; it was never my issue]

Ever curious,  I extracted the Blink+JHF19 package and found that /usr/sbin/systemctl is now an ELF binary-sized file:

[Expert@ralcp1:0]# cd /var/log/CPda/repository/CheckPoint#Major#All#6.0#5#6#BLINK_R82_10_T467_JHF_T19_GW/
[Expert@ralcp1:0]# tar -tvf Blink_image_1.1_Check_Point_R82.10_T467_JHF_T19_aarch64_SecurityGateway.tgz
-rwxr-xr-x builder/fw  8334200 2026-05-25 15:49 BlinkInstaller
-rw-r--r-- builder/fw     1066 2026-05-25 15:49 BlinkInstaller.config
-rw-r--r-- builder/fw      512 2026-05-25 15:49 BlinkInstaller.sha256
-rw-r--r-- builder/fw 5710360429 2026-05-25 15:49 CheckPoint_Gaia_fd.tgz
-rw-r--r-- builder/fw        512 2026-05-25 15:49 CheckPoint_Gaia_fd.tgz.sha256
# skip the rest
[Expert@ralcp1:0]# tar -xvf Blink_image_1.1_Check_Point_R82.10_T467_JHF_T19_aarch64_SecurityGateway.tgz  CheckPoint_Gaia_fd.tgz

[Expert@ralcp1:0]# tar -tzvf CheckPoint_Gaia_fd.tgz  |grep systemctl
-rw-r--r-- admin/root      15606 2026-03-12 08:14 usr/share/zsh/site-functions/_systemctl
-rw-r--r-- admin/root      19009 2026-03-12 08:14 usr/share/man/man1/systemctl.1.gz
-rw-r--r-- admin/root      13575 2026-03-12 08:14 usr/share/bash-completion/completions/systemctl
lrwxrwxrwx admin/root          0 2026-05-25 15:09 usr/sbin/halt -> ../bin/systemctl
lrwxrwxrwx admin/root          0 2026-05-25 15:09 usr/sbin/poweroff -> ../bin/systemctl
lrwxrwxrwx admin/root          0 2026-05-25 15:09 usr/sbin/reboot -> ../bin/systemctl
lrwxrwxrwx admin/root          0 2026-05-25 15:09 usr/sbin/runlevel -> ../bin/systemctl
lrwxrwxrwx admin/root          0 2026-05-25 15:09 usr/sbin/shutdown -> ../bin/systemctl
lrwxrwxrwx admin/root          0 2026-05-25 15:09 usr/sbin/telinit -> ../bin/systemctl
-rwxr-xr-x admin/root     202080 2026-03-12 08:15 usr/bin/systemctl

All the links are in place, too.

I compared this with the JHF-only package and the systemd RPM (that owns /usr/bin/systemctl) is not included.  So this issue is NOT fixed in the JHF-only packages; it's indeed Blink only, and appears to be fixed only in Blink + JHF 19 (my best guess).

I did check a Blink+JHF6 package for a comparison, and it, too, has the same systemctl ELF-sized binary.  However, best I can tell is that something post-install on Blink+JHF 6 may have overwritten it with the (broken) Bash stub again. I have a gateway that was Take 271 that I did Blink+JHF6 upgrade, but yet it still has the Bash stub instead of the ELF binary.

This doesn't make much sense, I admit.  Tomorrow, on this Blink+JHF6 gateway, I'm going run a Blink+JHF19 re-upgrade since it was successful tonight. 

Gotta admit... it's quite infuriating.  

[Duane_Toler]

Wrapping this up, the systemctl and reboot script issue wasn't a Blink or JHF error after all.  I was wrong on that. Apologies for the noise on the board.

All was not lost, as there was some useful info in the diagnostic path, including the searching through the Blink images versus JHF packages. Noticing the Deployment Agent's own reboot handler was also useful info, even as footnote.

There is still an issue regarding the Grub menu maintenance password as well as the Gaia snapshot name format.  I'll take these up with TAC separately.

Thanks to @emmap and @Bob_Zimmerman for your contributions, as well as eyes-and-ears.

"And now back to your regularly-scheduled program, already in progress..."

[Bob_Zimmerman]

Maybe try reverting in GRUB? It may have fewer restrictions.

That said, I've had major snapshot problems on two separate systems, now. I've taken a snapshot on R82, upgraded the system to R82.10, reverted to the snapshot, and wound up with a totally hosed system. Both would hang at different points during boot. Went into single-user, and lv_current was 100% used.