$FWDIRbin/cpstop -fwflag -proc almost always never works

[Duane_Toler]

Hey folks,

Working with hotfixes (usually the individual private ones, rather than Jumbo HFA), the installer script eventually calls "$FWDIR/bin/cpstop -fwflag -proc".  Obviously this is intended to stop the services, or at least some portion thereof.  That much I understand.

However, it never works.  Yes I do mean "never".  Ever.  Various products/versions over the years.  I always have to intervene manually or else the process hangs and/or fails.  At times, I've even had to dig into the installer script and manually edit the script to skip that process (no, that's never enjoyable).

I have to believe these scripts are tested somewhere, tho.

This case involves a SmartCenter (no gateway, no endpoint; just management).  R80.40 Jumbo 139 now.  I got a hotfix that is allowed (installer verify said it was good, as did TAC).  It won't go.  It hangs on cpstop -fwflag -proc.  There's no fw kernel driver to stop; it's just management, so this command isn't necessary.  It's just hanging, doing nothing.

[Expert@mercury:0]# ps xafwwwu|grep DASer -A 12

admin     7261  1.4  0.0 105028 20168 ?        Sl   10:45   1:02 /opt/CPda/bin/DAService

admin      407  0.0  0.0   3636  1144 ?        S    11:48   0:00  \_ /bin/sh -c ( . /opt/CPshared/5.0/tmp/.CPprofile.sh >/dev/null 2>&1 ;/opt/CPshrd-R80.40/bin/cpstop -fwflag -proc ) > /opt/CPInstLog/fileHkX9D8 2>&1

admin      408  0.0  0.0   3636   860 ?        S    11:48   0:00      \_ /bin/sh -c ( . /opt/CPshared/5.0/tmp/.CPprofile.sh >/dev/null 2>&1 ;/opt/CPshrd-R80.40/bin/cpstop -fwflag -proc ) > /opt/CPInstLog/fileHkX9D8 2>&1

admin      409  0.0  0.0   5736  2292 ?        S    11:48   0:00          \_ /opt/CPshrd-R80.40/bin/cpstop -fwflag -proc

Sitting there for over 12 minutes now.  Sigh...  I ran cpstop manually ahead of time.  So now what is it trying to do?  It doesn't work.

To R&D:  please don't do this anymore.  I appreciate the work you do. However, this one isn't it the right thing do.

Here's the output of that file being written (/opt/CPInstLog/fileHkX9D8):

tail /opt/CPInstLog/fileHkX9D8 

Stop Search Infrastructure...

Stopping RFL ...

RFL stopped

Stopping Solr ...

There is no process SOLR run.

Stop SmartView ...

Stop Log Indexer... 

Stop SmartLog Server... 

dbsync is not running

SmartView Monitor: Management stopped

Nothing is running because I ran CPSTOP manually first.  So again... what is it waiting to happen?? 

To make this work, I'm gonna move $FWDIR/bin/cpstop to another name, write a 1 line shell script in its place with "exit 0" so it will be happy and move on thinking everything is fine.  😕

[the_rock]

I wish I could tell you otherwise, but had customer with exact same issue, we had case with escalations for 3 months, and then we all got tired of it, left it as is with no solution. I hate to say this, but my impression was that it simply does not work...never, ever, as you said.

[Duane_Toler]

Here's some debug for you.   I installed the 'strace' RPM from the CentOS 5 repo. (btw, we need to have another discussion about this and your toolchain).

[Expert@mercury:0]# strace -p 409

Process 409 attached - interrupt to quit

[ Process PID=409 runs in 32 bit mode. ]

restart_syscall(<... resuming interrupted call ...>) = 0

rt_sigaction(SIGINT, {SIG_IGN}, {SIG_IGN}, 😎 = 0

rt_sigaction(SIGQUIT, {SIG_IGN}, {SIG_IGN}, 😎 = 0

rt_sigprocmask(SIG_BLOCK, [CHLD], [], 😎 = 0

clone(child_stack=0, flags=CLONE_PARENT_SETTID|SIGCHLD, parent_tidptr=0xfff0b7f0) = 8590

waitpid(8590, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0) = 8590

rt_sigaction(SIGINT, {SIG_IGN}, NULL, 😎 = 0

rt_sigaction(SIGQUIT, {SIG_IGN}, NULL, 😎 = 0

rt_sigprocmask(SIG_SETMASK, [], NULL, 😎 = 0

--- SIGCHLD (Child exited) @ 0 (0) ---

rt_sigprocmask(SIG_BLOCK, [CHLD], [], 😎 = 0

rt_sigaction(SIGCHLD, NULL, {SIG_DFL}, 😎 = 0

rt_sigprocmask(SIG_SETMASK, [], NULL, 😎 = 0

nanosleep({1, 0}, {1, 0})               = 0

rt_sigaction(SIGINT, {SIG_IGN}, {SIG_IGN}, 😎 = 0

rt_sigaction(SIGQUIT, {SIG_IGN}, {SIG_IGN}, 😎 = 0

rt_sigprocmask(SIG_BLOCK, [CHLD], [], 😎 = 0

clone(child_stack=0, flags=CLONE_PARENT_SETTID|SIGCHLD, parent_tidptr=0xfff0b7f0) = 8592

waitpid(8592, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0) = 8592

It's calling 'waitpid' repeatedly on process that I can't find.  It cycles roughly every 1000ms and the value for waitpid() is incrementing (as it would).

...

Here's another run of it, a few moments later:  still cycling through calls to 'waitpid()'.

nanosleep({1, 0}, {1, 0})               = 0
rt_sigaction(SIGINT, {SIG_IGN}, {SIG_IGN}, 😎 = 0
rt_sigaction(SIGQUIT, {SIG_IGN}, {SIG_IGN}, 😎 = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 😎 = 0
clone(child_stack=0, flags=CLONE_PARENT_SETTID|SIGCHLD, parent_tidptr=0xfff0b7f0) = 9415
waitpid(9415, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0) = 9415
rt_sigaction(SIGINT, {SIG_IGN}, NULL, 😎 = 0
rt_sigaction(SIGQUIT, {SIG_IGN}, NULL, 😎 = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 😎 = 0
--- SIGCHLD (Child exited) @ 0 (0) ---
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 😎 = 0
rt_sigaction(SIGCHLD, NULL, {SIG_DFL}, 😎 = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 😎 = 0
nanosleep({1, 0}, {1, 0})               = 0
rt_sigaction(SIGINT, {SIG_IGN}, {SIG_IGN}, 😎 = 0
rt_sigaction(SIGQUIT, {SIG_IGN}, {SIG_IGN}, 😎 = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 😎 = 0
clone(child_stack=0, flags=CLONE_PARENT_SETTID|SIGCHLD, parent_tidptr=0xfff0b7f0) = 9417
waitpid(9417, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0) = 9417
rt_sigaction(SIGINT, {SIG_IGN}, NULL, 😎 = 0
rt_sigaction(SIGQUIT, {SIG_IGN}, NULL, 😎 = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 😎 = 0
--- SIGCHLD (Child exited) @ 0 (0) ---
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 😎 = 0
rt_sigaction(SIGCHLD, NULL, {SIG_DFL}, 😎 = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 😎 = 0
nanosleep({1, 0}, ^C <unfinished ...>

[Duane_Toler]

Boom.  And that's how it's done:

[Expert@mercury:0]# cpinfo -y FW1

This is Check Point CPinfo Build 914000219 for GAIA
[FW1]
	HOTFIX_R80_40_JHF_T120_421_MAIN
	HOTFIX_NGM_DOCTOR_AUTOUPDATE
	HOTFIX_GOT_TPCONF_MGMT_AUTOUPDATE
	HOTFIX_R80_40_JUMBO_HF_MAIN	Take:  139

cd $CPDIR/bin

mv cpstop cpstop.orig

cat - >cpstop <<EOF

#/bin/sh

logger -t new_cpstop -p info "Running new cpstop"

$CPDIR/bin/cpstop.orig  # Because I'm such a nice perso

exit 0   # I really don't like having to do this

EOF
chmod 755 cpstop

clish

installer install 1 not-interactive

<wait about a minute... it's a small hotfix>

<auto-reboot>

Ta-Da!

Please don't use -fwflag -proc.  Please?

 [edit:  include chmod 755, which i did just didn't write it here the first time]