topic Re: SFWD process crash in Spark Firewall (SMB)

SFWD process crash

HristoGrigorov — Tue, 22 May 2018 12:00:54 GMT

Hi,

has any of you ran into something like this:

[cpWatchDog 2765 1744478736]@RD6281[18 May 9:20:47] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2765 1744478736]@RD6281[18 May 9:21:47] [SUCCESS] SFWD started successfully (pid=3551)
[cpWatchDog 2765 1744478736]@RD6281[18 May 9:35:25] [ERROR] Process SFWD terminated abnormally : Unhandled signal 6 (). Core dumped.
[cpWatchDog 2765 1744478736]@RD6281[18 May 9:36:25] [SUCCESS] SFWD started successfully (pid=4359)
[cpWatchDog 2765 1744478736]@RD6281[20 May 9:41:20] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2765 1744478736]@RD6281[20 May 9:42:20] [SUCCESS] SFWD started successfully (pid=18894)
[cpWatchDog 2765 1744478736]@RD6281[20 May 16:12:09] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2765 1744478736]@RD6281[20 May 16:13:09] [SUCCESS] SFWD started successfully (pid=20929)
[cpWatchDog 2765 1744478736]@RD6281[21 May 8:50:27] [ERROR] Process SFWD terminated abnormally : Unhandled signal 6 (). Core dumped.
[cpWatchDog 2765 1744478736]@RD6281[21 May 8:51:27] [SUCCESS] SFWD started successfully (pid=25168)
[cpWatchDog 2765 1744478736]@RD6281[21 May 9:14:11] [ERROR] Process SFWD terminated abnormally : Unhandled signal 6 (). Core dumped.
[cpWatchDog 2765 1744478736]@RD6281[21 May 9:15:11] [SUCCESS] SFWD started successfully (pid=25918)
[cpWatchDog 2765 1744478736]@RD6281[21 May 10:54:58] [ERROR] Process SFWD terminated abnormally :

Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2765 1744478736]@RD6281[21 May 10:55:58] [SUCCESS] SFWD started successfully (pid=27458)
[cpWatchDog 2765 1744478736]@RD6281[21 May 12:35:14] [ERROR] Process SFWD terminated abnormally : Unhandled signal 6 (). Core dumped.
[cpWatchDog 2765 1744478736]@RD6281[21 May 12:36:14] [SUCCESS] SFWD started successfully (pid=29000)
[cpWatchDog 2765 1744478736]@RD6281[22 May 14:53:05] [ERROR] Process SFWD terminated abnormally : Unhandled signal 6 (). Core dumped.
[cpWatchDog 2765 1744478736]@RD6281[22 May 14:54:05] [SUCCESS] SFWD started successfully (pid=5887)

Re: SFWD process crash

G_W_Albrecht — Wed, 23 May 2018 07:02:04 GMT

The old dumping issue - process SFWD needs some time off i think . In the shown timespan it crashes at least twice a day, so i would use USB Medium firmware update to rule out flash issues. This is from /var/log/log/cpwd.elg, what about /var/log/log/sfwd.el* and /var/log/messages ?

Re: SFWD process crash

PhoneBoy — Wed, 23 May 2018 08:27:04 GMT

Please open a TAC case so we can investigate.

Contact Support | Check Point Software

Re: SFWD process crash

HristoGrigorov — Thu, 24 May 2018 03:14:46 GMT

Yes, sorry I forgot to mention SR is currently under investigation.

Re: SFWD process crash

HristoGrigorov — Thu, 24 May 2018 03:15:35 GMT

Nothing suspicious there. But there is a core dump of the process that I send to R&D for examination.

Re: SFWD process crash

G_W_Albrecht — Thu, 24 May 2018 06:47:26 GMT

Yes, that is the thing to do here - as you can not replicate the crashing sfwd issue.

Re: SFWD process crash

Pedro_Espindola — Tue, 25 Sep 2018 19:12:43 GMT

Hello Hristo, how is the investigation going? Did you receive any news yet?

I found out I have 3 appliances in which the issue does not happen and 7 in which it happens. I am opening an SR to try to find out what is special about those 3 and maybe identify what is wrong in the other 7.

Re: SFWD process crash

HristoGrigorov — Tue, 09 Oct 2018 07:36:52 GMT

Hello Pedro,

Sorry for the late reply, I have somehow missed your question.

I opened SR on 18.09.2018 and since then there was not much progress with it. I have provided core dump to TAC but they said they cannot find anything interesting in it and asked for remote session few days ago. So far, nobody contacted me yet about it. In the mean time, situation is still the same and some times it is getting real bad. I mean just look at this:

[cpWatchDog 2158 1744364432]@RD6281[8 Oct 8:23:37] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 8:24:37] [SUCCESS] SFWD started successfully (pid=11666)
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 9:52:31] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 9:53:31] [SUCCESS] SFWD started successfully (pid=14153)
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 10:23:28] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 10:24:28] [SUCCESS] SFWD started successfully (pid=15008)
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 10:28:48] [ERROR] Process SFWD terminated abnormally : Unhandled signal 6 (). Core dumped.
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 10:29:48] [SUCCESS] SFWD started successfully (pid=15353)
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 10:34:56] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 10:35:56] [SUCCESS] SFWD started successfully (pid=15773)
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 10:51:45] [ERROR] Process SFWD terminated abnormally : Unhandled signal 6 (). Core dumped.
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 10:52:45] [SUCCESS] SFWD started successfully (pid=16393)
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 11:05:51] [ERROR] Process SFWD terminated abnormally : Unhandled signal 6 (). Core dumped.
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 11:06:51] [SUCCESS] SFWD started successfully (pid=16965)
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 11:12:31] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 11:13:31] [SUCCESS] SFWD started successfully (pid=17364)
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 16:42:06] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 16:43:06] [SUCCESS] SFWD started successfully (pid=22701)
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 21:29:02] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 21:30:02] [SUCCESS] SFWD started successfully (pid=23911)

Just prior the crash the system load average will go real high, 3.00 - 5.00 or so. I wonder what is difference between those 3 appliances (where you do not have such issue) and the other 7....

Re: SFWD process crash

HristoGrigorov — Tue, 09 Oct 2018 07:42:08 GMT

Btw, do you have Site-to-Site VPNs on those 3 appliances?

Re: SFWD process crash

Pedro_Espindola — Tue, 09 Oct 2018 14:17:46 GMT

Yes, I do have 3 tunnels configured in one of the 3 that work, with Azure and AWS.

Did they send you any builds? I opened a SR and they sent me a new build for 700/1400. Number of crashes were drastically reduced. I can now go a whole day without a crash.

Check with them if there is any update.

Re: SFWD process crash

HristoGrigorov — Tue, 09 Oct 2018 16:33:07 GMT

No, no builds for me yet. Perhaps they want to sort it out for you first and then sent me the same build

Do not agree on half-working solution!!! Ask for super-duper-rock-stable build that lasts at least a week (think it is anyway good to reboot these gnomes every weekend).

Re: SFWD process crash

HristoGrigorov — Thu, 11 Oct 2018 10:20:52 GMT

My SR has just been "escalated to Tier 3" (don't know if that is good or bad) and I am now waiting for TAC to contact me. In the mean time there us a new record on the number of crashes today:

[cpWatchDog 2158 1744515984]@RD6281[11 Oct 12:52:38] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2158 1744515984]@RD6281[11 Oct 12:56:06] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2158 1744515984]@RD6281[11 Oct 13:02:37] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2158 1744515984]@RD6281[11 Oct 13:06:55] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.

Re: SFWD process crash

HristoGrigorov — Mon, 29 Oct 2018 07:32:31 GMT

Well, don't want to sound over-optimistically here, but R77.20.81 seems to have solved sfwd crashes for me.

@Pedro: Have you tried it already? Is it the same for you?

Re: SFWD process crash

PhoneBoy — Mon, 29 Oct 2018 08:41:36 GMT

The feedback is always useful even if not 100% definitive

Re: SFWD process crash

Pedro_Espindola — Mon, 29 Oct 2018 14:43:02 GMT

Good to know Hristo,

I did not try this one yet, but I am using the build 451 of R77.20.80, which solved it for me. It probably has the same fix shipped in the new version.

I will try R77.20.81 as soon as possible and leave my feedback here.

Re: SFWD process crash

HristoGrigorov — Mon, 29 Oct 2018 16:33:43 GMT

Yeah, probably it is just like that. I was just asked by the devs to try latest firmware and see if it fixes the problem for me as well.

Re: SFWD process crash

HristoGrigorov — Mon, 05 Nov 2018 14:11:33 GMT

SFWD crashes are now history here. Good job CheckPoint! It took a while to fix it but end result is satisfying. One more step of having more stable SMBs.

Re: SFWD process crash

HristoGrigorov — Wed, 14 Nov 2018 17:15:04 GMT

Surprise!!! Crashed again today. Right after policy install. But I do not think that is the same "region" of code that was causing the crash before. Most impressive was that system load average sky rocketed to ~14 right before it.

I will see if it happens again. Definitely something very specific did it because it is otherwise very stable.

Re: SFWD process crash

RS_Daniel — Fri, 24 Jan 2020 15:14:48 GMT

Hi, Hristo... do you remember which directory was your core dump in? or the name?... thanks✌️

Re: SFWD process crash

HristoGrigorov — Fri, 24 Jan 2020 16:48:27 GMT

Nowadays all crashes (core dumps) and panics are placed in /logs directory.