SFWD process crash

[HristoGrigorov]

Hi,

has any of you ran into something like this:

[cpWatchDog 2765 1744478736]@RD6281[18 May  9:20:47] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2765 1744478736]@RD6281[18 May  9:21:47] [SUCCESS] SFWD started successfully (pid=3551)
[cpWatchDog 2765 1744478736]@RD6281[18 May  9:35:25] [ERROR] Process SFWD terminated abnormally : Unhandled signal 6 (). Core dumped.
[cpWatchDog 2765 1744478736]@RD6281[18 May  9:36:25] [SUCCESS] SFWD started successfully (pid=4359)
[cpWatchDog 2765 1744478736]@RD6281[20 May  9:41:20] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2765 1744478736]@RD6281[20 May  9:42:20] [SUCCESS] SFWD started successfully (pid=18894)
[cpWatchDog 2765 1744478736]@RD6281[20 May 16:12:09] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2765 1744478736]@RD6281[20 May 16:13:09] [SUCCESS] SFWD started successfully (pid=20929)
[cpWatchDog 2765 1744478736]@RD6281[21 May  8:50:27] [ERROR] Process SFWD terminated abnormally : Unhandled signal 6 (). Core dumped.
[cpWatchDog 2765 1744478736]@RD6281[21 May  8:51:27] [SUCCESS] SFWD started successfully (pid=25168)
[cpWatchDog 2765 1744478736]@RD6281[21 May  9:14:11] [ERROR] Process SFWD terminated abnormally : Unhandled signal 6 (). Core dumped.
[cpWatchDog 2765 1744478736]@RD6281[21 May  9:15:11] [SUCCESS] SFWD started successfully (pid=25918)
[cpWatchDog 2765 1744478736]@RD6281[21 May 10:54:58] [ERROR] Process SFWD terminated abnormally :

Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2765 1744478736]@RD6281[21 May 10:55:58] [SUCCESS] SFWD started successfully (pid=27458)
[cpWatchDog 2765 1744478736]@RD6281[21 May 12:35:14] [ERROR] Process SFWD terminated abnormally : Unhandled signal 6 (). Core dumped.
[cpWatchDog 2765 1744478736]@RD6281[21 May 12:36:14] [SUCCESS] SFWD started successfully (pid=29000)
[cpWatchDog 2765 1744478736]@RD6281[22 May 14:53:05] [ERROR] Process SFWD terminated abnormally : Unhandled signal 6 (). Core dumped.
[cpWatchDog 2765 1744478736]@RD6281[22 May 14:54:05] [SUCCESS] SFWD started successfully (pid=5887)

[G_W_Albrecht]

The old dumping issue - process SFWD needs some time off i think . In the shown timespan it crashes at least twice a day, so i would use USB Medium firmware update to rule out flash issues. This is from /var/log/log/cpwd.elg, what about /var/log/log/sfwd.el* and /var/log/messages ?

[HristoGrigorov]

Nothing suspicious there. But there is a core dump of the process that I send to R&D for examination.

[G_W_Albrecht]

Yes, that is the thing to do here - as you can not replicate the crashing sfwd issue.

[RS_Daniel]

Hi, Hristo... do you remember which directory was your core dump in? or the name?... thanks✌️ 

[HristoGrigorov]

Nowadays all crashes (core dumps) and panics are placed in /logs directory. 

[PhoneBoy]

Please open a TAC case so we can investigate.

Contact Support | Check Point Software 

[HristoGrigorov]

Yes, sorry I forgot to mention SR is currently under investigation.

[Pedro_Espindola]

Hello Hristo, how is the investigation going? Did you receive any news yet?

I found out I have 3 appliances in which the issue does not happen and 7 in which it happens. I am opening an SR to try to find out what is special about those 3 and maybe identify what is wrong in the other 7.

[HristoGrigorov]

Hello Pedro,

Sorry for the late reply, I have somehow missed your question.

I opened SR on 18.09.2018 and since then there was not much progress with it. I have provided core dump to TAC but they said they cannot find anything interesting in it and asked for remote session few days ago. So far, nobody contacted me yet about it. In the mean time, situation is still the same and some times it is getting real bad. I mean just look at this:

[cpWatchDog 2158 1744364432]@RD6281[8 Oct 8:23:37] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 8:24:37] [SUCCESS] SFWD started successfully (pid=11666)
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 9:52:31] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 9:53:31] [SUCCESS] SFWD started successfully (pid=14153)
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 10:23:28] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 10:24:28] [SUCCESS] SFWD started successfully (pid=15008)
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 10:28:48] [ERROR] Process SFWD terminated abnormally : Unhandled signal 6 (). Core dumped.
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 10:29:48] [SUCCESS] SFWD started successfully (pid=15353)
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 10:34:56] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 10:35:56] [SUCCESS] SFWD started successfully (pid=15773)
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 10:51:45] [ERROR] Process SFWD terminated abnormally : Unhandled signal 6 (). Core dumped.
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 10:52:45] [SUCCESS] SFWD started successfully (pid=16393)
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 11:05:51] [ERROR] Process SFWD terminated abnormally : Unhandled signal 6 (). Core dumped.
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 11:06:51] [SUCCESS] SFWD started successfully (pid=16965)
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 11:12:31] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 11:13:31] [SUCCESS] SFWD started successfully (pid=17364)
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 16:42:06] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 16:43:06] [SUCCESS] SFWD started successfully (pid=22701)
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 21:29:02] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 21:30:02] [SUCCESS] SFWD started successfully (pid=23911)

Just prior the crash the system load average will go real high, 3.00 - 5.00 or so. I wonder what is difference between those 3 appliances (where you do not have such issue) and the other 7....

[HristoGrigorov]

Btw, do you have Site-to-Site VPNs on those 3 appliances? 

[Pedro_Espindola]

Yes, I do have 3 tunnels configured in one of the 3 that work, with Azure and AWS.

Did they send you any builds? I opened a SR and they sent me a new build for 700/1400. Number of crashes were drastically reduced. I can now go a whole day without a crash.

Check with them if there is any update.

[HristoGrigorov]

No, no builds for me yet. Perhaps they want to sort it out for you first and then sent me the same build  

Do not agree on half-working solution!!! Ask for super-duper-rock-stable build that lasts at least a week (think it is anyway good to reboot these gnomes every weekend).

[HristoGrigorov]

My SR has just been "escalated to Tier 3" (don't know if that is good or bad) and I am now waiting for TAC to contact me. In the mean time there us a new record on the number of crashes today:

[cpWatchDog 2158 1744515984]@RD6281[11 Oct 12:52:38] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2158 1744515984]@RD6281[11 Oct 12:56:06] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2158 1744515984]@RD6281[11 Oct 13:02:37] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2158 1744515984]@RD6281[11 Oct 13:06:55] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.

[HristoGrigorov]

Well, don't want to sound over-optimistically here, but R77.20.81 seems to have solved sfwd crashes for me.  

@Pedro: Have you tried it already? Is it the same for you?

[PhoneBoy]

The feedback is always useful even if not 100% definitive 

[Pedro_Espindola]

Good to know Hristo,

I did not try this one yet, but I am using the build 451 of R77.20.80, which solved it for me. It probably has the same fix shipped in the new version.

I will try R77.20.81 as soon as possible and leave my feedback here.

[HristoGrigorov]

Yeah, probably it is just like that. I was just asked by the devs to try latest firmware and see if it fixes the problem for me as well. 

[HristoGrigorov]

SFWD crashes are now history here. Good job CheckPoint! It took a while to fix it but end result is satisfying. One more step of having more stable SMBs.

[HristoGrigorov]

Surprise!!! Crashed again today. Right after policy install. But I do not think that is the same "region" of code that was causing the crash before. Most impressive was that system load average sky rocketed to ~14 right before it.

I will see if it happens again. Definitely something very specific did it because it is otherwise very stable.