Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
HristoGrigorov

SFWD process crash

Hi,

has any of you ran into something like this:

[cpWatchDog 2765 1744478736]@RD6281[18 May  9:20:47] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2765 1744478736]@RD6281[18 May  9:21:47] [SUCCESS] SFWD started successfully (pid=3551)
[cpWatchDog 2765 1744478736]@RD6281[18 May  9:35:25] [ERROR] Process SFWD terminated abnormally : Unhandled signal 6 (). Core dumped.
[cpWatchDog 2765 1744478736]@RD6281[18 May  9:36:25] [SUCCESS] SFWD started successfully (pid=4359)
[cpWatchDog 2765 1744478736]@RD6281[20 May  9:41:20] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2765 1744478736]@RD6281[20 May  9:42:20] [SUCCESS] SFWD started successfully (pid=18894)
[cpWatchDog 2765 1744478736]@RD6281[20 May 16:12:09] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2765 1744478736]@RD6281[20 May 16:13:09] [SUCCESS] SFWD started successfully (pid=20929)
[cpWatchDog 2765 1744478736]@RD6281[21 May  8:50:27] [ERROR] Process SFWD terminated abnormally : Unhandled signal 6 (). Core dumped.
[cpWatchDog 2765 1744478736]@RD6281[21 May  8:51:27] [SUCCESS] SFWD started successfully (pid=25168)
[cpWatchDog 2765 1744478736]@RD6281[21 May  9:14:11] [ERROR] Process SFWD terminated abnormally : Unhandled signal 6 (). Core dumped.
[cpWatchDog 2765 1744478736]@RD6281[21 May  9:15:11] [SUCCESS] SFWD started successfully (pid=25918)
[cpWatchDog 2765 1744478736]@RD6281[21 May 10:54:58] [ERROR] Process SFWD terminated abnormally :

Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2765 1744478736]@RD6281[21 May 10:55:58] [SUCCESS] SFWD started successfully (pid=27458)
[cpWatchDog 2765 1744478736]@RD6281[21 May 12:35:14] [ERROR] Process SFWD terminated abnormally : Unhandled signal 6 (). Core dumped.
[cpWatchDog 2765 1744478736]@RD6281[21 May 12:36:14] [SUCCESS] SFWD started successfully (pid=29000)
[cpWatchDog 2765 1744478736]@RD6281[22 May 14:53:05] [ERROR] Process SFWD terminated abnormally : Unhandled signal 6 (). Core dumped.
[cpWatchDog 2765 1744478736]@RD6281[22 May 14:54:05] [SUCCESS] SFWD started successfully (pid=5887)

19 Replies
G_W_Albrecht
Legend
Legend

The old dumping issue - process SFWD needs some time off i think . In the shown timespan it crashes at least twice a day, so i would use USB Medium firmware update to rule out flash issues. This is from /var/log/log/cpwd.elg, what about /var/log/log/sfwd.el* and /var/log/messages ?

CCSE CCTE CCSM SMB Specialist
0 Kudos
HristoGrigorov

Nothing suspicious there. But there is a core dump of the process that I send to R&D for examination.

0 Kudos
G_W_Albrecht
Legend
Legend

Yes, that is the thing to do here - as you can not replicate the crashing sfwd issue.

CCSE CCTE CCSM SMB Specialist
0 Kudos
RS_Daniel
Advisor

Hi, Hristo... do you remember which directory was your core dump in? or the name?... thanks✌️ 

0 Kudos
HristoGrigorov

Nowadays all crashes (core dumps) and panics are placed in /logs directory. 

0 Kudos
PhoneBoy
Admin
Admin

Please open a TAC case so we can investigate.

Contact Support | Check Point Software 

0 Kudos
HristoGrigorov

Yes, sorry I forgot to mention SR is currently under investigation.

0 Kudos
Pedro_Espindola
Advisor

Hello Hristo, how is the investigation going? Did you receive any news yet?

I found out I have 3 appliances in which the issue does not happen and 7 in which it happens. I am opening an SR to try to find out what is special about those 3 and maybe identify what is wrong in the other 7.

HristoGrigorov

Hello Pedro,

Sorry for the late reply, I have somehow missed your question.

I opened SR on 18.09.2018 and since then there was not much progress with it. I have provided core dump to TAC but they said they cannot find anything interesting in it and asked for remote session few days ago. So far, nobody contacted me yet about it. In the mean time, situation is still the same and some times it is getting real bad. I mean just look at this:

[cpWatchDog 2158 1744364432]@RD6281[8 Oct 8:23:37] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 8:24:37] [SUCCESS] SFWD started successfully (pid=11666)
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 9:52:31] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 9:53:31] [SUCCESS] SFWD started successfully (pid=14153)
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 10:23:28] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 10:24:28] [SUCCESS] SFWD started successfully (pid=15008)
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 10:28:48] [ERROR] Process SFWD terminated abnormally : Unhandled signal 6 (). Core dumped.
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 10:29:48] [SUCCESS] SFWD started successfully (pid=15353)
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 10:34:56] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 10:35:56] [SUCCESS] SFWD started successfully (pid=15773)
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 10:51:45] [ERROR] Process SFWD terminated abnormally : Unhandled signal 6 (). Core dumped.
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 10:52:45] [SUCCESS] SFWD started successfully (pid=16393)
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 11:05:51] [ERROR] Process SFWD terminated abnormally : Unhandled signal 6 (). Core dumped.
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 11:06:51] [SUCCESS] SFWD started successfully (pid=16965)
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 11:12:31] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 11:13:31] [SUCCESS] SFWD started successfully (pid=17364)
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 16:42:06] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 16:43:06] [SUCCESS] SFWD started successfully (pid=22701)
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 21:29:02] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 21:30:02] [SUCCESS] SFWD started successfully (pid=23911)

Just prior the crash the system load average will go real high, 3.00 - 5.00 or so. I wonder what is difference between those 3 appliances (where you do not have such issue) and the other 7....

0 Kudos
HristoGrigorov

Btw, do you have Site-to-Site VPNs on those 3 appliances? 

0 Kudos
Pedro_Espindola
Advisor

Yes, I do have 3 tunnels configured in one of the 3 that work, with Azure and AWS.

Did they send you any builds? I opened a SR and they sent me a new build for 700/1400. Number of crashes were drastically reduced. I can now go a whole day without a crash.

Check with them if there is any update.

HristoGrigorov

No, no builds for me yet. Perhaps they want to sort it out for you first and then sent me the same build Smiley Happy 

Do not agree on half-working solution!!! Ask for super-duper-rock-stable build that lasts at least a week (think it is anyway good to reboot these gnomes every weekend).

HristoGrigorov

My SR has just been "escalated to Tier 3" (don't know if that is good or bad) and I am now waiting for TAC to contact me. In the mean time there us a new record on the number of crashes today:

[cpWatchDog 2158 1744515984]@RD6281[11 Oct 12:52:38] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2158 1744515984]@RD6281[11 Oct 12:56:06] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2158 1744515984]@RD6281[11 Oct 13:02:37] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2158 1744515984]@RD6281[11 Oct 13:06:55] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.

0 Kudos
HristoGrigorov

Well, don't want to sound over-optimistically here, but R77.20.81 seems to have solved sfwd crashes for me.  


@Pedro: Have you tried it already? Is it the same for you?

PhoneBoy
Admin
Admin

The feedback is always useful even if not 100% definitive 

Pedro_Espindola
Advisor

Good to know Hristo,

I did not try this one yet, but I am using the build 451 of R77.20.80, which solved it for me. It probably has the same fix shipped in the new version.

I will try R77.20.81 as soon as possible and leave my feedback here.

0 Kudos
HristoGrigorov

Yeah, probably it is just like that. I was just asked by the devs to try latest firmware and see if it fixes the problem for me as well. 

0 Kudos
HristoGrigorov

SFWD crashes are now history here. Good job CheckPoint! It took a while to fix it but end result is satisfying. One more step of having more stable SMBs.

HristoGrigorov

Surprise!!! Crashed again today. Right after policy install. But I do not think that is the same "region" of code that was causing the crash before. Most impressive was that system load average sky rocketed to ~14 right before it.

I will see if it happens again. Definitely something very specific did it because it is otherwise very stable.

0 Kudos

Leaderboard

Epsum factorial non deposit quid pro quo hic escorol.

Upcoming Events

    CheckMates Events