- CheckMates
- :
- Products
- :
- Quantum
- :
- SMB Gateways (Spark)
- :
- Re: SFWD process crash
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Are you a member of CheckMates?
×- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
SFWD process crash
Hi,
has any of you ran into something like this:
[cpWatchDog 2765 1744478736]@RD6281[18 May 9:20:47] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2765 1744478736]@RD6281[18 May 9:21:47] [SUCCESS] SFWD started successfully (pid=3551)
[cpWatchDog 2765 1744478736]@RD6281[18 May 9:35:25] [ERROR] Process SFWD terminated abnormally : Unhandled signal 6 (). Core dumped.
[cpWatchDog 2765 1744478736]@RD6281[18 May 9:36:25] [SUCCESS] SFWD started successfully (pid=4359)
[cpWatchDog 2765 1744478736]@RD6281[20 May 9:41:20] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2765 1744478736]@RD6281[20 May 9:42:20] [SUCCESS] SFWD started successfully (pid=18894)
[cpWatchDog 2765 1744478736]@RD6281[20 May 16:12:09] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2765 1744478736]@RD6281[20 May 16:13:09] [SUCCESS] SFWD started successfully (pid=20929)
[cpWatchDog 2765 1744478736]@RD6281[21 May 8:50:27] [ERROR] Process SFWD terminated abnormally : Unhandled signal 6 (). Core dumped.
[cpWatchDog 2765 1744478736]@RD6281[21 May 8:51:27] [SUCCESS] SFWD started successfully (pid=25168)
[cpWatchDog 2765 1744478736]@RD6281[21 May 9:14:11] [ERROR] Process SFWD terminated abnormally : Unhandled signal 6 (). Core dumped.
[cpWatchDog 2765 1744478736]@RD6281[21 May 9:15:11] [SUCCESS] SFWD started successfully (pid=25918)
[cpWatchDog 2765 1744478736]@RD6281[21 May 10:54:58] [ERROR] Process SFWD terminated abnormally :
Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2765 1744478736]@RD6281[21 May 10:55:58] [SUCCESS] SFWD started successfully (pid=27458)
[cpWatchDog 2765 1744478736]@RD6281[21 May 12:35:14] [ERROR] Process SFWD terminated abnormally : Unhandled signal 6 (). Core dumped.
[cpWatchDog 2765 1744478736]@RD6281[21 May 12:36:14] [SUCCESS] SFWD started successfully (pid=29000)
[cpWatchDog 2765 1744478736]@RD6281[22 May 14:53:05] [ERROR] Process SFWD terminated abnormally : Unhandled signal 6 (). Core dumped.
[cpWatchDog 2765 1744478736]@RD6281[22 May 14:54:05] [SUCCESS] SFWD started successfully (pid=5887)
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The old dumping issue - process SFWD needs some time off i think . In the shown timespan it crashes at least twice a day, so i would use USB Medium firmware update to rule out flash issues. This is from /var/log/log/cpwd.elg, what about /var/log/log/sfwd.el* and /var/log/messages ?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Nothing suspicious there. But there is a core dump of the process that I send to R&D for examination.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Yes, that is the thing to do here - as you can not replicate the crashing sfwd issue.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi, Hristo... do you remember which directory was your core dump in? or the name?... thanks✌️
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Nowadays all crashes (core dumps) and panics are placed in /logs directory.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Please open a TAC case so we can investigate.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Yes, sorry I forgot to mention SR is currently under investigation.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello Hristo, how is the investigation going? Did you receive any news yet?
I found out I have 3 appliances in which the issue does not happen and 7 in which it happens. I am opening an SR to try to find out what is special about those 3 and maybe identify what is wrong in the other 7.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello Pedro,
Sorry for the late reply, I have somehow missed your question.
I opened SR on 18.09.2018 and since then there was not much progress with it. I have provided core dump to TAC but they said they cannot find anything interesting in it and asked for remote session few days ago. So far, nobody contacted me yet about it. In the mean time, situation is still the same and some times it is getting real bad. I mean just look at this:
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 8:23:37] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 8:24:37] [SUCCESS] SFWD started successfully (pid=11666)
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 9:52:31] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 9:53:31] [SUCCESS] SFWD started successfully (pid=14153)
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 10:23:28] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 10:24:28] [SUCCESS] SFWD started successfully (pid=15008)
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 10:28:48] [ERROR] Process SFWD terminated abnormally : Unhandled signal 6 (). Core dumped.
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 10:29:48] [SUCCESS] SFWD started successfully (pid=15353)
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 10:34:56] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 10:35:56] [SUCCESS] SFWD started successfully (pid=15773)
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 10:51:45] [ERROR] Process SFWD terminated abnormally : Unhandled signal 6 (). Core dumped.
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 10:52:45] [SUCCESS] SFWD started successfully (pid=16393)
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 11:05:51] [ERROR] Process SFWD terminated abnormally : Unhandled signal 6 (). Core dumped.
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 11:06:51] [SUCCESS] SFWD started successfully (pid=16965)
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 11:12:31] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 11:13:31] [SUCCESS] SFWD started successfully (pid=17364)
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 16:42:06] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 16:43:06] [SUCCESS] SFWD started successfully (pid=22701)
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 21:29:02] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2158 1744364432]@RD6281[8 Oct 21:30:02] [SUCCESS] SFWD started successfully (pid=23911)
Just prior the crash the system load average will go real high, 3.00 - 5.00 or so. I wonder what is difference between those 3 appliances (where you do not have such issue) and the other 7....
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Btw, do you have Site-to-Site VPNs on those 3 appliances?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Yes, I do have 3 tunnels configured in one of the 3 that work, with Azure and AWS.
Did they send you any builds? I opened a SR and they sent me a new build for 700/1400. Number of crashes were drastically reduced. I can now go a whole day without a crash.
Check with them if there is any update.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
No, no builds for me yet. Perhaps they want to sort it out for you first and then sent me the same build
Do not agree on half-working solution!!! Ask for super-duper-rock-stable build that lasts at least a week (think it is anyway good to reboot these gnomes every weekend).
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
My SR has just been "escalated to Tier 3" (don't know if that is good or bad) and I am now waiting for TAC to contact me. In the mean time there us a new record on the number of crashes today:
[cpWatchDog 2158 1744515984]@RD6281[11 Oct 12:52:38] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2158 1744515984]@RD6281[11 Oct 12:56:06] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2158 1744515984]@RD6281[11 Oct 13:02:37] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
[cpWatchDog 2158 1744515984]@RD6281[11 Oct 13:06:55] [ERROR] Process SFWD terminated abnormally : Unhandled signal 11 (SIGSEGV). Core dumped.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Well, don't want to sound over-optimistically here, but R77.20.81 seems to have solved sfwd crashes for me.
@Pedro: Have you tried it already? Is it the same for you?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The feedback is always useful even if not 100% definitive
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Good to know Hristo,
I did not try this one yet, but I am using the build 451 of R77.20.80, which solved it for me. It probably has the same fix shipped in the new version.
I will try R77.20.81 as soon as possible and leave my feedback here.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Yeah, probably it is just like that. I was just asked by the devs to try latest firmware and see if it fixes the problem for me as well.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
SFWD crashes are now history here. Good job CheckPoint! It took a while to fix it but end result is satisfying. One more step of having more stable SMBs.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Surprise!!! Crashed again today. Right after policy install. But I do not think that is the same "region" of code that was causing the crash before. Most impressive was that system load average sky rocketed to ~14 right before it.
I will see if it happens again. Definitely something very specific did it because it is otherwise very stable.
