topic Re: Segmentation fault in Firewall and Security Management

Segmentation fault

Vincent_Bacher — Mon, 29 Jan 2018 16:32:10 GMT

Hello mates,

after upgrading several gateways to R80.10 T56 a customer always gets segmentation fault when perperforming "show configuration" command in clish.

I know that there are several sk regarding R77 but not R80.10.

Anybody who knows anything?

Best regards

Vincent

Re: Segmentation fault

KennyManrique — Mon, 29 Jan 2018 19:31:51 GMT

Hello Vincent,

Recently I had a similar problem on a 5200 Appliance upgraded to R80.10 T56 from R77.30.

According to sk106166 is a database corruption (tough is for R77.30 version). The problem jumped when I verified the status from central managment, connection was lost altough SIC status was communicating (about 18 hours after upgrade).

Meanwhile, I was able to connect to SSH, but I couldnt perform any "fw" operations, even some troubleshooting commands like tcpdump or netstat weren't able to execute; resulting in a core dump. From core dumps, it seems two processes crashed the gateway initially: in.msd and esc_db_complete.

I rebooted the gateway and used HW Diagnostic tools embeded on the device for test the hard drive. A second reboot was performed and entered to maintenance mode to execute a disk check using fsck based on sk92442.

After this, Appliance work as expected (8 days without the error by now)

Regards.

Re: Segmentation fault

Vincent_Bacher — Tue, 30 Jan 2018 12:46:18 GMT

Hello Kenny,

thank you for sharing your experience regarding this issue.

Well, this led me to create a sr at usercenter

We'll see.

best regards

Vincent

Re: Segmentation fault

Vincent_Bacher — Tue, 30 Jan 2018 16:59:36 GMT

Hello mates,

just received update from CP that this is a known issue and resolved in T70.

Will test that and keep you posted.

Best regards

Vincent

Re: Segmentation fault

KennyManrique — Tue, 30 Jan 2018 20:28:21 GMT

Great to hear that Vincent!

On R80.10 HFA document, does not mention a similar issue to the experienced on HFA 70. Does CP team tell you specifically which ID or problem is solved?

Regards.

Re: Segmentation fault

Vincent_Bacher — Tue, 30 Jan 2018 20:35:39 GMT

Hi Kenny,

Unfortunately not.

Just asked and keep you posted

Best regards

Vincent

Re: Segmentation fault

Vincent_Bacher — Wed, 31 Jan 2018 13:31:17 GMT

Just installed T70 on a test node and still facing this issue.

And no answer from case owner regarding bug ID or similar helpful info.

I'll wait a bit and push the escalate button if needed

Re: Segmentation fault

KennyManrique — Wed, 31 Jan 2018 13:53:42 GMT

Thanks for the update Vincent.

I was analizing the data of my experienced crash and I find out that all the strange crashes belong to inactive functionalities on the Gateway (AntiSpam and HTTPS Inspection).

Do you have a similiar issue?

Regards.

Re: Segmentation fault

Vincent_Bacher — Wed, 31 Jan 2018 14:15:14 GMT

Hm, i did not check if i have a similar issue.

But my co-workers nost notified me about a different issue.

We are now facing the error that my colleagues have added a vlan Interface and the virtual Interface is not shown in cphaprob -a if

added eth1.30 and we dont see the virtual if

# fw getifs
localhost eth1 ******
localhost eth5 *****
localhost eth1.2001 *****
localhost eth1.30 ******
localhost eth1.20 ******
localhost eth1.10 ******
#cphaprob -a if
Required interfaces: 3
Required secured interfaces: 1
eth5       UP                    sync(secured), multicast
eth1       UP                    non sync(non secured), multicast (eth1.2001 )
eth1       UP                    non sync(non secured), multicast (eth1.10   )
Virtual cluster interfaces: 4
eth1            ****
eth1.2001       *****
eth1.20         *****
eth1.10         *****

Re: Segmentation fault

Libin_Thomas — Thu, 15 Feb 2018 13:52:40 GMT

we are also facing the same issue , Intially it was working fine for couple days after that primary firewall got disconnected

from the cluster and in the cli if we perform any command it shows segmentation error.

TAC case also opened for the same but still engineers not able to find the root cause.

Re: Segmentation fault

VENKAT_S_P — Sat, 17 Feb 2018 01:11:59 GMT

Did you tried degrade?

Feeling lucky for not installing T56 after seeing this thread.

I have T42 installed and do not have any issues.

Re: Segmentation fault

KennyManrique — Sat, 17 Feb 2018 13:40:33 GMT

Hello Vincent,

Do you have any news about your issue?

Regards.

Re: Segmentation fault

Vincent_Bacher — Sat, 17 Feb 2018 16:42:34 GMT

Not yet. CP is waiting for a cpinfo.

Will create one on Monday. Will keep you posted.

Vincent

Re: Segmentation fault

Vincent_Bacher — Sat, 17 Feb 2018 16:44:40 GMT

Not yet. Will work bit more on this open sr

Cheers

Re: Segmentation fault

Vincent_Bacher — Fri, 23 Feb 2018 14:48:29 GMT

Now i have News.

They told me that "other customer had this issue also had Contact Awareness blade enabled that cause this issue."

They suggest disabling "Contact awareness blade and monitor the FW's to see if the issue persist"

Disbalbing IA balade? Funny!

And they want me to seht
[Expert@:0]# fw ctl set int panic_on_stk_size 300
This kernel parameter, will cause the device to crash once the issue re-occurs, and will provide us with a vmcore file

I asked the customer to decide if they want to do that

best
Vincent

Re: Segmentation fault

Hugo_Frauches — Tue, 27 Mar 2018 14:11:25 GMT

Hello guys,

I have one Cluster of 5200 with R80.10 installed with the latest Hotfix (take 70) and we are having the same problem, sometimes one of the Gateways stops and when i tried to use an FW cmd via clish the output its always the same:

/bin/cpfw_start: line12: 21612 Segmentation fault (core duumped) $FWDIR/bin /fw "$@"

I have an open service request on Checkpoint but they didnt found the solution yet, have anyone found what its causing this problem? Maybe its an bug with R80.10?

Re: Segmentation fault

KennyManrique — Tue, 27 Mar 2018 14:30:20 GMT

Hi Hugo.

Concidentally today, one of my customers had the same Segmentation Fault crash (again) on 5200 Appliance with R80.10 HFA 70.

I have a case opened since last week and like Vincent said, apparently HFA 70 was the solution (also according to sk123153 and sk123154)...but it seems not to be. I'm waiting for TAC response, I will update with the news.

Regards.

Re: Segmentation fault

Vincent_Bacher — Tue, 27 Mar 2018 14:46:42 GMT

Hi Hugo, Kenny

I think, the issue, Hugo is facing, is quiet different than mine. He mentioned that a core has been dumped and in my case there is no user/core dump at all.
In my case, R&D is actually working on the issue and they told me that they are just testing a possible solution.
Hopefully i'll get some news soon.

Few minutes ago i just tested this at a gateway running T85 and i get no segmentation fault.
But i wait for update from R&D to see if this is generally solved on T85 or just due to different config.

Re: Segmentation fault

Hugo_Frauches — Tue, 27 Mar 2018 15:41:29 GMT

Hello guys,

This problem its awful, my service request opened since last month and the support told me to do and "Fresh Install" on both 5200 Gateways, after i did this it take only 5 days to occur again. When its happens, my Gateway stays "Freezed" and after some time the node status shows as Down, and the only thing i can do to recover the member its to reboot the Gateway, after that the problem is removed (For a short time)...

Re: Segmentation fault

Vincent_Bacher — Tue, 03 Apr 2018 15:42:04 GMT

Well, my special issue is solved.

Solution was

# dbset snmp mode default

# dbset save

But I think that's just a solution for my special issue and all mentioned issues here are different.

Cheers

Vince