topic Re: Firewall crashed reboots - vmcore crash file R80.30 in General Topics

Firewall crashed reboots - vmcore crash file R80.30

beneaton — Thu, 27 Feb 2020 11:22:33 GMT

Hi,

A firewall rebooted yesterday evening unmanned.
The firewall is the standby member of an HA cluster so wasn't 'a problem' but we'd like to explore what could have caused it as to prevent it happening again or on the Active firewall.

R80.30 Take 111

/var/log/crash/<DATE>/ = vmcore_zero64 crash file - Appears corrupt and also cannot view in Notepad++ due to size.
/var/crash = Nothing
/var/log/dump/usermode = Nothing
uptime = matches SmartConsole uptime, etc.

/var/log/messages =
Feb 26 16:57:38 2020 FIREWALL#2 xpand[12527]: admin localhost p +installer:last_update_time Wed\ Feb\ 26\ 16\:57\:35\ 2020
Feb 26 16:57:46 2020 FIREWALL#2 xpand[12527]: admin localhost t -volatile:configurationChange
Feb 26 16:57:46 2020 FIREWALL#2 xpand[12527]: admin localhost t -volatile:configurationSave
Feb 26 17:45:28 2020 FIREWALL#2 xpand[12527]: admin localhost t +installer:last_sent_da_info 1582739128
Feb 26 17:45:28 2020 FIREWALL#2 xpand[12527]: Configuration changed from localhost by user admin by the service dbset
Feb 26 17:45:28 2020 FIREWALL#2 xpand[12527]: os is Gaia
Feb 26 17:45:47 2020 FIREWALL#2 xpand[12527]: admin localhost t -volatile:configurationChange
Feb 26 18:05:07 2020 FIREWALL#2 syslogd 1.4.1: restart.

I've looked into SK123853 which is "security gateway crashs with vmcore"
Solution - Upgrade to R80.10 Take 42 onwards or R80.20.
As per above, this is on R80.30 so is surpassing that.

(Additional note: I have checked their Revisions and there had been no sessions created since Feb 26 AM)

Any suggestions appreciated.

Thanks

Re: Firewall crashed reboots - vmcore crash file R80.30

_Val_ — Thu, 27 Feb 2020 11:55:11 GMT

Please open a TAC case

Re: Firewall crashed reboots - vmcore crash file R80.30

beneaton — Thu, 27 Feb 2020 12:19:22 GMT

Hi Val,

Now opened. Thanks.

Re: Firewall crashed reboots - vmcore crash file R80.30

Darren_Fine — Fri, 28 Feb 2020 08:36:12 GMT

Hi Beneaton,

We had a similar issue a few nights ago - upgraded a single node to R80.30 JHF 1140 from R80.10 . The device rebooted of its own accord twice while we were doing post install tests.

Had to take it offline and have opened a TAC case .

(we have successfully upgraded around 20 other gateways to R80.30 without incident so not sure why this has been an issue)

Will post findings from TAC case in case it assist other checkmaters..

Thanks

Re: Firewall crashed reboots - vmcore crash file R80.30

beneaton — Fri, 28 Feb 2020 10:56:18 GMT

Darren,

Great - Please keep us updated. I've got a TAC case open as well.

Interesting to see the outcome of both!

Cheers,

Ben

Re: Firewall crashed reboots - vmcore crash file R80.30

HENRY_SILA — Tue, 10 Mar 2020 11:04:17 GMT

Any update on this?

Re: Firewall crashed reboots - vmcore crash file R80.30

Darren_Fine — Tue, 21 Apr 2020 20:58:59 GMT

Hi ,

It turned out that there was a hardware fault on the 15600 chassis.

We have since RMA'd it and its working 100%

So guess it was just bad timing that the upgrade showed the issue.

(I think it was something to do with the raid - since it only occured with the reboot after the upgrade- the raid kept trying to create the mirror .)

Regards

Re: Firewall crashed reboots - vmcore crash file R80.30

Norman_Zhang — Wed, 04 Nov 2020 15:02:00 GMT

We also experienced similar problem with R80.30 HFA 221. Where both VRRP members reboot when establishing tunnel with AWS (as soon as the other end point any network route to Check Point), both members crash (primary crashes, fails to secondary, secondary crashes).

vmcore_zero64 generated but nothing useful so far.

TAC still investigating.

Re: Firewall crashed reboots - vmcore crash file R80.30

Mircea-PG — Mon, 01 Mar 2021 16:24:47 GMT

Did they get to the bottom of it in the end?

Re: Firewall crashed reboots - vmcore crash file R80.30

Vladimir — Mon, 01 Mar 2021 18:21:02 GMT

While I did not see crashes, the one case where I've seen issues with this setup resulted in this message from TAC:

reply fro "R&D that VTIs and VRRP are not supported together because VRRP can't work with point to point interfaces. "

Re: Firewall crashed reboots - vmcore crash file R80.30

Norman_Zhang — Mon, 01 Mar 2021 18:43:41 GMT

It was a bug in HFA 221. It was fixed in HFA 226.

https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk153152

ID:

PRJ-19494,
PRJ-19675