Firewall crashed reboots - vmcore crash file R80.30

[beneaton]

Hi,

A firewall rebooted yesterday evening unmanned.
The firewall is the standby member of an HA cluster so wasn't 'a problem' but we'd like to explore what could have caused it as to prevent it happening again or on the Active firewall.

R80.30 Take 111

/var/log/crash/<DATE>/ = vmcore_zero64 crash file - Appears corrupt and also cannot view in Notepad++ due to size.
/var/crash = Nothing
/var/log/dump/usermode = Nothing
uptime = matches SmartConsole uptime, etc.

/var/log/messages = 
Feb 26 16:57:38 2020 FIREWALL#2 xpand[12527]: admin localhost p +installer:last_update_time Wed\ Feb\ 26\ 16\:57\:35\ 2020
Feb 26 16:57:46 2020 FIREWALL#2 xpand[12527]: admin localhost t -volatile:configurationChange
Feb 26 16:57:46 2020 FIREWALL#2 xpand[12527]: admin localhost t -volatile:configurationSave
Feb 26 17:45:28 2020 FIREWALL#2 xpand[12527]: admin localhost t +installer:last_sent_da_info 1582739128
Feb 26 17:45:28 2020 FIREWALL#2 xpand[12527]: Configuration changed from localhost by user admin by the service dbset
Feb 26 17:45:28 2020 FIREWALL#2 xpand[12527]: os is Gaia
Feb 26 17:45:47 2020 FIREWALL#2 xpand[12527]: admin localhost t -volatile:configurationChange
Feb 26 18:05:07 2020 FIREWALL#2 syslogd 1.4.1: restart.

I've looked into SK123853 which is "security gateway crashs with vmcore"
Solution - Upgrade to R80.10 Take 42 onwards or R80.20. 
As per above, this is on R80.30 so is surpassing that.

(Additional note: I have checked their Revisions and there had been no sessions created since Feb 26 AM)

Any suggestions appreciated.

Thanks

[_Val_]

Please open a TAC case

[beneaton]

Hi Val,

Now opened. Thanks.

[Darren_Fine]

Hi Beneaton,

We had a similar issue a few nights ago - upgraded a single node to R80.30 JHF 1140 from R80.10 . The device rebooted of its own accord twice while we were doing post install tests.

Had to take it offline and have opened a TAC case .

(we have successfully upgraded around 20 other gateways to R80.30 without incident so not sure why this has been an issue)

Will post findings from TAC case in case it assist other checkmaters..

Thanks

[beneaton]

Darren,

Great - Please keep us updated. I've got a TAC case open as well.

Interesting to see the outcome of both!

Cheers,

Ben

[HENRY_SILA]

Any update on this?

[Darren_Fine]

Hi ,

It turned out that there was a hardware fault on the 15600 chassis.

We have since RMA'd it and its working 100%

So guess it was just bad timing that the upgrade showed the issue.

(I think it was something to do with the raid - since it only occured with the reboot after the upgrade- the raid kept trying to create the mirror .)

Regards

[Norman_Zhang]

We also experienced similar problem with R80.30 HFA 221. Where both VRRP members reboot when establishing tunnel with AWS (as soon as the other end point any network route to Check Point), both members crash (primary crashes, fails to secondary, secondary crashes).

vmcore_zero64 generated but nothing useful so far.

TAC still investigating.

[Mircea-PG]

Did they get to the bottom of it in the end?

[Norman_Zhang · Accepted solution]

It was a bug in HFA 221. It was fixed in HFA 226.

https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...

ID:

PRJ-19494,
PRJ-19675

[Vladimir]

While I did not see crashes, the one case where I've seen issues with this setup resulted in this message from TAC:

reply fro "R&D that VTIs and VRRP are not supported together because VRRP can't work with point to point interfaces. "