topic Re: Question regarding RMA a firewall that diagnostic tool show all OK in Firewall and Security Management

Question regarding RMA a firewall that diagnostic tool show all OK

LeeBingKang — Wed, 02 Aug 2023 02:46:11 GMT

Hi all,

Recently, I had a client faced an issue whereby one of their FULL HA firewalls suddenly hangs and this issue happened more than 2 times within few months.

The 1st hang is on around 30 May 2023 and the firewall is working fine as usual after reboot. Meanwhile, we do hardware diagnostic via command "diagMain" and found out the diagnostic result show OK. With that, we opened a case with TAC on this and TAC suggested to install the latest recommended Jumbo Hotfix (take 197) as it resolves some memnory related issues. After that, we installed the Jumbo Hotfix take 197 on both FULL HA firewall member successfully on 22 July 2023.

The 2nd hang is on 1/8/2023 and this hang happened on the same firewall when it acts as active firewall and management (since 22 July 2023). The firewall is booted up after a reboot and the diagnostic result (via command diagMain) also show all OK. However, we monitor it more than 30 minutes and aware that its CPU utilization is inconsistent (somehow will reach more than 100% for Java process). Moreover, we tried to move the active management

Hence, I would like seek all of your advice whereby is this hang happened more than 2 times within these few months can become the reason to do RMA?

Thank you.

Re: Question regarding RMA a firewall that diagnostic tool show all OK

the_rock — Wed, 02 Aug 2023 03:02:17 GMT

This certainly warrants TAC case for further investigation.

Andy

Re: Question regarding RMA a firewall that diagnostic tool show all OK

LeeBingKang — Wed, 02 Aug 2023 03:08:49 GMT

If you said so, I will open a case with TAC case for further investigation and hopefully can get RMA for that unit.

Re: Question regarding RMA a firewall that diagnostic tool show all OK

the_rock — Wed, 02 Aug 2023 09:13:56 GMT

Well, its simply my logical suggestion, based on all the details you had given.

Regards,

Andy

Re: Question regarding RMA a firewall that diagnostic tool show all OK

LeeBingKang — Wed, 02 Aug 2023 09:32:53 GMT

Yea @the_rock, I understand that it is your suggestion and I'm sorry on my wording in previous post.

So far, I already open a case with TAC on this and one thing that out of my expectation is that the problematic firewall's CPU came back to normal.

However, it is not a good things as we also hard to identify the cause of the hang issue.

Re: Question regarding RMA a firewall that diagnostic tool show all OK

Chris_Atkinson — Wed, 02 Aug 2023 10:07:58 GMT

Which management blades are enabled here?

Also is there a reason that Management isn't deployed separately for this cluster?

Re: Question regarding RMA a firewall that diagnostic tool show all OK

the_rock — Wed, 02 Aug 2023 15:52:02 GMT

Maybe have a look at below...TAC gave this to one of my colleagues yesterday for somewhat similar issue.

Andy

Something to try anyway...

https://support.checkpoint.com/results/sk/sk180437

Re: Question regarding RMA a firewall that diagnostic tool show all OK

LeeBingKang — Thu, 03 Aug 2023 01:02:46 GMT

Thanks about that and i try to look into it.

Re: Question regarding RMA a firewall that diagnostic tool show all OK

LeeBingKang — Thu, 03 Aug 2023 01:08:01 GMT

Hi Chris, the managment server part just enable normal management server blade, and no other blades enable (e.g. smartevent, endpoint management server).

This FULL HA setup is being built for long time ago and doing distributed will cause customer need to bare another security managemen server license, although I understand that distributed mode is recommended.

Re: Question regarding RMA a firewall that diagnostic tool show all OK

the_rock — Thu, 03 Aug 2023 01:25:22 GMT

SK actually tells you what to look for, so if those errors match, its definitely related.

Andy

Re: Question regarding RMA a firewall that diagnostic tool show all OK

the_rock — Thu, 03 Aug 2023 01:45:43 GMT

Forgot to mention, TAC, Im fairly sure of this, will ask you to run hardware diag tool, so be prepared for that.

Andy

https://support.checkpoint.com/results/sk/sk97251

Re: Question regarding RMA a firewall that diagnostic tool show all OK

LeeBingKang — Thu, 03 Aug 2023 05:46:50 GMT

I ran the hardware diagnostics during that moment (after reboot firewall when the hang issue happens) and the result shows all component are OK which is out of my expectation.

Meanwhile, do you have any idea if the disk test in that hardware diagnostics able to test on SSD as that firewall is using SSD.

Re: Question regarding RMA a firewall that diagnostic tool show all OK

Lesley — Thu, 03 Aug 2023 09:02:57 GMT

I assume you are running R80.40? This version will be EOL at the start of 2024.

Are you sure it is hardware related? It still can be software that causes the crash.

I would recommend to update the firewalls anyway, R81.20 is now recommended for installation.

Second feedback is that the active gateway is running the fw part and mgmt part.

Why not move the mgmt part over to the second backup firewall?

This could help with the load of the system.

Re: Question regarding RMA a firewall that diagnostic tool show all OK

LeeBingKang — Thu, 03 Aug 2023 09:22:40 GMT

Hi Lesley. Thanks for your feedback on this and I have read through your suggestion.

We can't simply help customer upgrade to R81.20 due to some certain reasons.

I understand that sometime software will cause crash, but I have no idea to find out that software causes it.

Yes, the firewall member 1 have the active on management and firewall.

Last but not least, there is another weird things which I not really think that it causes this issue whereby the hang issue happened few hours after we have done the remote backup and schedule backup setting.

Re: Question regarding RMA a firewall that diagnostic tool show all OK

the_rock — Thu, 03 Aug 2023 11:57:15 GMT

About SSD, not 100% certain, you may need to verify with TAC. Unless it shows it clearly in hardware diag tool results.

Andy

Re: Question regarding RMA a firewall that diagnostic tool show all OK

LeeBingKang — Fri, 04 Aug 2023 00:48:40 GMT

Thanks for your reply. I will ask TAC about this.

Re: Question regarding RMA a firewall that diagnostic tool show all OK

_Val_ — Fri, 04 Aug 2023 07:52:28 GMT

This does not sound like a HW issue but like a performance issue. I would advise you to investigate that further.

Re: Question regarding RMA a firewall that diagnostic tool show all OK

LeeBingKang — Mon, 07 Aug 2023 05:22:31 GMT

Hi @_Val_ , I'm working with TAC on this.

Re: Question regarding RMA a firewall that diagnostic tool show all OK

LeeBingKang — Mon, 07 Aug 2023 08:47:15 GMT

Hi all, it is unfortunate that TAC didn't find anything related to the issue from those files given (CPU spike detective, HCP report, and cpinfo file).

Hence, need to monitor and wait until the issue is happening again in future.

Re: Question regarding RMA a firewall that diagnostic tool show all OK

the_rock — Mon, 07 Aug 2023 13:31:17 GMT

I would definitely consider SK I gave you, as we had similar issue with customer recently.