topic sk181803 - Possible failure to boot on a Security Gateway with IPS, Anti-Bot, or Application Control in Firewall and Security Management

sk181803 - Possible failure to boot on a Security Gateway with IPS, Anti-Bot, or Application Control

Thomas_Eichelbu — Mon, 18 Dec 2023 16:57:10 GMT

Hello folks,

did someone already stumble over this SK ?

sk181803 - Possible failure to boot on a Security Gateway with IPS, Anti-Bot, or Application Control enabled

has anybody seen this problem in the wild already?
what does the community think? should we update all our customer to R81.10 Take 130 immediately?

what does this mean?

Check Point R81.20 and higher (what is R81.20 and higher? So no fix for R81.20 at the moment?)
Jumbo Hotfix Accumulator for R81.10 starting from Take 130
Check Point Quantum Spark R80.20.40 and higher

i hope this rare scenarios really strike very seldom, otherwise we will have alot of work to do!

best regards

Re: sk181803 - Possible failure to boot on a Security Gateway with IPS, Anti-Bot, or Application Con

the_rock — Mon, 18 Dec 2023 17:00:41 GMT

That seems like a pretty serious issue, thanks for posting about it.

Andy

Re: sk181803 - Possible failure to boot on a Security Gateway with IPS, Anti-Bot, or Application Con

CaseyB — Mon, 18 Dec 2023 17:11:14 GMT

I saw this included in the R81.10 Take 130 notes. Looks very serious to me, looking at getting all of our gateways updated today, it seems multiple events could trigger this issue.

Re: sk181803 - Possible failure to boot on a Security Gateway with IPS, Anti-Bot, or Application Con

the_rock — Mon, 18 Dec 2023 17:13:07 GMT

Agree, its definitely serious and as you said, multiple things could cause it.

Andy

Re: sk181803 - Possible failure to boot on a Security Gateway with IPS, Anti-Bot, or Application Con

Alex- — Mon, 18 Dec 2023 18:13:40 GMT

The versions you list are the ones not affected so R81.20 is supposedly safe from this issue.

R81 and R80.40 still have support well into 2024 so let's hope a fix for these versions is also in preparation.

Re: sk181803 - Possible failure to boot on a Security Gateway with IPS, Anti-Bot, or Application Con

the_rock — Mon, 18 Dec 2023 18:15:00 GMT

I sure hope so as well...

Andy

Re: sk181803 - Possible failure to boot on a Security Gateway with IPS, Anti-Bot, or Application Con

MatanYanay — Tue, 19 Dec 2023 05:42:01 GMT

Hi @Alex and All

We do plan to release jumbo for R80.40 and R81 next week or the week after with this fix

Thanks

Matan.

Re: sk181803 - Possible failure to boot on a Security Gateway with IPS, Anti-Bot, or Application Con

Thomas_Eichelbu — Tue, 19 Dec 2023 06:21:25 GMT

The question is, with what probability this issue will most likely strike?

one out of ten gateways?
one out of hundreds?
or more or less?

Since what JHF or signature update this issue has started?
For how long this issue exists?
My customers run several hundreds gateways in total, we never lost a gateway to an issue like that.
Our main reason when file systems get destroyed are power outages on ext3 filesystems.

best regards

Re: sk181803 - Possible failure to boot on a Security Gateway with IPS, Anti-Bot, or Application Con

Gera_Dorfman — Tue, 19 Dec 2023 07:37:27 GMT

The issue has been present in our code since R80.20, and is there for the last five years in all versions between R80.20 - R81.10. (It doesn't exist in R81.20).

The issue is a rare memory corruption leading to Gaia OS corruption.

During this period, hundreds of thousands of gateways have been successfully installed, upgraded, and updated, with no reported impact from this specific problem.

It was only after a recent report from a customer that we became aware of the issue.

In response, we have prepared fixes for all affected versions.

Re: sk181803 - Possible failure to boot on a Security Gateway with IPS, Anti-Bot, or Application Con

Thomas_Eichelbu — Tue, 19 Dec 2023 07:39:53 GMT

Hello Gera,

thank you for this information.
so it means the issue is very severe if it strikes, but the probability of getting hit is very rare.
good to know!

thank you

best regards

Re: sk181803 - Possible failure to boot on a Security Gateway with IPS, Anti-Bot, or Application Con

dehaasm — Tue, 19 Dec 2023 09:24:23 GMT

Hi,

As this is a very rare issue I believe the exact root cause is unknown, what does the fix contain exactly, how does it prevent the memory corruption to occur during policy install/AV-IPS updates could you share some details?

Re: sk181803 - Possible failure to boot on a Security Gateway with IPS, Anti-Bot, or Application Con

Gera_Dorfman — Tue, 19 Dec 2023 10:31:18 GMT

The root cause is known. We were able to trace and to fix this rare issue.

Thanks

Re: sk181803 - Possible failure to boot on a Security Gateway with IPS, Anti-Bot, or Application Con

the_rock — Tue, 19 Dec 2023 13:12:37 GMT

Thats great news @Gera_Dorfman

Thanks for confirming.

Andy

Re: sk181803 - Possible failure to boot on a Security Gateway with IPS, Anti-Bot, or Application Con

Harald_Hansen — Tue, 19 Dec 2023 13:36:27 GMT

I'd suggest putting that into the SK, to remove confusion and unnecessary urgency on your customers behalf.

One could guess this has a very rare rate, though it is nice to have it officially.

Re: sk181803 - Possible failure to boot on a Security Gateway with IPS, Anti-Bot, or Application Con

the_rock — Tue, 19 Dec 2023 13:56:10 GMT

That totally makes sense @Harald_Hansen . Otherwise, people may start panicking even if they are on say R81.10 version, but it sounds like its mostly under control, unless you are on R80.30 or below.

Andy

Re: sk181803 - Possible failure to boot on a Security Gateway with IPS, Anti-Bot, or Application Con

D_W — Tue, 19 Dec 2023 14:52:52 GMT

So it is recommended to install the not yet recommended JHF R81.10 T130 ?

Re: sk181803 - Possible failure to boot on a Security Gateway with IPS, Anti-Bot, or Application Con

the_rock — Tue, 19 Dec 2023 15:10:15 GMT

One of my colleagues opened TAC case yesterday asking that exact same question and answer was more less its client's decision when it comes to installing new jumbo hotfix. But, I get sk is brand new, so hard to provide valid answer right away if there are lots of unknowns...

Andy

Re: sk181803 - Possible failure to boot on a Security Gateway with IPS, Anti-Bot, or Application Con

D_W — Tue, 19 Dec 2023 14:57:53 GMT

Once we had a TAC Case and they asked why we installed the not yet recommended JHF 🙄

Re: sk181803 - Possible failure to boot on a Security Gateway with IPS, Anti-Bot, or Application Con

Gera_Dorfman — Tue, 19 Dec 2023 15:01:58 GMT

We will adjust the SK and will mention that it's a rare case. Thank You

Re: sk181803 - Possible failure to boot on a Security Gateway with IPS, Anti-Bot, or Application Con

the_rock — Tue, 19 Dec 2023 15:04:28 GMT

Personally, I wish there was an sk stating exactly that recommended take should be installed and ongoing ONLY when there is known vulnerability, so that way, it would clear any confusion. Otherwise, everyone "digests" it their own way...but then again, I think most people know by now that recommended take is the way to go.

Andy