sk181803 - Possible failure to boot on a Security Gateway with IPS, Anti-Bot, or Application Control

[Thomas_Eichelbu]

Hello folks, 

did someone already stumble over this SK ?

sk181803 - Possible failure to boot on a Security Gateway with IPS, Anti-Bot, or Application Control enabled

has anybody seen this problem in the wild already?
what does the community think? should we update all our customer to R81.10 Take 130 immediately?

what does this mean?

• Check Point R81.20 and higher (what is R81.20 and higher? So no fix for R81.20 at the moment?)
• Jumbo Hotfix Accumulator for R81.10 starting from Take 130
• Check Point Quantum Spark R80.20.40 and higher

i hope this rare scenarios really strike very seldom, otherwise we will have alot of work to do!

best regards

[the_rock]

That seems like a pretty serious issue, thanks for posting about it.

Andy

[CaseyB]

I saw this included in the R81.10 Take 130 notes. Looks very serious to me, looking at getting all of our gateways updated today, it seems multiple events could trigger this issue. 

[the_rock]

Agree, its definitely serious and as you said, multiple things could cause it.

Andy

[Alex-]

The versions you list are the ones not affected so R81.20 is supposedly safe from this issue.

R81 and R80.40 still have support well into 2024 so let's hope a fix for these versions is also in preparation.

[the_rock]

I sure hope so as well...

Andy

[MatanYanay]

Hi @Alex and All

We do plan to release jumbo for R80.40 and R81 next week or the week after with this fix 

Thanks 

Matan.

[Thomas_Eichelbu]

The question is, with what probability this issue will most likely strike?

one out of ten gateways?
one out of hundreds?
or more or less?

Since what JHF or signature update this issue has started?
For how long this issue exists? 
My customers run several hundreds gateways in total, we never lost a gateway to an issue like that.
Our main reason when file systems get destroyed are power outages on ext3 filesystems.

best regards

[Gera_Dorfman]

The issue has been present in our code since R80.20, and is there for the last five years in all versions between R80.20 - R81.10. (It doesn't exist in R81.20).

The issue is a rare memory corruption leading to Gaia OS corruption.

During this period, hundreds of thousands of gateways have been successfully installed, upgraded, and updated, with no reported impact from this specific problem.

It was only after a recent report from a customer that we became aware of the issue. 

In response, we have prepared fixes for all affected versions.

[Thomas_Eichelbu]

Hello Gera, 

thank you for this information.
so it means the issue is very severe if it strikes, but the probability of getting hit is very rare.
good to know!

thank you 


best regards

[dehaasm]

Hi,

As this is a very rare issue I believe the exact root cause is unknown, what does the fix contain exactly, how does it prevent the memory corruption to occur during policy install/AV-IPS updates could you share some details?

[Gera_Dorfman]

Hi

The root cause is known. We were able to trace and to fix this rare issue.

Thanks

[Harald_Hansen]

I'd suggest putting that into the SK, to remove confusion and unnecessary urgency on your customers behalf.

One could guess this has a very rare rate, though it is nice to have it officially. 

[the_rock]

That totally makes sense @Harald_Hansen . Otherwise, people may start panicking even if they are on say R81.10 version, but it sounds like its mostly under control, unless you are on R80.30 or below.

Andy

[Gera_Dorfman]

We will adjust the SK and will mention that it's a rare case. Thank You 

[the_rock]

That would be greatly appreciated!

Andy

[PetterD]

We already have several customers in panic due to this sk and patching tonight.. so more clarification in the sk would be appreciated;)

[the_rock]

Yes sir, totally agree.

Best,

Andy

[the_rock]

I actually told all my colleagues to inform our customers who are on R80.40 to either upgrade to R81.20 or R81.10 take 130, but maybe better wait till take 130 is officially recommended, which should be hopefully soon.

Most of clients are luckily already on either R81.10 or R81.20, maybe 1 or 2 on R80.40. It sounds from all I heard and read about this that chances are still very slim there would be an issue, but you never know, better be safe than sorry as they say.

Best and happy holidays.

Andy

[PetterD]

Hi!

Several customers already got really stressed out by this sk and limited information about the possibilty of it occuring and suspected it was maybe a corrupted signature that would wreck havoc on 10’s and in some cases 100’s of Firewalls.

If this has been present for 5 years and not something that could crash all Firewalls during christmas i would really weighing the possibility of issues with a fresh jumbo on all Firewalls at the same time with a 5 year old rare issue;) i have already sent the link to this thread to a few customers that were slightly in panic due to this sk.

i know it says «rare» but Check Point should really consider their description/language in these sk’s if its really not that urgent 😉

A sentence like «The issue has been identified as being present in the codebase for 5 years and is not related to any recent signature updates» would probably have stopped alot of panic. I Have worked with Check Point Firewall-1 for 22 years and cant remember an sk going as viral as this. 🙂

[the_rock]

All valid points mate, 100%.

Best,

Andy

[PetterD]

Would also like to add that what makes this a bit hard due to the lack of information is how rare this issue could potentially be in the future now that it is known.

- Does this mean that an external attacker could send a specially crafted packet to the FW and make it happen, or trick a user to visit a link with a specific URL etc.
- Would this have been a high CVSS CVE if it was a security researcher that found out about it and not Check Point.

Without more clarification it leaves us all a bit in the dark. The issuing of a High Security Alert/SK do point to that this is potentially more serious than just another rare software bug. However according to the sk it could also happen during a policy installation so who knows.. 

Basically, what all customers want to know is: Is this serious enough that i should cancel my IT-Teams Christmas Holiday to patch all Firewalls with a currently not
reccommended Jumbo 🙂

[Gera_Dorfman]

Hi,

The issue is not related to the external traffic / specially crafted packets.  SK181803 explains the scenario - it can happen as a result of the  policy installation which usually loads signatures package, or when the signatures' package is updated without policy installation. 

As I mentioned above the issue exists in the code for 5 years and during this period, hundreds of thousands of gateways have been successfully installed, upgraded, and updated, with no reported impact from this specific problem.  The specific signature that caused this behavior was removed immediately after we discovered this issue and in response we have prepared fixes for all affected versions.

[PetterD]

Thanks for the clarification! This type of information would be really appreciated in the sk 🙂

[the_rock]

Personally, and this is just me, sounds like likelihood of this happening is pretty small, but as they say, improbable does not mean impossible, thats sort of pillar of science, I still remember that from school : -)

Anyway, I agree with all you said @PetterD , I also wish sk was better worded to reflect the actual problem.

Best,

Andy

[CaseyB]

Since the update, I have been seeing this a lot on our ClusterXL:

Error: Update failed. Contract entitlement check failed. Gateway can not access internet ("https://updates.checkpoint.com/WebService/services/DownloadMetaDataService"). Check connectivity and proxy settings.

This is the error for IPS / Anti-Bot & Anti-Virus. Went from R81.10 Take 110 -> 130.

[Gera_Dorfman]

Hi,

The fix for the issue should not cause such behavior. Is there an SR opened so we can review ?

Thanks

[CaseyB]

Looking at the logs, it appears it happened 5 times yesterday and is now working fine. Problematic for a 3-4 hour window or so. No TAC SR, but I'll keep an eye on it.

[the_rock]

Sounds pretty serious...better open TAC case.

Andy

[the_rock]

Thats great news @Gera_Dorfman 

Thanks for confirming.

Andy