Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
Thomas_Eichelbu
Advisor
Advisor

sk181803 - Possible failure to boot on a Security Gateway with IPS, Anti-Bot, or Application Control

Hello folks, 

did someone already stumble over this SK ?

sk181803 - Possible failure to boot on a Security Gateway with IPS, Anti-Bot, or Application Control enabled

has anybody seen this problem in the wild already?
what does the community think? should we update all our customer to R81.10 Take 130 immediately?

what does this mean?

 

i hope this rare scenarios really strike very seldom, otherwise we will have alot of work to do!

best regards

 

0 Kudos
34 Replies
the_rock
Legend
Legend

That seems like a pretty serious issue, thanks for posting about it.

Andy

0 Kudos
CaseyB
Advisor

I saw this included in the R81.10 Take 130 notes. Looks very serious to me, looking at getting all of our gateways updated today, it seems multiple events could trigger this issue. 

0 Kudos
the_rock
Legend
Legend

Agree, its definitely serious and as you said, multiple things could cause it.

Andy

0 Kudos
Alex-
Leader Leader
Leader

The versions you list are the ones not affected so R81.20 is supposedly safe from this issue.

R81 and R80.40 still have support well into 2024 so let's hope a fix for these versions is also in preparation.

0 Kudos
the_rock
Legend
Legend

I sure hope so as well...

Andy

0 Kudos
MatanYanay
Employee
Employee

Hi @Alex and All

We do plan to release jumbo for R80.40 and R81 next week or the week after with this fix 

Thanks 

Matan.

 

Thomas_Eichelbu
Advisor
Advisor

The question is, with what probability this issue will most likely strike?

one out of ten gateways?
one out of hundreds?
or more or less?

Since what JHF or signature update this issue has started?
For how long this issue exists? 
My customers run several hundreds gateways in total, we never lost a gateway to an issue like that.
Our main reason when file systems get destroyed are power outages on ext3 filesystems.

best regards

 

0 Kudos
Gera_Dorfman
Employee
Employee

The issue has been present in our code since R80.20, and is there for the last five years in all versions between R80.20 - R81.10. (It doesn't exist in R81.20).

The issue is a rare memory corruption leading to Gaia OS corruption.

During this period, hundreds of thousands of gateways have been successfully installed, upgraded, and updated, with no reported impact from this specific problem.

It was only after a recent report from a customer that we became aware of the issue. 

In response, we have prepared fixes for all affected versions.

 

Thomas_Eichelbu
Advisor
Advisor

Hello Gera, 

thank you for this information.
so it means the issue is very severe if it strikes, but the probability of getting hit is very rare.
good to know!

thank you 


best regards

0 Kudos
dehaasm
Collaborator

Hi,

As this is a very rare issue I believe the exact root cause is unknown, what does the fix contain exactly, how does it prevent the memory corruption to occur during policy install/AV-IPS updates could you share some details?

0 Kudos
Gera_Dorfman
Employee
Employee

Hi

The root cause is known. We were able to trace and to fix this rare issue.

Thanks

Harald_Hansen
Advisor
Advisor

I'd suggest putting that into the SK, to remove confusion and unnecessary urgency on your customers behalf.

One could guess this has a very rare rate, though it is nice to have it officially. 

the_rock
Legend
Legend

That totally makes sense @Harald_Hansen . Otherwise, people may start panicking even if they are on say R81.10 version, but it sounds like its mostly under control, unless you are on R80.30 or below.

Andy

0 Kudos
Gera_Dorfman
Employee
Employee

We will adjust the SK and will mention that it's a rare case. Thank You 

0 Kudos
the_rock
Legend
Legend

That would be greatly appreciated!

Andy

0 Kudos
PetterD
Collaborator

We already have several customers in panic due to this sk and patching tonight.. so more clarification in the sk would be appreciated;)

CCSM / CCSE / CCVS / CCTE
(1)
the_rock
Legend
Legend

Yes sir, totally agree.

Best,

Andy

the_rock
Legend
Legend

I actually told all my colleagues to inform our customers who are on R80.40 to either upgrade to R81.20 or R81.10 take 130, but maybe better wait till take 130 is officially recommended, which should be hopefully soon.

Most of clients are luckily already on either R81.10 or R81.20, maybe 1 or 2 on R80.40. It sounds from all I heard and read about this that chances are still very slim there would be an issue, but you never know, better be safe than sorry as they say.

Best and happy holidays.

Andy

PetterD
Collaborator

Hi!

Several customers already got really stressed out by this sk and limited information about the possibilty of it occuring and suspected it was maybe a corrupted signature that would wreck havoc on 10’s and in some cases 100’s of Firewalls.

If this has been present for 5 years and not something that could crash all Firewalls during christmas i would really weighing the possibility of issues with a fresh jumbo on all Firewalls at the same time with a 5 year old rare issue;) i have already sent the link to this thread to a few customers that were slightly in panic due to this sk.

i know it says «rare» but Check Point should really consider their description/language in these sk’s if its really not that urgent 😉

A sentence like «The issue has been identified as being present in the codebase for 5 years and is not related to any recent signature updates» would probably have stopped alot of panic. I Have worked with Check Point Firewall-1 for 22 years and cant remember an sk going as viral as this. 🙂

CCSM / CCSE / CCVS / CCTE
the_rock
Legend
Legend

All valid points mate, 100%.

Best,

 

Andy

PetterD
Collaborator


Would also like to add that what makes this a bit hard due to the lack of information is how rare this issue could potentially be in the future now that it is known.

- Does this mean that an external attacker could send a specially crafted packet to the FW and make it happen, or trick a user to visit a link with a specific URL etc.
- Would this have been a high CVSS CVE if it was a security researcher that found out about it and not Check Point.


Without more clarification it leaves us all a bit in the dark. The issuing of a High Security Alert/SK do point to that this is potentially more serious than just another rare software bug. However according to the sk it could also happen during a policy installation so who knows.. 

Basically, what all customers want to know is: Is this serious enough that i should cancel my IT-Teams Christmas Holiday to patch all Firewalls with a currently not
reccommended Jumbo 🙂

CCSM / CCSE / CCVS / CCTE
0 Kudos
Gera_Dorfman
Employee
Employee

Hi,

The issue is not related to the external traffic / specially crafted packets.  SK181803 explains the scenario - it can happen as a result of the  policy installation which usually loads signatures package, or when the signatures' package is updated without policy installation. 

As I mentioned above the issue exists in the code for 5 years and during this period, hundreds of thousands of gateways have been successfully installed, upgraded, and updated, with no reported impact from this specific problem.  The specific signature that caused this behavior was removed immediately after we discovered this issue and in response we have prepared fixes for all affected versions.

PetterD
Collaborator

Thanks for the clarification! This type of information would be really appreciated in the sk 🙂

CCSM / CCSE / CCVS / CCTE
0 Kudos
the_rock
Legend
Legend

Personally, and this is just me, sounds like likelihood of this happening is pretty small, but as they say, improbable does not mean impossible, thats sort of pillar of science, I still remember that from school : -)

Anyway, I agree with all you said @PetterD , I also wish sk was better worded to reflect the actual problem.

Best,

Andy

CaseyB
Advisor

Since the update, I have been seeing this a lot on our ClusterXL:

Error: Update failed. Contract entitlement check failed. Gateway can not access internet ("https://updates.checkpoint.com/WebService/services/DownloadMetaDataService"). Check connectivity and proxy settings.

This is the error for IPS / Anti-Bot & Anti-Virus. Went from R81.10 Take 110 -> 130.

0 Kudos
Gera_Dorfman
Employee
Employee

Hi,

The fix for the issue should not cause such behavior. Is there an SR opened so we can review ?

Thanks

0 Kudos
CaseyB
Advisor

Looking at the logs, it appears it happened 5 times yesterday and is now working fine. Problematic for a 3-4 hour window or so. No TAC SR, but I'll keep an eye on it.

0 Kudos
the_rock
Legend
Legend

Sounds pretty serious...better open TAC case.

Andy

0 Kudos
the_rock
Legend
Legend

Thats great news @Gera_Dorfman 

Thanks for confirming.

Andy

0 Kudos

Leaderboard

Epsum factorial non deposit quid pro quo hic escorol.

Upcoming Events

    CheckMates Events