Who had outages with the geo_location.C error in sk174783

[Jan_Kleinhans]

Hello everybody,

we experienced a 5 hour full internet outage because of the error in geo_location.C created by Checkpoint.

As our Cluster failed over on thursday morning because of another bug we had these full outage as the services could not be started because of dynamic_objects process utilizing 100% CPU and not coming to an end.

Has anybody experienced the same problem in this hard impact? How did you handle it? What is your experience with checkpoint support in this moment?

Jan

[G_W_Albrecht]

Can you please mention your version and jumbo take ?

[Jan_Kleinhans]

R80.40 T119 as requested to install by Checkpoint Support because of latency with SSL Inspection. Take 119 produces the cluster crashes.

[Ruan_Kotze]

Thanks for the heads-up.  Busy troubleshooting a perplexing IPSEC tunnel issue and TAC had us deploy T119.

[Jan_Kleinhans]

Keep my fingers crossed for you.

[Ruan_Kotze]

OK so we got bitten hard as well.  All tunnels on the cluster failed - traffic dropped with local interface spoofing.  Gateways also randomly stopped passing DHCP traffic between VLANs.  Rolled back to T118 and all issues resolved.

Lesson learned for the nth time - be wary of installing ongoing takes as part of troubleshooting.  And only update on one cluster member so that you can roll-back quickly.

[PhoneBoy]

Link to the SK: https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...

[Peter_Thome]

Any information, why this sk can't be accessed with expert level as partner?

Especially, as this is linked in the supportcenter "Hot News" Slider and if you click the link, it's just showing: 

To view this solution, higher access level is required.

[PhoneBoy]

The only higher access level is internal, and yes, it is accessible to employees still.
Users shouldn't run into this issue anymore unless we happen to release a faulty update.
Why it's linked in "Hot News" is a separate question.
@Ronen_Zel ?

[Ronen_Zel]

Good catch. Since the issue was indeed already resolved, it should no longer appear in Support Center's "Hot News" section. The item is now removed.

Thanks for reporting this!

[the_rock]

We currently have a case with TAC where policy push takes 8 minutes when geo location to block all the countries, except maybe 7 or 8 is enabled, but as soon as you disable that rule, policy takes 3 minutes on cloud mgmt server. Engineer mentioned internal sk and something about geo_location.C file, but date shows September 1st, so not sure if its 100% related. They also mentioned dynamic update i=object issue, but customer does not even use those in the policy. I am still waiting for next steps, so will see what they advise.

[Jan_Kleinhans]

Hello,

we have also the problem that installing the policy takes a long time (didn't measure it) with updateable_ojects process taking 100% CPU for this time. I don't know if it started directly after the case mentioned in this sk or if it started later.
I opened a case for this problem today.

Regards,

Jan

[the_rock]

Would you mind sharing the status of that issue? Is it still ongoing?

[Jan_Kleinhans]

Yes the slow installation process is ongoing. In our environment the installation takes 10 minutes. But then it's working so that it doesn't have a big impact. 
Case is opened a few minutes so we will have to wait till checkpoint requests some information or will have a clue.

[the_rock]

Ok, sounds good. In our case as well, policy does work, it just takes more than double time when geo block rule is enabled, but lets see what TAC says. As long as policy functions, I will take that : )

[Micky_Michaeli]

Hi @Jan_Kleinhans , @the_rock 

We are delivering to JHFs an improvement to the loading process of Geo Updatable objects. It's improving the policy installation duration when Geo Updatable objects are used in policy.

You can open SR for port-fix in case you wish to get this as a private fix instead of waiting for the GA release of the different JHFs as it can take several weeks.

Thanks.

[the_rock]

Thank you for the response. So is there an official solution for customers who still have this issue or do we have to wait until next jumbo hotfix? I get port-fix can be requested, but I am little apprehensive about it, because it causes issues later on when next general jumbo take has to be installed...

[G_W_Albrecht]

As @Micky_Michaeli wrote: You can either wait a few weeks until this HF is included in GA Jumbo or request a port to the GA Jumbo you have installed. As next GA Jumbo should already contain this HF your thought is unnecessary...

[the_rock]

@G_W_Albrecht ...I get what he said, no argument there :). I was more actually wondering if there was some sort of workaround for the time being that may not require rebooting the box, due to the port fix install.

[Micky_Michaeli]

Hi @the_rock,

I think that we don’t have other WA than installing the HF that requires a reboot.

Thanks.

[the_rock]

Thanks Micky. So, will this be permanently fixed in jumbo take 121 for R80.40?

[PhoneBoy]

Most likely a future one.