topic Re: Who had outages with the geo_location.C error in sk174783 in Firewall and Security Management

Who had outages with the geo_location.C error in sk174783

Jan_Kleinhans — Mon, 02 Aug 2021 08:19:39 GMT

Hello everybody,

we experienced a 5 hour full internet outage because of the error in geo_location.C created by Checkpoint.

As our Cluster failed over on thursday morning because of another bug we had these full outage as the services could not be started because of dynamic_objects process utilizing 100% CPU and not coming to an end.

Has anybody experienced the same problem in this hard impact? How did you handle it? What is your experience with checkpoint support in this moment?

Jan

Re: Who had outages with the geo_location.C error in sk174783

G_W_Albrecht — Mon, 02 Aug 2021 09:56:56 GMT

Can you please mention your version and jumbo take ?

Re: Who had outages with the geo_location.C error in sk174783

Jan_Kleinhans — Mon, 02 Aug 2021 09:59:38 GMT

R80.40 T119 as requested to install by Checkpoint Support because of latency with SSL Inspection. Take 119 produces the cluster crashes.

Re: Who had outages with the geo_location.C error in sk174783

Ruan_Kotze — Mon, 02 Aug 2021 11:05:39 GMT

Thanks for the heads-up. Busy troubleshooting a perplexing IPSEC tunnel issue and TAC had us deploy T119.

Re: Who had outages with the geo_location.C error in sk174783

Jan_Kleinhans — Mon, 02 Aug 2021 11:22:22 GMT

Keep my fingers crossed for you.

Re: Who had outages with the geo_location.C error in sk174783

PhoneBoy — Mon, 02 Aug 2021 16:23:23 GMT

Link to the SK: https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk174783&partition=Advanced&product=Quantum

Re: Who had outages with the geo_location.C error in sk174783

Ruan_Kotze — Tue, 03 Aug 2021 08:16:27 GMT

OK so we got bitten hard as well. All tunnels on the cluster failed - traffic dropped with local interface spoofing. Gateways also randomly stopped passing DHCP traffic between VLANs. Rolled back to T118 and all issues resolved.

Lesson learned for the nth time - be wary of installing ongoing takes as part of troubleshooting. And only update on one cluster member so that you can roll-back quickly.

Re: Who had outages with the geo_location.C error in sk174783

Peter_Thome — Wed, 01 Sep 2021 09:47:51 GMT

Any information, why this sk can't be accessed with expert level as partner?

Especially, as this is linked in the supportcenter "Hot News" Slider and if you click the link, it's just showing:

To view this solution, higher access level is required.

Re: Who had outages with the geo_location.C error in sk174783

PhoneBoy — Wed, 01 Sep 2021 15:39:26 GMT

The only higher access level is internal, and yes, it is accessible to employees still.
Users shouldn't run into this issue anymore unless we happen to release a faulty update.
Why it's linked in "Hot News" is a separate question.
@Ronen_Zel ?

Re: Who had outages with the geo_location.C error in sk174783

Ronen_Zel — Thu, 02 Sep 2021 05:39:36 GMT

Good catch. Since the issue was indeed already resolved, it should no longer appear in Support Center's "Hot News" section. The item is now removed.

Thanks for reporting this!

Re: Who had outages with the geo_location.C error in sk174783

the_rock — Thu, 02 Sep 2021 11:29:20 GMT

We currently have a case with TAC where policy push takes 8 minutes when geo location to block all the countries, except maybe 7 or 8 is enabled, but as soon as you disable that rule, policy takes 3 minutes on cloud mgmt server. Engineer mentioned internal sk and something about geo_location.C file, but date shows September 1st, so not sure if its 100% related. They also mentioned dynamic update i=object issue, but customer does not even use those in the policy. I am still waiting for next steps, so will see what they advise.

Re: Who had outages with the geo_location.C error in sk174783

Jan_Kleinhans — Thu, 02 Sep 2021 11:35:19 GMT

Hello,

we have also the problem that installing the policy takes a long time (didn't measure it) with updateable_ojects process taking 100% CPU for this time. I don't know if it started directly after the case mentioned in this sk or if it started later.
I opened a case for this problem today.

Regards,

Jan

Re: Who had outages with the geo_location.C error in sk174783

the_rock — Thu, 02 Sep 2021 11:40:25 GMT

Would you mind sharing the status of that issue? Is it still ongoing?

Re: Who had outages with the geo_location.C error in sk174783

Jan_Kleinhans — Thu, 02 Sep 2021 11:46:58 GMT

Yes the slow installation process is ongoing. In our environment the installation takes 10 minutes. But then it's working so that it doesn't have a big impact.
Case is opened a few minutes so we will have to wait till checkpoint requests some information or will have a clue.

Re: Who had outages with the geo_location.C error in sk174783

the_rock — Thu, 02 Sep 2021 11:51:11 GMT

Ok, sounds good. In our case as well, policy does work, it just takes more than double time when geo block rule is enabled, but lets see what TAC says. As long as policy functions, I will take that : )

Re: Who had outages with the geo_location.C error in sk174783

Micky_Michaeli — Mon, 06 Sep 2021 08:28:23 GMT

Hi @Jan_Kleinhans , @the_rock

We are delivering to JHFs an improvement to the loading process of Geo Updatable objects. It's improving the policy installation duration when Geo Updatable objects are used in policy.

You can open SR for port-fix in case you wish to get this as a private fix instead of waiting for the GA release of the different JHFs as it can take several weeks.

Thanks.

Re: Who had outages with the geo_location.C error in sk174783

the_rock — Mon, 06 Sep 2021 11:11:48 GMT

Thank you for the response. So is there an official solution for customers who still have this issue or do we have to wait until next jumbo hotfix? I get port-fix can be requested, but I am little apprehensive about it, because it causes issues later on when next general jumbo take has to be installed...

Re: Who had outages with the geo_location.C error in sk174783

G_W_Albrecht — Mon, 06 Sep 2021 11:48:05 GMT

As @Micky_Michaeli wrote: You can either wait a few weeks until this HF is included in GA Jumbo or request a port to the GA Jumbo you have installed. As next GA Jumbo should already contain this HF your thought is unnecessary...

Re: Who had outages with the geo_location.C error in sk174783

the_rock — Mon, 06 Sep 2021 12:03:38 GMT

@G_W_Albrecht ...I get what he said, no argument there :). I was more actually wondering if there was some sort of workaround for the time being that may not require rebooting the box, due to the port fix install.

Re: Who had outages with the geo_location.C error in sk174783

Micky_Michaeli — Sun, 19 Sep 2021 06:28:23 GMT

Hi @the_rock,

I think that we don’t have other WA than installing the HF that requires a reboot.

Thanks.