topic Re: Policy push overwrote default route on cluster active gateway in Firewall and Security Management

Policy push overwrote default route on cluster active gateway

the_rock — Fri, 25 Nov 2022 00:35:43 GMT

Hey guys,

I really hope someone can shed some light with this. So, one of our colleagues went into client's environment (they use smart-1 cloud) and 6000 series cluster and simply added couple of IP addresses to block group and once policy was applied, we noticed that active member could not be accessed.

At this point, thankfully, ssh to backup worked fine, so once we ssh-ed to active from backup, noticed that default route was gone. Now, in my 15 years with CP, I had NEVER seen or heard of problem like this. Keep in mind, failover never happened, however, there was Internet outage, as default route was gone. Default route was added back via clish afterwards and we did push policy couple of times afterwards and it was fine.

Now, just to try and figure this out ourselves, we downloaded audit.log from /var/log/audit dir, but it was not useful at all, as it does not have any timestamps, but we searched for words, such as route, default, delete, but no luck. We are 99.99% sure that something else caused this, rather than policy push, but really hard to say what at this point.

Also checked /var/log/messages files, but no luck there either. There was no one who was even logged into firewalls before this issue happened, so it begs the question HOW this happened.

We ended up opening TAC case for it, but after doing zoom meeting, gentleman told us would consult further internally and see what else can be done to try and find the reason.

If anyone else has an idea or any other file(s) we could check, it would be greatly appreciated!

Thanks as always.

Re: Policy push overwrote default route on cluster active gateway

DirkB — Fri, 25 Nov 2022 09:34:30 GMT

As a newcomer at Checkpoint just a shot in the dark: is a CloningGroup defined? We once had strange effects with it (admin, cadmin ...), there were also accesses to cluster members lost (but not the default route 😉 ) But Routing is group feature ...

Re: Policy push overwrote default route on cluster active gateway

the_rock — Fri, 25 Nov 2022 11:35:29 GMT

Hey @DirkB , thanks for the response, but thats definitely not it, sorry. Lets see what TAC comes back with, as we are totally out of ideas where to even look next, as we checked everything we could humanly think of.

Re: Policy push overwrote default route on cluster active gateway

BikeMan — Fri, 25 Nov 2022 11:42:02 GMT

Hi,

You could check /var/log/routed.log. May be more info in it.

Rgds,

Re: Policy push overwrote default route on cluster active gateway

the_rock — Fri, 25 Nov 2022 11:46:00 GMT

Ok, thats good idea, ty, will check that in couple hours and report back.

Cheers.

Re: Policy push overwrote default route on cluster active gateway

DirkB — Fri, 25 Nov 2022 12:19:06 GMT

only stupid casually if DHCP (I don't know 😉 ):Are Kernel Routes activated, Kernel Routes are not activated by default.

Is Ping activated for the default route? If so, I think the route is deleted from table per design, if ping fails (or failed temporary) to next hop (and readded, if Ping succeeds) ... but it would be unclear to me how the static entries would be accessed - perhaps with save config (with the policies) ... wich is the timestamp of configs (can you track an order?). Perhaps just a coincidence in time.

Re: Policy push overwrote default route on cluster active gateway

the_rock — Fri, 25 Nov 2022 12:26:11 GMT

I dont personally think that would have anything to do with it, regardless about the ping failing. Let us see what TAC says next, because we absolutely have to give a reason to the customer, this cannot happen again, specially given the fact it caused Internet outage.

Re: Policy push overwrote default route on cluster active gateway

the_rock — Fri, 25 Nov 2022 12:27:32 GMT

I attached a file with relevant times when it happened and messages from routed_messages and routed.log. Not sure if they matter, but I saw same messages for the last year, so hard to believe its relevant, but who knows : - )

Re: Policy push overwrote default route on cluster active gateway

the_rock — Tue, 06 Dec 2022 22:34:18 GMT

In case anyone ever has this issue, please be mindful that this happened AFTER we upgraded customer from R80.40 to R81.10 and we found out with help from TAC escalations its caused by ISP redundancy. so appears something in R81.10 is different than in R80.40 for this, what, no clue. I also replicated this in my lab as well I will update once I find out 100% what exactly is causing the behavior, as it happened twice already.

Re: Policy push overwrote default route on cluster active gateway

the_rock — Thu, 08 Dec 2022 03:20:32 GMT

For anyone who has ISP redundancy, TAC gave below change as issue was replicated, so you would have to do this. What exactly it does, not 100% sure, never got an answer, though it says to disable the script uncomment the line. I definitely will get an answer as to what is EXACT purpose of this.

Please uncomment this line "exit 1" from file $FWDIR/bin/cpisp_update file on the gateway:

# To disable the script uncomment the following line.
# exit 1

set MISP_MAX_ISPS = 10
set ISP_STATUS_FILE = "$FWDIR/conf/cpispstatus.conf"

Re: Policy push overwrote default route on cluster active gateway

Blason_R — Thu, 08 Dec 2022 03:10:10 GMT

Thats wonderful finding let me replicate that in my scenario and test it out.

Re: Policy push overwrote default route on cluster active gateway

the_rock — Thu, 08 Dec 2022 03:19:40 GMT

I hope you never have the same problem, as its really frustrating. But, now that we know what was causing it, its an easy fix. I will definitely make sure to get an explanation if upgrade caused this, as it never happened on R80.40.

Re: Policy push overwrote default route on cluster active gateway

BikeMan — Thu, 08 Dec 2022 07:58:15 GMT

Hi,

Could you share the version you are using on the Smart-1 and on the firewall ?

I am managing about 180 modules all over the world and the new company standard is to use ISP redundancy every where. Currently running 80.40 and 80.20, upgrade to 81.10 is planned next year...

Thanks,

Re: Policy push overwrote default route on cluster active gateway

the_rock — Thu, 08 Dec 2022 11:42:11 GMT

I hate to say this, but unless CP actually fixed this permanently, you WILL most likely encounter this problem. Mgmt is S1C (smart-1 cloud) and gateways are 6400, all running R81.10. Now, keep in mind, when S1C was on R81.10 (upgrades are managed and scheduled by CP) and gateways on R80.40, this NEVER happened. Once gateways were upgraded to R81.10, thats when issue occurred 1st time maybe a week later and then 2nd time 2 weeks after. This is why Im pressing TAC to provide logic as to whether this is an issue in R81+ and how it can be avoided without having to modify that file.

Another interesting thing I will also try to confirm is whether this ONLY affects HA isp config or load sharing as well. As soon as I get the info, will update here.

Re: Policy push overwrote default route on cluster active gateway

the_rock — Thu, 08 Dec 2022 11:53:45 GMT

Also @BikeMan , to add to my last comment, I also saw something interesting below:

https://sc1.checkpoint.com/documents/R81.10/WebAdminGuides/EN/CP_R81.10_Quantum_SecurityGateway_Guide/Topics-FWG/ISP-Redundancy-CLI.htm

See this part:

The ISP Redundancy Script

When the Security Gateway starts, or an ISP link state changes, the $FWDIR/bin/cpisp_update script runs on the Security Gateway.

This script changes the default route of the Security Gateway.

Warning - We do not recommend that you make any changes to this script.

***************************************

Now, here is my own logic. NEITHER of those scenarios applied to the customer. So, obviously, once gateways were upgraded, they had to be rebooted, so according to the document, it would imply that default route would change every time fw is rebooted?? That makes no sense. Also, their primary ISP link never failed either. Anyway, lets see what TAC says.

Re: Policy push overwrote default route on cluster active gateway

Ilya_Yusupov — Fri, 09 Dec 2022 11:27:33 GMT

Hi @the_rock ,

I reviewed the case and i am sorry but the provided solution is wrong here, uncomment that line means you disable ISPR means failover of ISP will not work.

In this case customer is running with BGP which its not supported with ISPR, even if it worked for a customer on previous version we can't guarantee that it worked correctly as dynamic routing and ISPR are impacting routing of each other.

There is some discussion with RnD about the case but not sure there will be a solution.

Will keep update once we have some conclusions.

Thanks,

Ilya

Re: Policy push overwrote default route on cluster active gateway

the_rock — Fri, 09 Dec 2022 12:26:25 GMT

Thanks @Ilya_Yusupov , very grateful for your update. So, just wondering, I could not find any documents or articles stating that ISPR is not supported with BGP. Would you be able to provide that please?

Thanks in advance.

Re: Policy push overwrote default route on cluster active gateway

Ilya_Yusupov — Fri, 09 Dec 2022 12:36:46 GMT

Sure i will share it on Sunday as i am not in front a laptop, its not only bgp but dynamic routing with ispr.

Re: Policy push overwrote default route on cluster active gateway

the_rock — Fri, 09 Dec 2022 12:57:46 GMT

No rush. I will ask via the TAC case.

Re: Policy push overwrote default route on cluster active gateway

the_rock — Fri, 09 Dec 2022 15:11:15 GMT

K, TAC provided the link, but here is what makes no sense to me personally.

https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk170418&partition=Basic&product=Anti-Bot,

this part:

PMTR-68991ISP Redundancy is not supported if Dynamic Routing is configured (because the ISP Redundancy feature must create a static default route that overrides the default route created by dynamic routing).

well, customer's DG IS INDEED derived from ISPR and NOT bgp, so I cant connect the dots here as to why this would even apply to them.