topic Re: route flipping on R80.40 in Firewall and Security Management

route flipping on R80.40

Steffen_Appel — Mon, 25 May 2020 08:19:33 GMT

We have upgraded from R80.10 to R80.40 (HF48) and have a route flipping issue:

# ip route get a.b.c.d

a.b.c.d via <correct next hop ip> dev eth5 src <correct source>

# ip route get a.b.c.d

a.b.c.d via <correct next hop ip> dev eth2 src <correct source>

For whatever reason the interface in the routing table is changed from eth5 to eth2.

In fw monitor you can see it as well, the first packet goes to eth5 correctly, the second one after a route flip goes to eth2, which is wrong.

[vs_0][fw_2] eth5:O[44]: ****** -> ***** (UDP) len=200 id=61683 UDP: 2464 -> 49910

[vs_0][fw_2] eth2:O[44]: ****** -> ****** (UDP) len=200 id=49885 UDP: 2464 -> 49910

Did anyone have similiar problems?

TAC case is opened.

Re: route flipping on R80.40

Timothy_Hall — Mon, 25 May 2020 12:27:01 GMT

Is your firewall statically or dynamically routed? What does the actual routing table (netstat -rn) show for the destination network?

Re: route flipping on R80.40

Steffen_Appel — Mon, 25 May 2020 12:31:37 GMT

It is completly static routed.

Re: route flipping on R80.40

Yair_Shahar — Tue, 26 May 2020 09:33:42 GMT

Hi,

Do you happen to have this route duplicate on both interfaces?
Sound weird issue networking wise, I would say checking Gaia configuration in clish (show configuration), in config/active and in kernel (ip route, ifconfig)

also - any chance there is physical interface flapping going on?

Thanks,
Yair

Re: route flipping on R80.40

Steffen_Appel — Tue, 26 May 2020 10:51:05 GMT

No interface flapping.
The problem occurs on both cluster nodes.

Only one static route to the destination host:
set static-route a.b.c.d/32 nexthop gateway address ****** on

And in active:
routed:instance:default:static:network:a.b.c.d t
routed:instance:default:static:network:a.b.c.d:masklen:32 t
routed:instance:default:static:network:a.b.c.d:masklen:32:gateway t
routed:instance:default:static:network:a.b.c.d:masklen:32:gateway:address:****** t

Re: route flipping on R80.40

Steffen_Appel — Tue, 26 May 2020 12:58:33 GMT

We have only seen it for UDP and ICMP never for TCP.

Re: route flipping on R80.40

Timothy_Hall — Tue, 26 May 2020 13:45:40 GMT

> We have only ssen it for UDP and ICMP never for TCP.

This statement makes no sense unless you are using Policy Based Routing (PBR), are you? Routing is performed by the Linux IP Driver and is not generally influenced by anything in Check Point's code that would be distinguishing service or protocol, unless a feature such as ISP Redundancy is in use.

Regular IP routing only looks at destination IP address and could care less about service or protocol. Please provide the output of netstat -renv from expert mode for the route in question, once when it is showing eth2 and the other when it is showing eth5. We need to see the live routing table and associated flags/use.

Re: route flipping on R80.40

Steffen_Appel — Tue, 26 May 2020 13:35:13 GMT

No pbr is used:

show pbr
PBR Summary

PBR has 0 tables
PBR has 0 rules

netstat shows:

a.b.c.d ****** 255.255.255.255 UGH 0 0 0 eth5 in the correct case

since the node is not in production right now of course I cannot provide the incorrect one.

Just for the info, there are about 130 static routes on the gateway.

Re: route flipping on R80.40

Timothy_Hall — Tue, 26 May 2020 14:55:42 GMT

Without being able to see the unredacted value of the route next hop address and the full unredacted eth2 and eth5 interface configuration it is difficult to surmise what is wrong. Feel free to PM this information to me without redacting the IP addresses. If you aren't comfortable with doing that you'll need to work through TAC.

Re: route flipping on R80.40

Steffen_Appel — Wed, 27 May 2020 07:51:24 GMT

Did you receive the PN?

Re: route flipping on R80.40

Timothy_Hall — Wed, 27 May 2020 12:58:23 GMT

No I don't see your PM.

Re: route flipping on R80.40

Steffen_Appel — Wed, 27 May 2020 13:30:56 GMT

PM sent.

Re: route flipping on R80.40

Ilya_Yusupov — Sun, 31 May 2020 15:48:47 GMT

Hi @Steffen_Appel ,

do you have ISPR configured on the system?

Re: route flipping on R80.40

Steffen_Appel — Tue, 02 Jun 2020 05:52:30 GMT

No ISP redundancy configured no.

Re: route flipping on R80.40

Ilya_Yusupov — Tue, 02 Jun 2020 06:12:18 GMT

Strange, did you open TAC case? if so can you share the number so i can follow it?

Re: route flipping on R80.40

Steffen_Appel — Tue, 02 Jun 2020 07:16:13 GMT

Yes TAC case is open, will PM you the number.

Had a two day debugging season on the WE.

Re: route flipping on R80.40

Steffen_Appel — Tue, 30 Jun 2020 05:17:46 GMT

R&D is still investigating.

Re: route flipping on R80.40

Steffen_Appel — Wed, 29 Jul 2020 07:30:59 GMT

Still no solution, but additional customers with the same problem.

Re: route flipping on R80.40

John_Fleming — Fri, 31 Jul 2020 21:08:50 GMT

by chance does

show route all

from clish show anything strange?

I would enable trace (set trace kernel all on, set trace static all on, etc) and then check /var/log/routed.log to see if you can get some hints on what is going on as routed is the only process that should be making routing changes so it would be the place to debug.

Re: route flipping on R80.40

Timothy_Hall — Fri, 31 Jul 2020 23:09:29 GMT

Actually I've received some inside information about this case, and it appears to be a problem with the Linux route cache (ip route show cache) which is somehow getting cached route entries that are associated with the wrong interface. The main routing table (ip route show or netstat -rn) always shows the problematic route associated with the correct interface. So this would appear to be a Gaia/Linux bug, and I find it interesting that the IP route cache functionality was abandoned in the 3.6 version of the Linux kernel, but unfortunately there is no apparent way to disable it permanently. A temporary workaround is to flush the route cache with the ip route flush cache command but the problem just comes back later.