Routing loop with exponential packet growth

[Kaspars_Zibarts]

Just wondering if anyone else has seen this with chassis running VSX (R76SP30). Some of our WAN sites complained of occasional slowness which we managed to drill down to "packet storms". When we looked at the actual traffic it was silly routing loop between FW and MPLS router. (Not that) funny part is that even though IP TTL is correctly reduced by both nodes, firewall doubles number of packets in response in each cycle! So it it goes 1 - 2 - 4 - 8 etc. 

We started off with TTL 123 so it generate insane amount of packets (2^123...)

Has anyone seen anything like it? 

SR on it's way but would be good to know if we can avoid this somehow.

[Timothy_Hall]

A quite timely post as I was just working on a new section of my book's second edition I titled "Routing Supernet Loops".

The issue is caused by adding supernetted routes to the firewall's routing table for the entire RFC1918 address space pointing to the internal core router.  The core router then of course has a default route pointing back to the firewall. 

So suppose the firewall has all RFC1918 address space routed to the core internal router like this:

10.0.0.0/8 -> core router

172.16.0.0/12 -> core router

192.168.0.0/16 -> core router

default -> Internet perimeter router

On the internal core router there are 10 RFC1918 networks actually in use with directly-attached or static routes referencing them, and a default gateway:

192.168.0.0/21 -> some internal nexthop

192.168.9.0/24 - direct

192.168.10.0/24 - direct

default -> firewall

Now suppose a packet arrives at the core router bound for 192.168.66.1.  The core router doesn't have a route for that network so it hits the default route and sends it to the firewall.  Assuming the firewall security policy allows it, the firewall will then source NAT the packet if configured to do so, and then based on IP routing send it back to the core router, who then flips it right back to the firewall.  If NAT was present, the firewall will kill the packet at that point because it is arriving with a source IP address that the firewall assigned in the NAT operation which should not happen. 

However if there were no NAT rules matching the traffic (or it matched an anti-NAT rule) the packet will loop between the firewall and core router until the TTL is finally decremented to zero, at which point the packet will be discarded.  Now imagine a nasty scanning worm starts rampaging inside your network and is sending hundreds or thousands of probe packets per second to randomized nonexistent RFC1918 addresses that don't exist in your network, and each one of these individual packets gets looped 255 times between the core router and the firewall.  This effect crashed entire networks when Code Red and Nimda got loose in corporate networks, and it took *weeks* for some organizations to fully recover.

There are two solutions:

1) On the firewall, only route the specific RFC1918 addresses you are actually using to your core router

2) If you must supernet RFC1918 routes from your firewall to core router, add these routes on your core router:

192.168.0.0/16 -> Null0

172.16.0.0/12 -> Null0

10.0.0.0/8 -> Null0

HOWEVER if you are using any RFC1918 space on your DMZ networks attached to the firewall (or accessing them through VPNs), make sure to add specific static routes to these networks via the firewall on your core router before doing the above!

--
My book "Max Power: Check Point Firewall Performance Optimization"
now available via http://maxpowerfirewalls.com.

[Kaspars_Zibarts]

I guess I don't mind occasional loop due to dynamic nature of WAN/MPLS connections.

I was more curious about the fact that specifically chassis doubled forwarded packets with every loop cycle  

If a loop dies out after bouncing 120 packets it's not a big drama. But if you change that number to 2^120.. then we're starting to impact our WAN users. 

Haven't seen this on appliances in my IPSO/Splat/XOS/Gaia history.

The null route option is definitely something we need to look into. Thanks for the idea!

[Timothy_Hall]

Is there any kind of cluster or High Availability involved here, either on the firewall itself or the surrounding network infrastructure?  The only situation I can see causing packet amplification like that would be a cluster split-brain situation, or if the traffic was broadcast-oriented at either Layer 2 (broadcast MAC address) or Layer 3 (broadcast IP address).

--
My book "Max Power: Check Point Firewall Performance Optimization"
now available via http://maxpowerfirewalls.com.

[Kaspars_Zibarts]

It's 41k chassis "cluster" with 3 SGMs in each. So not traditional cluster or HA - it's an SMO. I have verified that MAC address always comes from the active chassis (had a suspicion that standby chassis might "chip in")

We don't have 100% proof but it seemed to have started after we expanded from single SGM per chassis to three..

[Timothy_Hall]

41Ks and higher run their own fork of Check Point code and have a special CLI among other differences.  Will definitely need to engage TAC to get a 41k-64k expert on this one, is certainly possible that the new SGMs are misconfigured and not coordinating the handling of traffic properly amongst themselves.

--
My book "Max Power: Check Point Firewall Performance Optimization"
now available via http://maxpowerfirewalls.com.