Checkpoint remote backups failing due to ICMP redirect blocks?

[AaronW]

I'm trying to do remote firewalls backups to a SFTP server at one of our sites.  However, whenever a remote firewall tries to make the connection I see the port 22 traffic being passed on the firewall at the destination, but the I immediately see a ICMP redirect drop on that firewall and the connection never completes.  

The flow of traffic is 

source Firewalls (1550/3600/6200) > Velo Cloud SDWAN > WAN > Velo Cloud SDWAN > destination firewall (3600) > SFTP server. 

But from the trace and looking at the logs I always see the ICMP redirect being dropped on the destination firewall. 

The firewalls at the destination site do work fine and I can create the backups and make ssh connections to that local server.  

Any idea on how do I get these backups working?  Site to site traffic works fine as long as its not coming from the source Firewalls.  Do I have to disable ICMP redirect protection on the firewall?  I'd rather not do that if there is another way. 

[Lari_Luoma]

My first thought is that the ICMP Redirect drop is probably a symptom of a routing issue rather than the actual cause of the backup failure.

One thing to keep in mind is that when a Check Point gateway performs an SFTP backup, the connection originates from the gateway itself, not from a host behind it. Because of that, the source IP used for the SSH session may not be what you expect, and the return path can end up being different than for normal site-to-site traffic.

A few things I'd check before considering disabling ICMP Redirect protection:

• Verify the actual source IP of the backup connection (tcpdump or fw monitor on the source gateway).
• Verify the return path from the SFTP server back to that source IP.
• Check for any asymmetric routing between the VeloCloud devices, destination firewall, and server.
• Look at the routing table on the destination firewall to see whether it thinks there is a better next-hop for the source gateway.

The sequence you described:

TCP/22 accepted
ICMP Redirect dropped
Connection fails

often points to the firewall detecting a suboptimal path or routing asymmetry. In many cases, the dropped redirect itself is not what breaks the connection.

I'd also run:

fw ctl zdebug + drop

and confirm exactly what is being dropped. If the only dropped packet is the ICMP Redirect message, I'd focus more on the routing than on the ICMP protection setting.

Personally, I wouldn't disable ICMP Redirect protection as a first step. If the routing is correct, the backup should work without relying on redirects in the first place.

My suspicion is that the backup traffic is being sourced from a firewall interface address, and either the SFTP server or an intermediate device is trying to send the return traffic through a different path. The ICMP Redirect is just exposing that routing inconsistency.

[PhoneBoy]

We block ICMP Redirects by default.
While you can enable them per this SK, your best bet is to fix the underlying routing issue.
https://support.checkpoint.com/results/sk/sk112772 

[AaronW]

That's the problem, I'm not seeing what the routing issue is.  The gateway of the remote server is the VIP of that network on the Checkpoint.  The checkpoint sends all traffic that isn't on a local network to the Velo, or the internet, and the firewalls on the other end are directly connected to the Velo network that is advertising all the remote networks.  Thanks for the answer, I'll have to double check everything again. 

[PhoneBoy]

It might help to understand WHEN ICMP Redirects are issued.

For discussion purposes, let's assume the target firewall has an IP of 192.0.2.1/24 on eth0 and receives traffic on this interface for 192.0.2.100.
The ICMP Redirect says "hey, 192.168.0.100 is on the same subnet, dial direct." 

Given the diagram you provided, it suggests a lack of proper segmentation on the destination network as the SFTP server should be properly behind the firewall (on a different segment from the SD-WAN device). 

[AaronW]

Each firewall cluster and members are setup with WAN network, VeloCloud SD-WAN networks, and IP on other VLAN like the main data networks, voip, etc.  None of these are overlapping. 

So I see a remote site try to initiate the backup.  First thing I see is SSH from a source IP on the remote sites VeloCloud network to the backup server.  This is accepted by the backup site firewall policy.  Then I see the backup site Firewall send an ICMP request to the backup server which is dropped as ICMP redirect.  

The backup server is on a directly connected network to its local firewall cluster, and that firewall cluster is its gateway.  

When the local firewall gets ssh traffic that is passed, and the destination is a host on a network that it is directly connected to why does it send ICMP to that host?  A host that is using that same firewalls as its own network gateway.