How to disable local anti-spoofing in R81.20 (cluster with bridged interfaces)?

[AlekzNet]

Hi All,

We have a ClusterXL (R81.20 JHF76) with bridged interfaces. This cluster is managed through the bridge. Basically, this design is mentioned in https://sc1.checkpoint.com/documents/R77/CP_R77_SecurityGatewayTech_WebAdmin/96332.htm
(section “Management over Bridge”)

While it's reachable from the management station, the cluster itself can not reach anything because of the "local anti-spoofing".

For example, this is "ping 1.1.1.1":

@;6007031.309;[kern];[tid_0];[SIM4];sim_pkt_send_drop_notification: (0,0) received drop, reason: Local Spoofing (12), conn: <10.1.9.114,10405,1.1.1.1,0,1>;
@;6007031.310;[kern];[tid_0];[SIM4];sim_pkt_send_drop_notification: sending packet dropped notification drop mode: 0 debug mode: 1 send as is: 0 track_lvl: 2, conn: <10.1.9.114,10405,1.1.1.1,0,1>;
@;6007031.311;[kern];[tid_0];[SIM4];sim_pkt_send_drop_notification: sending single drop notification, conn: <10.1.9.114,10405,1.1.1.1,0,1>;
@;6007054.312;[kern];[tid_0];[SIM4];pkt_handle_no_match: drop packet due to local spoofing, conn: <10.1.9.114,10405,1.1.1.1,0,1>;
@;6007054.313;[kern];[tid_0];[SIM4];do_packet_finish: SIMPKT_IN_DROP vsid=0, conn:<10.1.9.114,10405,1.1.1.1,0,1>;

[vs_0][fw_1] eth1.121:o[44]: 10.1.9.115 -> 1.1.1.1 (ICMP) len=84 id=36827
ICMP: type=8 code=0 echo request id=35174 seq=15
[vs_0][fw_1] eth1.121:O[44]: 10.1.9.114 -> 1.1.1.1 (ICMP) len=84 id=36827
ICMP: type=8 code=0 echo request id=26935 seq=15
[vs_0][ppak_0] eth2:i[44]: 10.1.9.114 -> 1.1.1.1 (ICMP) len=84 id=36827
ICMP: type=8 code=0 echo request id=26935 seq=15

eth1.121 10.1.9.114 - Cluster IP
eth1.121 10.1.9.115 - FW1 (active FW)
eth2 - bridge sub-interface

Traffic flow:

eth1.121 -> Switch -> eth2

I tried the following:

• Disabled anti-spoofing on all interfaces (in "Network Topology")
• Extended Cluster Anti-Spoofing is off
• fw ctl set int fw_local_interface_anti_spoofing 0
• fw ctl set int fw_antispoofing_enabled 0

But it did not work.

sk105899   does not mention R81.20 (BTW, fwx_bridge_reroute_enabled=1 breaks access to the passive gateway).

Any ideas?

Thank you in advance!

Alex

[AlekzNet]

There is a related thread: https://community.checkpoint.com/t5/Management/Disable-quot-Local-interface-address-spoofing-quot/m-...

But the suggestions from that thread do not work.

[the_rock]

What do you get when you run ip r g 8.8.8.8?

Andy

[AlekzNet]

# ip r g 8.8.8.8
8.8.8.8 via 10.1.9.113 dev eth1.121 src 10.1.9.115
cache

10.1.9.113 is the default gateway (it's the "Switch" in this flow: eth1.121 -> Switch -> eth2)

# ip r
default via 10.1.9.113 dev eth1.121 proto 7

[Timothy_Hall]

fw ctl set int fw_local_interface_anti_spoofing 0

fw ctl set int fw_antispoofing_enabled 0

fw  ctl  set  int  sim_anti_spoofing_enabled  0  -a

SecureXL enforces anti-spoofing as well, and the first two commands do not disable it.  The third command will, see A Primer on Anti-Spoofing  and sk117618 that has this command as a easily-missed footnote.

[AlekzNet]

Yes, tried this one already - it did not work:

# fw ctl set int sim_anti_spoofing_enabled 0 -a
PPAK 0: Get before set operation succeeded of sim_anti_spoofing_enabled
kiss_params: failed to update VS 0

# fw ctl get int sim_anti_spoofing_enabled -a
FW:
Get operation failed: failed to get parameter sim_anti_spoofing_enabled
PPAK 0: sim_anti_spoofing_enabled = 0

Still see "local anti-spoofing" drops

[AlekzNet]

Two months ago I logged a case for the CP Diamond and then to ATAM teams -no results either.

[AlekzNet]

Interestingly enough, R81.20 is not mentioned in either sk117618 or sk105899. Is the documentation not up to date, or is there is something very different in R81.20?

[AlekzNet]

Things are getting more strange.

It does not matter whether sim_anti_spoofing_enabled is set to - 0 or 1, with fwaccel on or off. But when I set fw_local_interface_anti_spoofing = 0 and installed the policy, the result changed.

Quick topology reminder.

"Internet"
    |
br1 (eth3)
   FW - eth1.121 -+
br1 (eth2)        |
    |             |
    +-- Switch ---+    

In other words, the configuration is the same, but instead of "local anti-spoofing" I'm getting this (a kind of asymmetrical routing?):

#fw ctl zdebug + drop | fgrep 1.1.1.

@@;381561397.18118721;[vs_0];[tid_2];[fw4_2];fw_log_drop_ex: Packet proto=1 1.1.1.1:0 -> 10.1.9.115:22318 dropped by fw_first_packet_state_checks Reason: ICMP reply does not match a previous request

#fw monitor -F "0,0,1.1.1.1,0,0" -F "1.1.1.1,0,0,0

[vs_0][fw_2] eth1.121:o[44]: 10.1.9.115 -> 1.1.1.1 (ICMP) len=84 id=5630
ICMP: type=8 code=0 echo request id=313 seq=5
[vs_0][fw_2] eth1.121:O[44]: 10.1.9.114 -> 1.1.1.1 (ICMP) len=84 id=5630
ICMP: type=8 code=0 echo request id=43556 seq=5
[vs_0][ppak_0] eth2:i[44]: 10.1.9.114 -> 1.1.1.1 (ICMP) len=84 id=5630
ICMP: type=8 code=0 echo request id=43556 seq=5
[vs_0][fw_0] eth2:i[44]: 10.1.9.114 -> 1.1.1.1 (ICMP) len=84 id=5630
ICMP: type=8 code=0 echo request id=43556 seq=5
[vs_0][fw_0] eth2:I[44]: 10.1.9.114 -> 1.1.1.1 (ICMP) len=84 id=5630
ICMP: type=8 code=0 echo request id=43556 seq=5
[vs_0][fw_0] eth3:o[44]: 10.1.9.114 -> 1.1.1.1 (ICMP) len=84 id=5630
ICMP: type=8 code=0 echo request id=43556 seq=5
[vs_0][fw_0] eth3:O[44]: 10.1.9.114 -> 1.1.1.1 (ICMP) len=84 id=5630
ICMP: type=8 code=0 echo request id=43556 seq=5
[vs_0][ppak_0] eth3:i[44]: 1.1.1.1 -> 10.1.9.114 (ICMP) len=84 id=37813
ICMP: type=0 code=0 echo reply id=43556 seq=5
[vs_0][fw_2] eth3:i[44]: 1.1.1.1 -> 10.1.9.114 (ICMP) len=84 id=37813
ICMP: type=0 code=0 echo reply id=43556 seq=5
[vs_0][fw_2] eth3:I[44]: 1.1.1.1 -> 10.1.9.115 (ICMP) len=84 id=37813
ICMP: type=0 code=0 echo reply id=313 seq=5
[vs_0][fw_2] eth2:o[44]: 1.1.1.1 -> 10.1.9.115 (ICMP) len=84 id=37813
ICMP: type=0 code=0 echo reply id=313 seq=5

So, somehow,  I've got past the local anti-spoofing, but now the return packets are getting blocked on the internal interface of the bridge in the outgoing ("southward") direction.

FWIW, the current setting:

# fw ctl get int sim_anti_spoofing_enabled -a
FW:
Get operation failed: failed to get parameter sim_anti_spoofing_enabled
PPAK 0: sim_anti_spoofing_enabled = 1

[AlekzNet]

OK, at least it's repeatable now. In order to disable local anti-spoofing "live", the policy must be installed:

1. fw ctl set int fw_local_interface_anti_spoofing 0
2. Install the policy

[AlekzNet]

Next. While ping is still not working (dropped by "fw_first_packet_state_checks Reason: ICMP reply does not match a previous request;" -- see above). DNS IS working:

# fw ctl zdebug + drop | fgrep 1.1.1.1

[vs_0][fw_0] eth1.121:o[44]: 10.1.9.115 -> 1.1.1.1 (UDP) len=51 id=47855
UDP: 42759 -> 53
[vs_0][fw_0] eth1.121:O[44]: 10.1.9.114 -> 1.1.1.1 (UDP) len=51 id=47855
UDP: 21694 -> 53
[vs_0][ppak_0] eth2:i[44]: 10.1.9.114 -> 1.1.1.1 (UDP) len=51 id=47855
UDP: 21694 -> 53
[vs_0][fw_1] eth2:i[44]: 10.1.9.114 -> 1.1.1.1 (UDP) len=51 id=47855
UDP: 21694 -> 53
[vs_0][fw_1] eth2:I[44]: 10.1.9.114 -> 1.1.1.1 (UDP) len=51 id=47855
UDP: 21694 -> 53
[vs_0][fw_1] eth3:o[44]: 10.1.9.114 -> 1.1.1.1 (UDP) len=51 id=47855
UDP: 21694 -> 53
[vs_0][fw_1] eth3:O[44]: 10.1.9.114 -> 1.1.1.1 (UDP) len=51 id=47855
UDP: 21694 -> 53
[vs_0][ppak_0] eth3:i[44]: 1.1.1.1 -> 10.1.9.114 (UDP) len=67 id=7847
UDP: 53 -> 21694
[vs_0][fw_0] eth3:i[44]: 1.1.1.1 -> 10.1.9.114 (UDP) len=67 id=7847
UDP: 53 -> 21694
[vs_0][fw_0] eth3:I[44]: 1.1.1.1 -> 10.1.9.115 (UDP) len=67 id=7847
UDP: 53 -> 42759
[vs_0][fw_0] eth2:o[44]: 1.1.1.1 -> 10.1.9.115 (UDP) len=67 id=7847
UDP: 53 -> 42759
[vs_0][fw_0] eth2:O[44]: 1.1.1.1 -> 10.1.9.115 (UDP) len=67 id=7847
UDP: 53 -> 42759
[vs_0][ppak_0] eth1.121:i[44]: 1.1.1.1 -> 10.1.9.115 (UDP) len=67 id=7847
UDP: 53 -> 42759
[vs_0][fw_0] eth1.121:i[44]: 1.1.1.1 -> 10.1.9.115 (UDP) len=67 id=7847
UDP: 53 -> 42759

Settings:

fw_local_interface_anti_spoofing = 0
fw_antispoofing_enabled = 1
fwx_bridge_reroute_enabled = 0
fwx_perform_gateway_hide = 1
sim_anti_spoofing_enabled = 0

[AlekzNet]

More strange things. The management station is located on the "Internet" side of the firewall.

fw ctl set int fwx_bridge_reroute_enabled 1

- Passive gateway is not pingable from the management station, Active is pingable
- The mgmt station is pingable from the active, 1.1.1.1 is not. Nothing is pingable from the passive.

fw ctl set int fwx_bridge_reroute_enabled 0

- Both passive and active are pingable from the mgmt
- Neither mgmt, nor 1.1.1.1 is pingable from either active or passive

It does not matter what fwx_perform_gateway_hide is set to: 0, 1 or 2. And it does not matter if the policy is installed afterwards or not (unlike fw_local_interface_anti_spoofing).

I'm lost ...

[AlekzNet]

Apparently, managing ClusterXL through the bridge interface is not supported in R81.20 (just like IP-addresses on the bridges). Only a single gateway can be managed through the bridge. And this is what fwx_bridge_reroute_enabled is for. (Passive FW is not accessible when fwx_bridge_reroute_enabled=1).

That would explain why I'm getting VERY strange and unstable results.

Shouldn't it be listed in the "Limitations in Bridge Mode" documentation section? Or, better, can it be properly implemented?

OK, time well spent ... 😕

[PhoneBoy]

With ElasticXL in R82, only the active cluster member (SMO) needs to be accessible to push policy.
Which means this should work properly in this scenario.

[AlekzNet]

First of all - Merry Christmas everybody! 🎄Regardless if you celebrate it or not - may it be a peaceful, healthy and happy year for us all!

ElasticXL is definitely a good idea, and definitely might help,  provided it's implemented correctly.

Speaking of management over "clustered bridge". This topology is described in, for example,  R77 (https://sc1.checkpoint.com/documents/R77/CP_R77_SecurityGatewayTech_WebAdmin/96332.htm), and in R81.20 (https://sc1.checkpoint.com/documents/R81.20/WebAdminGuides/EN/CP_R81.20_Installation_and_Upgrade_Gui...)

It's not a new concept. In both documents "a security gateway" is mentioned, not "a cluster". At the same time, it's nowhere stated, that this topology is not supported by ClusterXL.

Please correct me if I'm wrong, but it resembles a typical situation, when a project is officially completed, but some features are either not tested, or not implemented correctly and not working. But since the project is "completed and paid successfully", these "features" are simply not mentioned anywhere: in either supported or unsupported sections 😊

My cumulative 30+ hour testing rather confirms this "theory". Also, it appeared, that the documentation  is either absent, or not 100% correct, or not up-to-date (e.g. R81.20 is not in the list of related OSes).

Examples:

https://support.checkpoint.com/results/sk/sk105899

- R81.20 is not in the version list
- fwx_perform_gateway_hide does not change anything related
- fw_local_interface_anti_spoofing - not mentioned that the policy must be installed
- fwx_bridge_reroute_enabled - not mentioned that it's not applicable for ClusterXL (it breaks it)
- sim_anti_spoofing_enabled - does not change anything related
- fw_antispoofing_enabled - does not change anything related

https://sc1.checkpoint.com/documents/R81.20/WebAdminGuides/EN/CP_R81.20_Installation_and_Upgrade_Gui...

- it does not matter if the bridge interfaces or subordinates are in the Topology, it does not change anything
- it's not mentioned what topology should be applied to the management interface: Internet or According to the routing table. Even if the default route goes through the management interface, the result is different (even if all traffic is allowed by the FW)

https://sc1.checkpoint.com/documents/R81.20/WebAdminGuides/EN/CP_R81.20_Installation_and_Upgrade_Gui...

- "A Security Gateway cannot filter or transmit packets that it inspected before on a Bridge interface (to avoid double-inspection)." - and next doco section describes how to configure it (isn't what fwx_bridge_reroute_enabled is for?)

https://sc1.checkpoint.com/documents/R81.20/WebAdminGuides/EN/CP_R81.20_Installation_and_Upgrade_Gui...

- "Assigning an IP address to a Bridge interface in ClusterXL." - OK, while I do agree, that assigning IP-addresses to the bridge interfaces on CusterXL breaks cluster synchronization (the bridge subordinates on the active firewall begin to flap - what looks like a bug, rather than a "feature"), I managed to configure addresses on the bridges and keep the cluster in sync (OK, it was not easy, and the config was not 100% symmetrical). What again hints on a "non fully completed project" 😉

As a workaround, I tried MDPS, but it's even more buggy and even less documented - the Sync interface (while up and passing the sync traffic) was not visible on the dplane, so the cluster sync was broken. I did not waste time on the troubleshooting and removed MDPS for the time being.

 ... and I can keep going on and on ...

What I achieved so far ( and more indications of a non-finished project 😉

- the cluster is in full sync
- both members are reachable rom the management station
- both members are pingable from everywhere
- both members send logs to the management station
- both members can access DNS everywhere

What is not working:

- the cluster members can not PING anything on the North side
- the cluster members can not reach the CheckPoint site for updates or use RAD (possibly can be solved by adding a proxy on the South side of the FW).

In both cases, the return traffic is blocked on the "internal" bridge subordinate - see above

Anyway, the quest continues!

[AlekzNet]

OK, if management over "clustered bridge" is not fixed by CP, but you have to use it, here's some tips:

- use a clustered interface for the management
- disable Extended Cluster Anti-Spoofing
- disable anti-spoofing for the related interface
- set fw_local_interface_anti_spoofing to 0 and install the policy (if you can)
- you might have to "fw unloadlocal" on the passive fw, but disable the "South" bridge sub-interface to prevent possible spanning-tree loop (yes, this thing also works incorrectly)
- to get access to the primary firewall you might have to temporary set fwx_bridge_reroute_enabled to 1
- you might have to install the policy several times in stages (e.g. on the active only, then on the passive, then on the cluster)

Another solution would be to use a single firewall with IP-address configured on the bridge (and manage it through this address) and have a spare firewall on a cold standby,

Any other ideas? Please let me know. On my side, I will try to chase the Diamond and the ATAM teams.

[AlekzNet]

Sooooo.... After another round of troubleshooting...

It turned out, that for TCP and ICMP, no connection table entries are created for the IP-address of the physical firewall (only for the cluster IP).
For UDP such entries are properly created.

Example.

f1 - cluster member address
c1 - cluster address
h1 - host on the "Internet"

If you run "fw tab -t connections -z | fgrep h1", and try to connect from f1 to h1, you will get the following (simplified! and without port/proto numbers):

For UDP

c1 h1
f1 h1
h1 c1
h1 f1

And everything works.

For ICMP/TCP

c1 h1
f1 h1
h1 c1

And the return traffic gets dropped.

Looks like a bug in my optic. The CP case is updated, of course.