ClusterXL 80.40 HF 78 - Passive unit is using the sync interface to send management packets

[osef]

Hello everyone,

English is not my first langage, I will try to do my best.

I'm currently facing a strange issue with the passive unit of my clusterXL.

This unit is using is sync interface (bond0) instead of is LAN interface (bond1.994) to send packets on the network...

I will use a ssh connexion from the passive unit to the server 10.3.12.237 to illustrate the problem 

1) configuration of bond0 (sync) and bond1.994 (LAN)

20: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
link/ether 00:1c:7f:6a:e7:03 brd ff:ff:ff:ff:ff:ff
inet 192.168.230.10/29 brd 192.168.230.15 scope global bond0
valid_lft forever preferred_lft forever

24: bond1.994@bond1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
link/ether 00:1c:7f:6a:e7:00 brd ff:ff:ff:ff:ff:ff
inet 192.168.127.53/29 brd 192.168.127.55 scope global bond1.994
valid_lft forever preferred_lft forever

2) routing table from the passive unit to reach 10.3.12.237 : 

[Expert@FW-EXT-B:0]# ip route | grep 10.3
10.3.0.0/16 via 192.168.127.49 dev bond1.994 proto 7
10.3.11.19 via 192.168.127.49 dev bond1.994 proto 7
10.30.1.0/24 via 192.168.127.49 dev bond1.994 proto 7
10.30.2.0/24 via 192.168.127.49 dev bond1.994 proto 7
10.30.3.0/24 via 192.168.127.49 dev bond1.994 proto 7
10.30.4.0/24 via 192.168.127.49 dev bond1.994 proto 7
10.30.5.0/24 via 192.168.127.49 dev bond1.994 proto 7

[Expert@FW-EXT-B:0]# ip route get 10.3.12.237
10.3.12.237 via 192.168.127.49 dev bond1.994 src 192.168.127.53
cache

So the appliance need to use bond1.994 and the source address 192.168.127.53 to reach the server.

But if I try to ping the server 10.3.12.237, the echo-request is send on bond0 with the source IP of bond1.994...

[Expert@FW-EXT-B:0]# tcpdump -enni bond0 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on bond0, link-type EN10MB (Ethernet), capture size 262144 bytes
11:38:15.397306 02:00:00:00:00:01 > 00:1c:7f:6a:e7:1b, ethertype IPv4 (0x0800), length 98: 192.168.127.53 > 10.3.12.237: ICMP echo request, id 52348, seq 1, length 64
11:38:16.397865 02:00:00:00:00:01 > 00:1c:7f:6a:e7:1b, ethertype IPv4 (0x0800), length 98: 192.168.127.53 > 10.3.12.237: ICMP echo request, id 52348, seq 2, length 64
11:38:17.397766 02:00:00:00:00:01 > 00:1c:7f:6a:e7:1b, ethertype IPv4 (0x0800), length 98: 192.168.127.53 > 10.3.12.237: ICMP echo request, id 52348, seq 3, length 64
11:38:18.397684 02:00:00:00:00:01 > 00:1c:7f:6a:e7:1b, ethertype IPv4 (0x0800), length 98: 192.168.127.53 > 10.3.12.237: ICMP echo request, id 52348, seq 4, length 64

The destination mac address is the mac address of the sync interface on the active unit :

[Expert@FW-EXT-A:0]# ip -s l | grep -B 1 1b
5: eth1-04: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc noqueue master bond0 state UP mode DEFAULT qlen 1000
link/ether 00:1c:7f:6a:e7:1b brd ff:ff:ff:ff:ff:ff
--
9: eth2-04: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc noqueue master bond0 state UP mode DEFAULT qlen 1000
link/ether 00:1c:7f:6a:e7:1b brd ff:ff:ff:ff:ff:ff
--
20: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT qlen 1000
link/ether 00:1c:7f:6a:e7:1b brd ff:ff:ff:ff:ff:ff

The echo-reply, are going to be received on bond1.994 (which is normal since 192.168.127.53 belong to vlan 994, my core-switch send the reply packets to the right interface)

[Expert@FW-EXT-B:0]# tcpdump -enni bond1.994 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on bond1.994, link-type EN10MB (Ethernet), capture size 262144 bytes
11:39:39.210851 00:08:e3:ff:fd:90 > 00:1c:7f:6a:e7:00, ethertype IPv4 (0x0800), length 98: 10.3.12.237 > 192.168.127.53: ICMP echo reply, id 20093, seq 1, length 64
11:39:40.210972 00:08:e3:ff:fd:90 > 00:1c:7f:6a:e7:00, ethertype IPv4 (0x0800), length 98: 10.3.12.237 > 192.168.127.53: ICMP echo reply, id 20093, seq 2, length 64
11:39:41.211072 00:08:e3:ff:fd:90 > 00:1c:7f:6a:e7:00, ethertype IPv4 (0x0800), length 98: 10.3.12.237 > 192.168.127.53: ICMP echo reply, id 20093, seq 3, length 64
11:39:42.211134 00:08:e3:ff:fd:90 > 00:1c:7f:6a:e7:00, ethertype IPv4 (0x0800), length 98: 10.3.12.237 > 192.168.127.53: ICMP echo reply, id 20093, seq 4, length 64

Do you know why the passive unit is doing this ? The active unit has the exact same routing table and is correctly using bond1.994 to reach the LAN... 

We are going to update to JH 87 in two days but I want to be sure it's not a configuration issue...

[PhoneBoy]

If you’re still seeing this after you upgrade it’s probably worth a TAC case.

[osef]

Hello, the upgrade didn't solve the issue 😞

I will open a ticket, thanks

[Maarten_Sjouw]

What does the: show management interface 
command show? Is it the interface that you configured in the gateway object and how is routing setup? is the traffic to management routed over the Sync interface?

[osef]

Hello,

The management interface is : 

FW-EXT-A> show management interface
bond1.994

What do you mean by "Is it the interface that you configured in the gateway object" ?

We are doing OSPF between the clusterXL and the core-switch. There is no traffic routed over the sync interface in the routing table :

[Expert@FW-EXT-A:0]# ip route | grep bond0
192.168.230.8/29 dev bond0 proto kernel scope link src 192.168.230.9

The routing of the gateway is fine... It's like the system is ignoring the routing table. We have a ticket with TAC now 🙂

Thanks !

[Timothy_Hall]

Hmm this could be an issue with the IP route cache, see this thread:

https://community.checkpoint.com/t5/Enterprise-Appliances-and-Gaia/route-flipping-on-R80-40/m-p/8620...

[osef]

I read the topic and I don't think it's the same issue.

In my case, the routing table always show bond1.994 as the output interface. I tried the "ip route cache flush" command but it didn't change anything 😞

We did a failover and when the unit B is the active one, the problem disappear... I don't know yet if the same problem appear on the unit A when it is in standby...

Not related to the problem, I bought your book and it's great !

[osef · Accepted solution]

Ok, it's new since 80.40...

https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...