I've looked some more on this issue.
First off, the issue is reproducible on
- R77.20.80 (990172392)
- R77.20.85 (990172755)
- R77.20.86 (990172855)
- R77.20.87 (990173004)
Second, I found out more specifically how to trigger the behaviour. I noticed that the 790 was actually passing traffic in certain situations.
Reproducing drop scenario
The following findings are reproduced on build 392. Some of the steps look slightly different on other builds (for example, I didn't test the entire boot up sequence on other builds), but the general behaviour with packets getting dropped is reproducible on all four tested builds.
- Host 1 and host 2 are connected to SW1
- Host 1 is moved to LAN4 of 790, which is off
- When the 790 is booted with cables inserted in the bridged setup (A), the br0 is passing traffic. There comes a "hole" in forwarding during bootup, which I assume is because the 790 defaults with interfaces in the integrated switch, and when CP processes get started, the two specified interfaces get disabled/then moved to br0. Anyway - things work fine! It keeps on running like this for the entire duration of this step - I haven't tested it extensively though, as I was interested in the next step.
- Disconnecting LAN3 from SW1, and reconnecting again to SW1 (both immediately after and also with a delay of 15 min), and br0 still forwards traffic.
- With LAN3 connected to SW1, host 1 is disconnected from LAN4, and connected to another port on SW1. Then, host 1 is moved back to LAN4. From now on, the forwarding behaviour of br0 is as observed in the original post. That is, br0 forwards traffic from LAN4 (host 1) to LAN3 (SW1/host 2), but traffic from host 2 is not seen any more (with some exception, see further below).
- This behaviour can be replicated to further LAN ports joined to br0. So joining LAN5 and LAN6 to br0, moving SW1 from LAN3 to LAN5, and return traffic is received on LAN5. Move host 1 to SW1 and back to LAN3, and return traffic to LAN5 is killed off once more.
- In some rare cases, it's possible to have br0 receive and forward 1-5 return packets(!), before dropping traffic again. Unplugging LAN3 from SW1, waiting for it to go down, then re-connecting it. It's reproducible, but hard to do - as other packets received first will then be allowed, and the ARP reply and ICMP replies will then be dropped. But it *is* reproducible. I have only been successful doing so on build 392 though.
I set up a SPAN port on SW1, and confirmed that the ARP and ICMP replies were sent out from SW1 to the 790.
Temporary restoration of forwarding
Here is an example of the last step above
tcpdump -n -i br0 arp or icmp
...
21:35:29.108554 ARP, Request who-has 192.168.1.8 tell 192.168.1.54, length 46
21:35:30.132596 ARP, Request who-has 192.168.1.8 tell 192.168.1.54, length 46
21:35:31.156585 ARP, Request who-has 192.168.1.8 tell 192.168.1.54, length 46
21:35:32.180577 ARP, Request who-has 192.168.1.8 tell 192.168.1.54, length 46
21:35:33.204571 ARP, Request who-has 192.168.1.8 tell 192.168.1.54, length 46
21:35:34.228514 ARP, Request who-has 192.168.1.8 tell 192.168.1.54, length 46
21:35:34.228789 ARP, Reply 192.168.1.8 is-at 28:92:4a:38:32:74, length 46
21:35:34.229616 IP 192.168.1.54 > 192.168.1.8: ICMP echo request, id 2512, seq 46, length 64
21:35:34.229676 IP 192.168.1.54 > 192.168.1.8: ICMP echo request, id 2512, seq 47, length 64
21:35:34.229946 IP 192.168.1.8 > 192.168.1.54: ICMP echo reply, id 2512, seq 46, length 64
21:35:34.229993 IP 192.168.1.8 > 192.168.1.54: ICMP echo reply, id 2512, seq 47, length 64
21:35:35.230736 IP 192.168.1.54 > 192.168.1.8: ICMP echo request, id 2512, seq 48, length 64
21:35:35.231071 IP 192.168.1.8 > 192.168.1.54: ICMP echo reply, id 2512, seq 48, length 64
21:35:36.244910 IP 192.168.1.54 > 192.168.1.8: ICMP echo request, id 2512, seq 49, length 64
21:35:37.268842 IP 192.168.1.54 > 192.168.1.8: ICMP echo request, id 2512, seq 50, length 64
21:35:38.292854 IP 192.168.1.54 > 192.168.1.8: ICMP echo request, id 2512, seq 51, length 64
21:35:39.316810 IP 192.168.1.54 > 192.168.1.8: ICMP echo request, id 2512, seq 52, length 64
21:35:40.340907 IP 192.168.1.54 > 192.168.1.8: ICMP echo request, id 2512, seq 53, length 64
21:35:41.364814 IP 192.168.1.54 > 192.168.1.8: ICMP echo request, id 2512, seq 54, length 64
And with fw monitor
[vs_0][fw_0] LAN4:i[84]: 192.168.1.54 -> 192.168.1.8 (ICMP) len=84 id=9355
ICMP: type=8 code=0 echo request id=3661 seq=22
[vs_0][fw_0] LAN4:I[84]: 192.168.1.54 -> 192.168.1.8 (ICMP) len=84 id=9355
ICMP: type=8 code=0 echo request id=3661 seq=22
[vs_0][fw_0] LAN3:o[84]: 192.168.1.54 -> 192.168.1.8 (ICMP) len=84 id=9355
ICMP: type=8 code=0 echo request id=3661 seq=22
[vs_0][fw_0] LAN3:O[84]: 192.168.1.54 -> 192.168.1.8 (ICMP) len=84 id=9355
ICMP: type=8 code=0 echo request id=3661 seq=22
[vs_0][fw_0] LAN3:i[84]: 192.168.1.8 -> 192.168.1.54 (ICMP) len=84 id=5489
ICMP: type=0 code=0 echo reply id=3661 seq=22
[vs_0][fw_0] LAN3:I[84]: 192.168.1.8 -> 192.168.1.54 (ICMP) len=84 id=5489
ICMP: type=0 code=0 echo reply id=3661 seq=22
[vs_0][fw_0] LAN4:o[84]: 192.168.1.8 -> 192.168.1.54 (ICMP) len=84 id=5489
ICMP: type=0 code=0 echo reply id=3661 seq=22
[vs_0][fw_0] LAN4:O[84]: 192.168.1.8 -> 192.168.1.54 (ICMP) len=84 id=5489
ICMP: type=0 code=0 echo reply id=3661 seq=22
I thought that since I now had ICMP through too (now host 1 had an ARP entry of host 2), I could maybe see the drop reason - but unfortunately fw ctl zdebug +drop did not report any drops.
Looking closer on the drops
I thought that maybe some kind of cache/filter was put into place, so I started looking at how the bridge module was configured.
[Expert@fw01]# pwd
/proc/sys/net
[Expert@fw01]# tail bridge/*
==> bridge/bridge-nf-call-arptables <==
1
==> bridge/bridge-nf-call-ip6tables <==
1
==> bridge/bridge-nf-call-iptables <==
1
==> bridge/bridge-nf-filter-pppoe-tagged <==
0
==> bridge/bridge-nf-filter-vlan-tagged <==
0
==> bridge/bridge-nf-pass-vlan-input-dev <==
0
==> bridge/forwarding <==
1
It looked like br0 should send packets to netfilter-arptables and iptables (and per default also ebtables). Disabling these forwardings with
for f in bridge-nf-*; do echo 0 > $f; done
didn't have any effect though.
So I thought that maybe some rules were getting programmed in ebtables or iptables? If that is still active on a check point kernel... I had my doubts. But kernel config said something might be enabled? Not sure if it's enough though...
[Expert@fw01]# zcat /proc/config.gz | grep -i 'bridge_.*y$'
CONFIG_BRIDGE_NETFILTER=y
CONFIG_BRIDGE_IGMP_SNOOPING=y
[Expert@fw01]# zcat /proc/config.gz | grep -i 'netfilter.*y$'
CONFIG_NETFILTER=y
CONFIG_NETFILTER_ADVANCED=y
CONFIG_BRIDGE_NETFILTER=y
I succesfully placed some ebtables and iptables binaries and libs on the 790, but all of them reported kernel modules missing.
[Expert@fw01]# ./sbin/ebtables -t broute -L
modprobe: module ebtables not found
modprobe: failed to load module ebtables
The kernel doesn't support the ebtables 'broute' table.
[Expert@fw01]# ./sbin/ebtables -L
modprobe: module ebtables not found
modprobe: failed to load module ebtables
The kernel doesn't support the ebtables 'filter' table.
[Expert@fw01]# ./sbin/iptables -L -n
modprobe: module ip_tables not found
modprobe: failed to load module ip_tables
iptables v1.4.21: can't initialize iptables table `filter': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
I'm thinking maybe the CP kernel fw module is doing something similar to ebtables? But I have no idea how to inspect this.
The closest thing I found online to this, was this post on serverfault (Linux bridge (brctl) is dropping packets) but since normal ebtables isn't available, I'm out of luck to inspect this.
Is there a diagram similar to this one (packet flow in netfilter) for the SMB devices running embedded Gaia?
