topic Re: VSX arp debuging in Firewall and Security Management

VSX arp debuging

Dor_Marcovitch — Tue, 29 May 2018 18:56:43 GMT

hey,

i have 2 VSs on the VSX which are sharing the same external subnet (connected to the same vSwitch)

in addition each VS has it's own public subnet, meaning the upstream router has route to network X to VS1 and route to network Y to VS2.

i have internal client on VS1 that i hide behind ip from subnet X. this client fails to access the internet. i can see only icmp request , without reply.

when i hide it behind the VS1 ip it has internet.

further checks revealed that the upstream router has a lot of logs of arp flapping for the public ip of VS1.

when i have checked those MAC address it seams that they match to the wrp interface of VS1 and VS2 which faces the upstream router, which is weird.

checking tcpdump on VS2 for arp packets, i could see that for some reason VS2 sends arp reply for VS1 public ip with it's public interface MAC address..

this situation can be caused by

VS2 has interface with this IP
NAT is configured on VS2 with this IP
local.arp was configured manually with this IP and MAC

non of the above is configured on this environment.

anyone has direction?
which service is responsible for ARP?
how do i debug the FW to get the reason for it?

i am using R80.10

thanks

Re: VSX arp debuging

Kaspars_Zibarts — Wed, 30 May 2018 06:34:49 GMT

The only thing that comes to my mind is local.arp (static ARP) on VSX that could do such thing (or a bug?)

If you search for all local.arp files, i.e. from expert shell

find / -name local.arp

you definitely don't see anything for CTX2?

/var/opt/CPsuite-R80/fw1/CTX/CTX00002/conf/local.arp

And GAIA config (check instance 2 of course)

grep ip:arp:entry:static:instance /config/active

I trust it cannot be interface as you said that subnets are different nor "automatic ARP for manual NAT in R80.10" is used (Automatic creation of Proxy ARP for Manual NAT rules on Security Gateway R80.10 )

Re: VSX arp debuging

Dor_Marcovitch — Wed, 30 May 2018 07:26:00 GMT

no results for "find / -name local.arp"

no results for "grep ip:arp:entry:static:instance /config/active"

no file "/var/opt/CPsuite-R80/fw1/CTX/CTX00002/conf/local.arp"

Re: VSX arp debuging

Kaspars_Zibarts — Wed, 30 May 2018 07:45:07 GMT

any chances you could provide interface info for both VS and all four interfaces so we can have full picture?

Also tcpdump with -e option to see MACs.

I understand if you want to replace public IPs with dummy as long as subnets are the same length

Re: VSX arp debuging

Dor_Marcovitch — Wed, 30 May 2018 08:15:46 GMT

this should be good

02:10:28.276365 00:CC:c1:AA:BB:CC > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: arp who-has 1.1.1.130 tell 1.1.1.130
02:10:28.276582 00:CC:c1:AA:BB:DD > 00:CC:c1:AA:BB:CC, ethertype ARP (0x0806), length 42: arp reply 1.1.1.130 is-at 00:CC:c1:AA:BB:DD <<<< VS3 MAC
02:11:02.565CC9 00:CC:c1:AA:BB:DD > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: arp who-has 1.1.1.130 tell 1.1.1.130
02:11:02.565186 00:CC:c1:AA:BB:CC > 00:CC:c1:AA:BB:DD, ethertype ARP (0x0806), length 42: arp reply 1.1.1.130 is-at 00:CC:c1:AA:BB:CC <<<< VS4 MAC

wrp193 Link encap:Ethernet HWaddr 00:CC:C1:AA:BB:DD
inet addr:1.1.1.130 Bcast:1.1.1.143 Mask:255.255.255.240
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1134008 errors:0 dropped:0 overruns:0 frame:0
TX packets:33559CC errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:75381018 (71.8 MiB) TX bytes:320265716 (305.4 MiB)

wrp257 Link encap:Ethernet HWaddr 00:CC:C1:AA:BB:CC
inet addr:1.1.1.132 Bcast:1.1.1.143 Mask:255.255.255.240
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1140005 errors:0 dropped:0 overruns:0 frame:0
TX packets:451113 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:75194768 (71.7 MiB) TX bytes:58062241 (55.3 MiB)

Re: VSX arp debuging

Kaspars_Zibarts — Wed, 30 May 2018 08:43:41 GMT

Obviously this gratuitous ARP below has wrong IP on wrp257 / VS4, should have been .132

02:10:28.276365 00:CC:c1:AA:BB:CC > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: arp who-has 1.1.1.130 tell 1.1.1.130

Are you in a position to delete this interface (since it does not work anyways)? Then just check with tcpdump that this G-ARP above is gone, and then re-create interface.

Re: VSX arp debuging

Dor_Marcovitch — Wed, 30 May 2018 09:29:55 GMT

re-adding this interface on VS4 stopped this wrong arp reply for now, i will go ahead and check with the upstream router for why we dont see the reply of the ping.

thanks for the help

Re: VSX arp debuging

Kaspars_Zibarts — Wed, 30 May 2018 10:01:09 GMT

Ouch, not even funny. But hey, one step forward!

Re: VSX arp debuging

Dor_Marcovitch — Wed, 30 May 2018 13:33:32 GMT

the Fat Lady did not sing yet... those arp came back.

Re: VSX arp debuging

Kaspars_Zibarts — Wed, 30 May 2018 14:40:13 GMT

Hmm - did it appear after topology or policy push? Did you notice? Did you see correct GARPs at some point? Feels like something configured in the policy..

Re: VSX arp debuging

Maarten_Sjouw — Wed, 30 May 2018 20:39:51 GMT

Keep in mind that on a Cisco router the default ARP cache is 4 hours.

That said, you can force a gratuitous ARP by using this command:

# arping -c 4 -A -I eth1 1.1.1.130

with this command you can see which ARP's are created for NAT:

fw ctl arp

You can also use ifconfig in each VS to see what the MAC should be on that interface.

Re: VSX arp debuging

Dor_Marcovitch — Wed, 30 May 2018 22:39:13 GMT

this command helped me "fw ctl arp" locate the reason it came back after i re-added the switch or something else.

thanks,

dor