VSX arp debuging

[Dor_Marcovitch]

hey,

i have 2 VSs on the VSX which are sharing the same external subnet (connected to the same vSwitch)

in addition each VS has it's own public subnet, meaning the upstream router has route to network X to VS1 and route to network Y to VS2.

i have internal client on VS1 that i hide behind ip from subnet X. this client fails to access the internet. i can see only icmp request , without reply.

when i hide it behind the VS1 ip it has internet.

further checks revealed that the upstream router has a lot of logs of arp flapping for the public ip of VS1.

when i have checked those MAC address it seams that they match to the wrp interface of VS1 and VS2 which faces the upstream router, which is weird.

checking tcpdump on VS2 for arp packets, i could see that for some reason VS2 sends arp reply for VS1 public ip with it's public interface MAC address..

this situation can be caused by

1. VS2 has interface with this IP
2. NAT is configured on VS2 with this IP
3. local.arp was configured manually with this IP and MAC

non of the above is configured on this environment.

1. anyone has direction?
2. which service is responsible for ARP?
3. how do i debug the FW to get the reason for it?

i am using R80.10

thanks

[Kaspars_Zibarts]

The only thing that comes to my mind is local.arp (static ARP) on VSX that could do such thing (or a bug?)

If you search for all local.arp files, i.e. from expert shell

find / -name local.arp

you definitely don't see anything for CTX2?

/var/opt/CPsuite-R80/fw1/CTX/CTX00002/conf/local.arp

And GAIA config (check instance 2 of course)

grep ip:arp:entry:static:instance /config/active

I trust it cannot be interface as you said that subnets are different nor "automatic ARP for manual NAT in R80.10" is used (Automatic creation of Proxy ARP for Manual NAT rules on Security Gateway R80.10 )

[Dor_Marcovitch]

no results for "find / -name local.arp"

no results for "grep ip:arp:entry:static:instance /config/active"

no file "/var/opt/CPsuite-R80/fw1/CTX/CTX00002/conf/local.arp"

[Kaspars_Zibarts]

any chances you could provide interface info for both VS and all four interfaces so we can have full picture? 

Also tcpdump with -e option to see MACs. 

I understand if you want to replace public IPs with dummy  as long as subnets are the same length

[Dor_Marcovitch]

this should be good

02:10:28.276365 00:CC:c1:AA:BB:CC > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: arp who-has 1.1.1.130 tell 1.1.1.130
02:10:28.276582 00:CC:c1:AA:BB:DD > 00:CC:c1:AA:BB:CC, ethertype ARP (0x0806), length 42: arp reply 1.1.1.130 is-at 00:CC:c1:AA:BB:DD <<<< VS3 MAC
02:11:02.565CC9 00:CC:c1:AA:BB:DD > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: arp who-has 1.1.1.130 tell 1.1.1.130
02:11:02.565186 00:CC:c1:AA:BB:CC > 00:CC:c1:AA:BB:DD, ethertype ARP (0x0806), length 42: arp reply 1.1.1.130 is-at 00:CC:c1:AA:BB:CC <<<< VS4 MAC

wrp193 Link encap:Ethernet HWaddr 00:CC:C1:AA:BB:DD
inet addr:1.1.1.130 Bcast:1.1.1.143 Mask:255.255.255.240
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1134008 errors:0 dropped:0 overruns:0 frame:0
TX packets:33559CC errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:75381018 (71.8 MiB) TX bytes:320265716 (305.4 MiB)

wrp257 Link encap:Ethernet HWaddr 00:CC:C1:AA:BB:CC
inet addr:1.1.1.132 Bcast:1.1.1.143 Mask:255.255.255.240
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1140005 errors:0 dropped:0 overruns:0 frame:0
TX packets:451113 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:75194768 (71.7 MiB) TX bytes:58062241 (55.3 MiB)

[Kaspars_Zibarts · Accepted solution]

Obviously this gratuitous ARP below has wrong IP on wrp257 / VS4, should have been .132

02:10:28.276365 00:CC:c1:AA:BB:CC > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: arp who-has 1.1.1.130 tell 1.1.1.130

Are you in a position to delete this interface (since it does not work anyways)? Then just check with tcpdump that this G-ARP above is gone, and then re-create interface.

[Dor_Marcovitch]

re-adding this interface on VS4 stopped this wrong arp reply for now, i will go ahead and check with the upstream router for why we dont see the reply of the ping.

thanks for the help

[Kaspars_Zibarts]

Ouch, not even funny. But hey, one step forward!

[Dor_Marcovitch]

the Fat Lady did not sing yet... those arp came back.

[Kaspars_Zibarts]

Hmm - did it appear after topology or policy push? Did you notice? Did you see correct GARPs at some point? Feels like something configured in the policy..  

[Maarten_Sjouw]

Keep in mind that on a Cisco router the default ARP cache is 4 hours.

That said, you can force a gratuitous ARP by using this command:

# arping -c 4 -A -I eth1 1.1.1.130

with this command you can see which ARP's are created for NAT:

fw ctl arp

You can also use ifconfig in each VS to see what the MAC should be on that interface.

[Dor_Marcovitch]

this command helped me "fw ctl arp" locate the reason it came back after i re-added the switch or something else.

thanks,

dor