Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
Dor_Marcovitch
Advisor

VSX arp debuging

Jump to solution

hey,

i have 2 VSs on the VSX which are sharing the same external subnet (connected to the same vSwitch)

in addition each VS has it's own public subnet, meaning the upstream router has route to network X to VS1 and route to network Y to VS2.

i have internal client on VS1 that i hide behind ip from subnet X. this client fails to access the internet. i can see only icmp request , without reply.

when i hide it behind the VS1 ip it has internet.

further checks revealed that the upstream router has a lot of logs of arp flapping for the public ip of VS1.

when i have checked those MAC address it seams that they match to the wrp interface of VS1 and VS2 which faces the upstream router, which is weird.

checking tcpdump on VS2 for arp packets, i could see that for some reason VS2 sends arp reply for VS1 public ip with it's public interface MAC address..

this situation can be caused by

  1. VS2 has interface with this IP
  2. NAT is configured on VS2 with this IP
  3. local.arp was configured manually with this IP and MAC

non of the above is configured on this environment.

  1. anyone has direction?
  2. which service is responsible for ARP?
  3. how do i debug the FW to get the reason for it?

i am using R80.10

thanks

0 Kudos
Reply
1 Solution

Accepted Solutions
Kaspars_Zibarts
Authority
Authority

Obviously this gratuitous ARP below has wrong IP on wrp257 / VS4, should have been .132

02:10:28.276365 00:CC:c1:AA:BB:CC > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: arp who-has 1.1.1.130 tell 1.1.1.130

Are you in a position to delete this interface (since it does not work anyways)? Then just check with tcpdump that this G-ARP above is gone, and then re-create interface.

View solution in original post

0 Kudos
Reply
11 Replies
Kaspars_Zibarts
Authority
Authority

The only thing that comes to my mind is local.arp (static ARP) on VSX that could do such thing (or a bug?)

If you search for all local.arp files, i.e. from expert shell

find / -name local.arp

you definitely don't see anything for CTX2?

/var/opt/CPsuite-R80/fw1/CTX/CTX00002/conf/local.arp

And GAIA config (check instance 2 of course)

grep ip:arp:entry:static:instance /config/active

I trust it cannot be interface as you said that subnets are different nor "automatic ARP for manual NAT in R80.10" is used (Automatic creation of Proxy ARP for Manual NAT rules on Security Gateway R80.10 )

0 Kudos
Reply
Dor_Marcovitch
Advisor

no results for "find / -name local.arp"

no results for "grep ip:arp:entry:static:instance /config/active"

no file "/var/opt/CPsuite-R80/fw1/CTX/CTX00002/conf/local.arp"

0 Kudos
Reply
Kaspars_Zibarts
Authority
Authority

any chances you could provide interface info for both VS and all four interfaces so we can have full picture? 

Also tcpdump with -e option to see MACs. 

I understand if you want to replace public IPs with dummy Smiley Happy as long as subnets are the same length

0 Kudos
Reply
Dor_Marcovitch
Advisor

this should be good

02:10:28.276365 00:CC:c1:AA:BB:CC > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: arp who-has 1.1.1.130 tell 1.1.1.130
02:10:28.276582 00:CC:c1:AA:BB:DD > 00:CC:c1:AA:BB:CC, ethertype ARP (0x0806), length 42: arp reply 1.1.1.130 is-at 00:CC:c1:AA:BB:DD <<<< VS3 MAC
02:11:02.565CC9 00:CC:c1:AA:BB:DD > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: arp who-has 1.1.1.130 tell 1.1.1.130
02:11:02.565186 00:CC:c1:AA:BB:CC > 00:CC:c1:AA:BB:DD, ethertype ARP (0x0806), length 42: arp reply 1.1.1.130 is-at 00:CC:c1:AA:BB:CC <<<< VS4 MAC


wrp193 Link encap:Ethernet HWaddr 00:CC:C1:AA:BB:DD
inet addr:1.1.1.130 Bcast:1.1.1.143 Mask:255.255.255.240
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1134008 errors:0 dropped:0 overruns:0 frame:0
TX packets:33559CC errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:75381018 (71.8 MiB) TX bytes:320265716 (305.4 MiB)


wrp257 Link encap:Ethernet HWaddr 00:CC:C1:AA:BB:CC
inet addr:1.1.1.132 Bcast:1.1.1.143 Mask:255.255.255.240
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1140005 errors:0 dropped:0 overruns:0 frame:0
TX packets:451113 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:75194768 (71.7 MiB) TX bytes:58062241 (55.3 MiB)

0 Kudos
Reply
Kaspars_Zibarts
Authority
Authority

Obviously this gratuitous ARP below has wrong IP on wrp257 / VS4, should have been .132

02:10:28.276365 00:CC:c1:AA:BB:CC > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: arp who-has 1.1.1.130 tell 1.1.1.130

Are you in a position to delete this interface (since it does not work anyways)? Then just check with tcpdump that this G-ARP above is gone, and then re-create interface.

View solution in original post

0 Kudos
Reply
Dor_Marcovitch
Advisor

re-adding this interface on VS4 stopped this wrong arp reply for now, i will go ahead and check with the upstream router for why we dont see the reply of the ping.

thanks for the help

Kaspars_Zibarts
Authority
Authority

Ouch, not even funny. But hey, one step forward! Smiley Happy

0 Kudos
Reply
Dor_Marcovitch
Advisor

the Fat Lady did not sing yet... those arp came back.

0 Kudos
Reply
Kaspars_Zibarts
Authority
Authority

Hmm - did it appear after topology or policy push? Did you notice? Did you see correct GARPs at some point? Feels like something configured in the policy..  

0 Kudos
Reply
Maarten_Sjouw
Champion
Champion

Keep in mind that on a Cisco router the default ARP cache is 4 hours.

That said, you can force a gratuitous ARP by using this command:

# arping -c 4 -A -I eth1 1.1.1.130

with this command you can see which ARP's are created for NAT:

fw ctl arp

You can also use ifconfig in each VS to see what the MAC should be on that interface.

Regards, Maarten
0 Kudos
Reply
Dor_Marcovitch
Advisor

this command helped me "fw ctl arp" locate the reason it came back after i re-added the switch or something else.

thanks,

dor

0 Kudos
Reply