VSX on VMware and ACI R81.10 Take 66 virtual Switch, ARP first x replys seems to get ignored

[Felix_Burs]

Hi all,

currently we're facing some strange behavior in one of our Checkpoint environments, We have a VSX install on VMware with ACI. 

For one vlan in a virtual switch we have a strange behavior for ping and arp.

Pings in the local vlan look like this:

[Expert@Firewall:3]# ping 172.26.60.222
PING 172.26.60.222 (172.26.60.222) 56(84) bytes of data.
64 bytes from 172.26.60.222: icmp_seq=21 ttl=64 time=1607 ms
64 bytes from 172.26.60.222: icmp_seq=22 ttl=64 time=607 ms
64 bytes from 172.26.60.222: icmp_seq=23 ttl=64 time=4.97 ms
64 bytes from 172.26.60.222: icmp_seq=24 ttl=64 time=5.22 ms
64 bytes from 172.26.60.222: icmp_seq=25 ttl=64 time=5.12 ms
64 bytes from 172.26.60.222: icmp_seq=26 ttl=64 time=5.04 ms
64 bytes from 172.26.60.222: icmp_seq=27 ttl=64 time=5.40 ms
64 bytes from 172.26.60.222: icmp_seq=28 ttl=64 time=5.12 ms
^C
--- 172.26.60.222 ping statistics ---
28 packets transmitted, 8 received, 71% packet loss, time 27004ms
rtt min/avg/max/mdev = 4.977/280.735/1607.496/538.836 ms, pipe 2

as the ping works fine, it will work for a few mins but if you wait for a few mins and then try again you will face the same issue

[Expert@Firewall:3]# ping 172.26.60.222
PING 172.26.60.222 (172.26.60.222) 56(84) bytes of data.
64 bytes from 172.26.60.222: icmp_seq=41 ttl=64 time=2262 ms
64 bytes from 172.26.60.222: icmp_seq=43 ttl=64 time=262 ms
64 bytes from 172.26.60.222: icmp_seq=42 ttl=64 time=1262 ms
64 bytes from 172.26.60.222: icmp_seq=44 ttl=64 time=5.15 ms
64 bytes from 172.26.60.222: icmp_seq=45 ttl=64 time=5.10 ms
64 bytes from 172.26.60.222: icmp_seq=46 ttl=64 time=5.36 ms
64 bytes from 172.26.60.222: icmp_seq=47 ttl=64 time=5.17 ms
64 bytes from 172.26.60.222: icmp_seq=48 ttl=64 time=5.24 ms
^C
--- 172.26.60.222 ping statistics ---
48 packets transmitted, 8 received, 83% packet loss, time 47005ms
rtt min/avg/max/mdev = 5.109/476.805/2262.779/787.822 ms, pipe 3

I also got one where the first replies were in the wrong order but i guess this is due to the differend reply times

[Expert@Firewall:3]# ping 172.26.60.222
PING 172.26.60.222 (172.26.60.222) 56(84) bytes of data.
64 bytes from 172.26.60.222: icmp_seq=11 ttl=64 time=929 ms
64 bytes from 172.26.60.222: icmp_seq=10 ttl=64 time=1929 ms
64 bytes from 172.26.60.222: icmp_seq=9 ttl=64 time=2929 ms
64 bytes from 172.26.60.222: icmp_seq=12 ttl=64 time=5.11 ms
64 bytes from 172.26.60.222: icmp_seq=13 ttl=64 time=5.19 ms
64 bytes from 172.26.60.222: icmp_seq=14 ttl=64 time=5.01 ms
64 bytes from 172.26.60.222: icmp_seq=15 ttl=64 time=5.18 ms
64 bytes from 172.26.60.222: icmp_seq=16 ttl=64 time=5.17 ms
^C
--- 172.26.60.222 ping statistics ---
16 packets transmitted, 8 received, 50% packet loss, time 15003ms
rtt min/avg/max/mdev = 5.015/726.724/2929.358/1057.264 ms, pipe 3

If you do a tcp dump you will see that there are a lot of arp requests and replys but they seem to be ingnored for the first xx times and then somewhen start working:

Ping

[Expert@Firewall:3]# ping 172.26.60.193
PING 172.26.60.193 (172.26.60.193) 56(84) bytes of data.
64 bytes from 172.26.60.193: icmp_seq=26 ttl=64 time=107 ms
64 bytes from 172.26.60.193: icmp_seq=25 ttl=64 time=1107 ms
64 bytes from 172.26.60.193: icmp_seq=27 ttl=64 time=0.825 ms
64 bytes from 172.26.60.193: icmp_seq=28 ttl=64 time=0.858 ms
64 bytes from 172.26.60.193: icmp_seq=29 ttl=64 time=0.958 ms
^C
--- 172.26.60.193 ping statistics ---
29 packets transmitted, 5 received, 82% packet loss, time 28001ms
rtt min/avg/max/mdev = 0.825/243.623/1107.759/434.044 ms, pipe 2

And the tcpdump

Click to Expand

[Expert@Firewall:1]# tcpdump -i bond1.2375 -vvv -s 0 host 172.26.60.193
tcpdump: listening on bond1.2375, link-type EN10MB (Ethernet), capture size 262144 bytes
09:38:10.665343 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.26.60.193 tell 172.26.60.220, length 28
09:38:10.665401 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.26.60.193 tell 172.26.60.220, length 46
09:38:10.665507 ARP, Ethernet (len 6), IPv4 (len 4), Reply 172.26.60.193 is-at b4:0c:25:e0:40:12 (oui Unknown), length 46
09:38:11.666644 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.26.60.193 tell 172.26.60.220, length 28
09:38:11.666826 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.26.60.193 tell 172.26.60.220, length 46
09:38:11.666987 ARP, Ethernet (len 6), IPv4 (len 4), Reply 172.26.60.193 is-at b4:0c:25:e0:40:12 (oui Unknown), length 46
09:38:12.668645 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.26.60.193 tell 172.26.60.220, length 28
09:38:12.668733 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.26.60.193 tell 172.26.60.220, length 46
09:38:12.668888 ARP, Ethernet (len 6), IPv4 (len 4), Reply 172.26.60.193 is-at b4:0c:25:e0:40:12 (oui Unknown), length 46
09:38:14.664678 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.26.60.193 tell 172.26.60.220, length 28
09:38:14.664816 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.26.60.193 tell 172.26.60.220, length 46
09:38:14.664926 ARP, Ethernet (len 6), IPv4 (len 4), Reply 172.26.60.193 is-at b4:0c:25:e0:40:12 (oui Unknown), length 46
09:38:15.666646 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.26.60.193 tell 172.26.60.220, length 28
09:38:15.666720 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.26.60.193 tell 172.26.60.220, length 46
09:38:15.666915 ARP, Ethernet (len 6), IPv4 (len 4), Reply 172.26.60.193 is-at b4:0c:25:e0:40:12 (oui Unknown), length 46
09:38:16.668651 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.26.60.193 tell 172.26.60.220, length 28
09:38:16.668715 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.26.60.193 tell 172.26.60.220, length 46
09:38:16.668814 ARP, Ethernet (len 6), IPv4 (len 4), Reply 172.26.60.193 is-at b4:0c:25:e0:40:12 (oui Unknown), length 46
09:38:18.664716 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.26.60.193 tell 172.26.60.220, length 28
09:38:18.664945 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.26.60.193 tell 172.26.60.220, length 46
09:38:18.665030 ARP, Ethernet (len 6), IPv4 (len 4), Reply 172.26.60.193 is-at b4:0c:25:e0:40:12 (oui Unknown), length 46
09:38:19.666627 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.26.60.193 tell 172.26.60.220, length 28
09:38:19.666689 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.26.60.193 tell 172.26.60.220, length 46
09:38:19.666837 ARP, Ethernet (len 6), IPv4 (len 4), Reply 172.26.60.193 is-at b4:0c:25:e0:40:12 (oui Unknown), length 46
09:38:20.668723 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.26.60.193 tell 172.26.60.220, length 28
09:38:20.668838 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.26.60.193 tell 172.26.60.220, length 46
09:38:20.669003 ARP, Ethernet (len 6), IPv4 (len 4), Reply 172.26.60.193 is-at b4:0c:25:e0:40:12 (oui Unknown), length 46
09:38:22.664719 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.26.60.193 tell 172.26.60.220, length 28
09:38:22.664957 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.26.60.193 tell 172.26.60.220, length 46
09:38:22.665023 ARP, Ethernet (len 6), IPv4 (len 4), Reply 172.26.60.193 is-at b4:0c:25:e0:40:12 (oui Unknown), length 46
09:38:23.666635 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.26.60.193 tell 172.26.60.220, length 28
09:38:23.666851 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.26.60.193 tell 172.26.60.220, length 46
09:38:23.666861 ARP, Ethernet (len 6), IPv4 (len 4), Reply 172.26.60.193 is-at b4:0c:25:e0:40:12 (oui Unknown), length 46
09:38:24.668653 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.26.60.193 tell 172.26.60.220, length 28
09:38:24.668862 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.26.60.193 tell 172.26.60.220, length 46
09:38:24.668870 ARP, Ethernet (len 6), IPv4 (len 4), Reply 172.26.60.193 is-at b4:0c:25:e0:40:12 (oui Unknown), length 46
09:38:26.664678 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.26.60.193 tell 172.26.60.220, length 28
09:38:26.664734 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.26.60.193 tell 172.26.60.220, length 46
09:38:26.665210 ARP, Ethernet (len 6), IPv4 (len 4), Reply 172.26.60.193 is-at b4:0c:25:e0:40:12 (oui Unknown), length 46
09:38:27.666649 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.26.60.193 tell 172.26.60.220, length 28
09:38:27.666701 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.26.60.193 tell 172.26.60.220, length 46
09:38:27.666922 ARP, Ethernet (len 6), IPv4 (len 4), Reply 172.26.60.193 is-at b4:0c:25:e0:40:12 (oui Unknown), length 46
09:38:28.668641 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.26.60.193 tell 172.26.60.220, length 28
09:38:28.668804 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.26.60.193 tell 172.26.60.220, length 46
09:38:28.668816 ARP, Ethernet (len 6), IPv4 (len 4), Reply 172.26.60.193 is-at b4:0c:25:e0:40:12 (oui Unknown), length 46
09:38:30.664697 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.26.60.193 tell 172.26.60.220, length 28
09:38:30.664966 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.26.60.193 tell 172.26.60.220, length 46
09:38:30.664975 ARP, Ethernet (len 6), IPv4 (len 4), Reply 172.26.60.193 is-at b4:0c:25:e0:40:12 (oui Unknown), length 46
09:38:31.666644 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.26.60.193 tell 172.26.60.220, length 28
09:38:31.666889 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.26.60.193 tell 172.26.60.220, length 46
09:38:31.666903 ARP, Ethernet (len 6), IPv4 (len 4), Reply 172.26.60.193 is-at b4:0c:25:e0:40:12 (oui Unknown), length 46
09:38:32.668666 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.26.60.193 tell 172.26.60.220, length 28
09:38:32.668716 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.26.60.193 tell 172.26.60.220, length 46
09:38:32.668873 ARP, Ethernet (len 6), IPv4 (len 4), Reply 172.26.60.193 is-at b4:0c:25:e0:40:12 (oui Unknown), length 46
09:38:34.664699 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.26.60.193 tell 172.26.60.220, length 28
09:38:34.664838 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.26.60.193 tell 172.26.60.220, length 46
09:38:34.664937 ARP, Ethernet (len 6), IPv4 (len 4), Reply 172.26.60.193 is-at b4:0c:25:e0:40:12 (oui Unknown), length 46
09:38:35.666644 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.26.60.193 tell 172.26.60.220, length 28
09:38:35.666848 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.26.60.193 tell 172.26.60.220, length 46
09:38:35.666917 ARP, Ethernet (len 6), IPv4 (len 4), Reply 172.26.60.193 is-at b4:0c:25:e0:40:12 (oui Unknown), length 46
09:38:35.771198 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.26.60.214 (Broadcast) tell 172.26.60.193, length 46
09:38:35.771214 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.26.60.214 (Broadcast) tell 172.26.60.193, length 46
09:38:35.771339 IP (tos 0x0, ttl 64, id 27013, offset 0, flags [DF], proto ICMP (1), length 84)
172.26.60.220 > 172.26.60.193: ICMP echo request, id 10466, seq 25, length 64
09:38:35.771343 IP (tos 0x0, ttl 64, id 27894, offset 0, flags [DF], proto ICMP (1), length 84)
172.26.60.220 > 172.26.60.193: ICMP echo request, id 10466, seq 26, length 64
09:38:35.772285 IP (tos 0x0, ttl 64, id 50960, offset 0, flags [none], proto ICMP (1), length 84)
172.26.60.193 > 172.26.60.220: ICMP echo reply, id 10466, seq 26, length 64
09:38:35.772310 IP (tos 0x0, ttl 64, id 50961, offset 0, flags [none], proto ICMP (1), length 84)
172.26.60.193 > 172.26.60.220: ICMP echo reply, id 10466, seq 25, length 64
09:38:36.666424 IP (tos 0x0, ttl 64, id 28484, offset 0, flags [DF], proto ICMP (1), length 84)
172.26.60.220 > 172.26.60.193: ICMP echo request, id 10466, seq 27, length 64
09:38:36.667166 IP (tos 0x0, ttl 64, id 51105, offset 0, flags [none], proto ICMP (1), length 84)
172.26.60.193 > 172.26.60.220: ICMP echo reply, id 10466, seq 27, length 64
09:38:37.666602 IP (tos 0x0, ttl 64, id 28843, offset 0, flags [DF], proto ICMP (1), length 84)
172.26.60.220 > 172.26.60.193: ICMP echo request, id 10466, seq 28, length 64
09:38:37.667411 IP (tos 0x0, ttl 64, id 51274, offset 0, flags [none], proto ICMP (1), length 84)
172.26.60.193 > 172.26.60.220: ICMP echo reply, id 10466, seq 28, length 64
09:38:38.666610 IP (tos 0x0, ttl 64, id 29408, offset 0, flags [DF], proto ICMP (1), length 84)
172.26.60.220 > 172.26.60.193: ICMP echo request, id 10466, seq 29, length 64
09:38:38.667511 IP (tos 0x0, ttl 64, id 52115, offset 0, flags [none], proto ICMP (1), length 84)
172.26.60.193 > 172.26.60.220: ICMP echo reply, id 10466, seq 29, length 64
09:38:50.287621 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.26.60.193 (Broadcast) tell 172.26.60.193, length 46
09:38:50.287631 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.26.60.193 (Broadcast) tell 172.26.60.193, length 46

We've already tried the solution described in sk94564 and here https://community.checkpoint.com/t5/Cloud-Network-Security/Incompatibility-between-CISCO-ACI-and-VSX... but both did not work for us.

The fact that we have a second vlan in a second virtual switch with a similar setup and everything works fine makes it a lot more confusing.

Any further ideas or hints to test?

[PhoneBoy]

What version/JHF?
I recommend opening a TAC case in parallel.