VRRP Issue - Interface stuck in Init State

[Malik1]

Hi Experts,

We upgraded RAM in one of our  4800 cluster , post-firewall reboot we noticed one of the interafce on standby member is in VRRP initialize state.

We do see the arp entry of the active member on the standby member of that interface but don't see the arp entry of the standby member on the active member for the interface in the init state.

We have done the following workaround to mitigate this:

• We noticed that the virtual IP was changed automatically to the physical IP) which was later corrected.
• Hotfix upgraded
• Manual ARP entry added on active member 
• Anti Spoofing disabled/enabled on the cluster
• Firewall rebooted
• Removing/adding interface from VRRP cluster.
• Tried changing the Switch port which is connected to Stanby member interface.
• The network team confirmed they are receiving Mac address for the standby interface on the switch end.

Further Analysis.

• Stanby member  interface eth1-01.xxxx is in vrrp initialize state however on active memebr  interface eth1-01.xxxx is in master state.
• We unable to ping the active member interface from standby firewall and vice versa .
• Able to receive the arp entries on stanby member  however we are not receiving arp entries from standby on active member 
• All other sub-interfaces are working perfectly fine.
• Ideally, the route for the interface on  standby firewall should be directly connected(same as on active member and other interfaces ) however the best route is default route.
• As per ASP drop logs, active member is dropping packets for stnaby member  due to local anti spoofing . the main reason is active memeber  has no entry in its arp table.
• As per tcpdump logs on active member when we are trying to ping the remote end IP , its is getting ICMP echo message but it is not replying.

Is there something we are missing and what can bee done to resolve the issue. ?

Regards

Sijeel 

[the_rock]

Hi Sijeel,

Just so we can get better idea, can you please send below outputs from BOTH members?

Andy

cphaprob state

cphaprob -a if

cphaprob list

cphaprob syncstat

cphaprob roles

cphaprob tablestat

Andy

[Malik1]

Hi Andy, 

Please find the output. I have also added output for below command

show vrrp summary

show vrrp interface eth1-01.1491 ( Problematic interface on standby)

show route destination 163.166.149.35 (on standby )

show route destination 163.166.149.36 (on active )

show route direct 

Regards,

Sijeel

[the_rock]

I have a call in 10 mins, but I quickly reviewed show interface command for problematic one and vmac and IDs looks correct to me. This happened AFTER upgrade you said? 

[the_rock]

Can you also please send topology config, SPECIFICALLY for problematic interface? Please blur out any sensitive data.

Andy

[Malik1]

 Please find the topology 

[the_rock]

Thanks, but I more meant topology properties ONLY for interface you have issue with, if you double click on it. I think it would also be worth to contact TAC, so they can do remote session and see whats going on.

[Malik1]

Hi .

Please find the details , it been 2 months my team has raised a case with TAC but they arent able to identify the issue,

Do u have any idea what can be an issue as the connected route is not showing in the routing table? what could be a reason for such an issue,The link status is up for the interface 

[the_rock]

Im really not sure, as looking at your screenshots, all seems correct to me. If TAC case has been ongoing for 2 months, did they at least escalate it? Sounds way too long for an issue like this...if connected route is missing, tells me something with the interface is wrong, since its not static route, cant be added manually. What steps did TAC provide you so far?

Andy

[Malik1]

the rest of the sub  interfaces are fine the issue is with this interafce. if this was ab issue with the physical interface then we would have faced the issue with all the sub-interfaces 

.TAC has taken dumps , asked to update HF . Below steps have been done till now. 

• We have noticed that virtual IP 163.166.149.33 was changed automatically to 163.166.149.36(physical IP) which was later corrected .
• Hotfix upgraded
• Manual ARP entry added on NOKFW35
• Anti Spoofing disabled/enabled on cluster
• Firewall rebooted
• Removing/adding interface eth1-01.1491 from VRRP cluster.
• Tried with changing Switch port which is connected with NOKFW36 eth1-01 interface .
• Network team confirmed they are receiving mac address for vlan 1491 on the switch end.

As per zdebug NOKFW35 is dropping packets for 163.166.149.36 due to local local  spoofing (screenshot attached)

[the_rock]

• Anti-Spoofing drops indicate that the source IP address of the packet received on a certain interface is not a part of the defined interface's topology.

• Local interface address spoofing drops indicate that the Security Gateway / Cluster member received a packet with a source IP address that belongs to one of the local interfaces on the Security Gateway / Cluster member.

https://support.checkpoint.com/results/sk/sk115276

[Chris_Atkinson]

What type of NIC card is populated in the expansion slot?

Anything odd in /var/log/messages ?

Can you please confirm the version & jumbo that this gateway/cluster is installed with?

(Note 4800 appliances are no longer supported).

[Malik1]

Hi Chris, 

R80.40 +take 180 is installed . I dont see any odd messages also, the rest of the logical  interfaces on the physical port are working fine. the issue is with 1 logical interface.

Line card model : CPAP-4-1F Type 4 ports 1 gbe sfp

[Chris_Atkinson]

Note Take 196 is the current recommended release and includes the following VRRP fixes for awareness.

[the_rock]

Hey @Malik1 ,

I think, as always, @Chris_Atkinson brings up a good point. To add to what he said, I would strongly consider also upgrade to R81.10, jumbo 94, if you can, as its super stable.

Not sure if TAC made that recommendation yet.

Andy