topic Re: VRRP Issue - Interface stuck in Init State in General Topics

VRRP Issue - Interface stuck in Init State

Malik1 — Mon, 01 May 2023 13:51:42 GMT

Hi Experts,

We upgraded RAM in one of our 4800 cluster , post-firewall reboot we noticed one of the interafce on standby member is in VRRP initialize state.

We do see the arp entry of the active member on the standby member of that interface but don't see the arp entry of the standby member on the active member for the interface in the init state.

We have done the following workaround to mitigate this:

We noticed that the virtual IP was changed automatically to the physical IP) which was later corrected.
Hotfix upgraded
Manual ARP entry added on active member
Anti Spoofing disabled/enabled on the cluster
Firewall rebooted
Removing/adding interface from VRRP cluster.
Tried changing the Switch port which is connected to Stanby member interface.
The network team confirmed they are receiving Mac address for the standby interface on the switch end.

Further Analysis.

Stanby member interface eth1-01.xxxx is in vrrp initialize state however on active memebr interface eth1-01.xxxx is in master state.
We unable to ping the active member interface from standby firewall and vice versa .
Able to receive the arp entries on stanby member however we are not receiving arp entries from standby on active member
All other sub-interfaces are working perfectly fine.
Ideally, the route for the interface on standby firewall should be directly connected(same as on active member and other interfaces ) however the best route is default route.
As per ASP drop logs, active member is dropping packets for stnaby member due to local anti spoofing . the main reason is active memeber has no entry in its arp table.
As per tcpdump logs on active member when we are trying to ping the remote end IP , its is getting ICMP echo message but it is not replying.

Is there something we are missing and what can bee done to resolve the issue. ?

Regards

Sijeel

Re: VRRP Issue - Interface stuck in Init State

the_rock — Mon, 01 May 2023 14:10:11 GMT

Hi Sijeel,

Just so we can get better idea, can you please send below outputs from BOTH members?

Andy

cphaprob state

cphaprob -a if

cphaprob list

cphaprob syncstat

cphaprob roles

cphaprob tablestat

Andy

Re: VRRP Issue - Interface stuck in Init State

Malik1 — Mon, 01 May 2023 17:42:48 GMT

Hi Andy,

Please find the output. I have also added output for below command

show vrrp summary

show vrrp interface eth1-01.1491 ( Problematic interface on standby)

show route destination 163.166.149.35 (on standby )

show route destination 163.166.149.36 (on active )

show route direct

Regards,

Sijeel

Re: VRRP Issue - Interface stuck in Init State

the_rock — Mon, 01 May 2023 17:50:30 GMT

I have a call in 10 mins, but I quickly reviewed show interface command for problematic one and vmac and IDs looks correct to me. This happened AFTER upgrade you said?

Re: VRRP Issue - Interface stuck in Init State

the_rock — Mon, 01 May 2023 18:01:12 GMT

Can you also please send topology config, SPECIFICALLY for problematic interface? Please blur out any sensitive data.

Andy

Re: VRRP Issue - Interface stuck in Init State

Malik1 — Tue, 02 May 2023 09:06:28 GMT

Please find the topology

Re: VRRP Issue - Interface stuck in Init State

the_rock — Tue, 02 May 2023 11:43:35 GMT

Thanks, but I more meant topology properties ONLY for interface you have issue with, if you double click on it. I think it would also be worth to contact TAC, so they can do remote session and see whats going on.

Re: VRRP Issue - Interface stuck in Init State

Malik1 — Tue, 02 May 2023 12:41:58 GMT

Hi .

Please find the details , it been 2 months my team has raised a case with TAC but they arent able to identify the issue,

Do u have any idea what can be an issue as the connected route is not showing in the routing table? what could be a reason for such an issue,The link status is up for the interface

Re: VRRP Issue - Interface stuck in Init State

Chris_Atkinson — Tue, 02 May 2023 13:04:13 GMT

What type of NIC card is populated in the expansion slot?

Anything odd in /var/log/messages ?

Can you please confirm the version & jumbo that this gateway/cluster is installed with?

(Note 4800 appliances are no longer supported).

Re: VRRP Issue - Interface stuck in Init State

the_rock — Tue, 02 May 2023 13:03:36 GMT

Im really not sure, as looking at your screenshots, all seems correct to me. If TAC case has been ongoing for 2 months, did they at least escalate it? Sounds way too long for an issue like this...if connected route is missing, tells me something with the interface is wrong, since its not static route, cant be added manually. What steps did TAC provide you so far?

Andy

Re: VRRP Issue - Interface stuck in Init State

Malik1 — Tue, 02 May 2023 14:00:11 GMT

Hi Chris,

R80.40 +take 180 is installed . I dont see any odd messages also, the rest of the logical interfaces on the physical port are working fine. the issue is with 1 logical interface.

Line card model : CPAP-4-1F Type 4 ports 1 gbe sfp

Re: VRRP Issue - Interface stuck in Init State

Malik1 — Tue, 02 May 2023 14:09:07 GMT

the rest of the sub interfaces are fine the issue is with this interafce. if this was ab issue with the physical interface then we would have faced the issue with all the sub-interfaces

.TAC has taken dumps , asked to update HF . Below steps have been done till now.

We have noticed that virtual IP 163.166.149.33 was changed automatically to 163.166.149.36(physical IP) which was later corrected .
Hotfix upgraded
Manual ARP entry added on NOKFW35
Anti Spoofing disabled/enabled on cluster
Firewall rebooted
Removing/adding interface eth1-01.1491 from VRRP cluster.
Tried with changing Switch port which is connected with NOKFW36 eth1-01 interface .
Network team confirmed they are receiving mac address for vlan 1491 on the switch end.

As per zdebug NOKFW35 is dropping packets for 163.166.149.36 due to local local spoofing (screenshot attached)

Re: VRRP Issue - Interface stuck in Init State

the_rock — Tue, 02 May 2023 14:14:34 GMT

Anti-Spoofing drops indicate that the source IP address of the packet received on a certain interface is not a part of the defined interface's topology.
Local interface address spoofing drops indicate that the Security Gateway / Cluster member received a packet with a source IP address that belongs to one of the local interfaces on the Security Gateway / Cluster member.

https://support.checkpoint.com/results/sk/sk115276

Re: VRRP Issue - Interface stuck in Init State

Chris_Atkinson — Wed, 03 May 2023 03:31:11 GMT

Note Take 196 is the current recommended release and includes the following VRRP fixes for awareness.

Re: VRRP Issue - Interface stuck in Init State

the_rock — Wed, 03 May 2023 13:24:03 GMT

Hey @Malik1 ,

I think, as always, @Chris_Atkinson brings up a good point. To add to what he said, I would strongly consider also upgrade to R81.10, jumbo 94, if you can, as its super stable.

Not sure if TAC made that recommendation yet.

Andy