gateways failing until policies are pushed

[kehagen]

Hello,

I have r77 gateways 4400.  Two of them are in a cluster.  We lost connectivity to the inside last night. I pushed the policies and then I could get connectivity.  Then same thing happened this morning.  I looked at the switch and a mac address is flapping on the inside with the mac of the firewall cluster.  I can get to the inside, but it seems to be a recurring problem and not sure what is causing it.  Any ideas where to look?  Thanks.

Ken

[the_rock]

Hey Kenny,

Sounds like could be routing issue. Maybe do zdebug when this happens if you have console, since it sounds like ssh fails when issue is there. PLEASE upgrade, no one in TAC will even bother with this, its long time unsupported version.

Andy

[kehagen]

Thanks Andy.  I will try that.  Yes we’re trying to upgrade asap.  

[the_rock]

Yes sir, you should! Now I will go about my weekend birthday celebration...one year older, man, nothing to celebrate 🤣🤣

Cheers,

Andy

[kehagen]

Happy birthday! and thanks for your suggestion.  

ken

[the_rock]

Thank you! Have a nice weekend.

Andy

[Timothy_Hall]

Sounds like an ARP issue to me, as a policy installation will force a gratuitous ARP for all firewall and NAT addresses if the cluster object is not set to use VMAC (which is the default behavior).  Next time you have an outage, check the ARP caches of the surrounding routers, are they losing the IP to MAC mapping for the firewall and/or NAT addresses?  Command fw ctl arp might be helpful to diagnose.  If it is found to be an ARP issue, you can try setting VMAC on the cluster, reinstall policy twice and see if it helps.

[kehagen]

Thank you Timothy I will try that today. I go onsite to troubleshoot today. 

[the_rock]

Hey Kenny,

Let us know how it goes. I see what Tim is saying, if arp is not there, it will never work, sort of goes without saying. Mind you, that coommand would show you if there are ny proxy arp entries.

Andy

[the_rock]

You can also run just arp, as I did below in my lab. So in my case, 172.16.10.1 is our lab Fortigate.

Andy

[Expert@CP-STANDALONE-backup:0]# arp
Address HWtype HWaddress Flags Mask Iface
172.16.10.233 ether 50:06:00:07:00:00 C eth0
172.16.10.126 ether 00:0c:29:27:56:d6 C eth0
172.16.10.1 ether e8:1c:ba:4e:89:87 C eth0
[Expert@CP-STANDALONE-backup:0]#

[kehagen]

When doing arp on the problem gateways, I only get 2 arps, one for management and one for the cluster interface.  When comparing to a known good gateway, there are many more arps for all the devices behind the firewall.

[kehagen]

the arps look ok on the inside.  

on the firewalls i get:

[Expert@gto-fw-1:0]# fw ctl arp
No proxy ARP entries
[Expert@gto-fw-1:0]#

I checked the vmac and it is already applied.

[the_rock]

Can you just run arp?

Andy

[kehagen]

yes, i did run arp, i only see 2 arps.  mgmt and cluster.  for some reason it is not getting all the other arps from the devices inside.  could it be a certificate or license problem?  the cert is good until 12/26/23, so that is next thing to do after i fix this.

[the_rock]

Apologies mate, missed your first response, my bad. Long day troubleshooting Cisco/Fortigate vpn issue lol. Anyway, so here is my suggestion...can you verify that routes are similar? Just type route from expert mode and compare.

Kind regards,

Andy

[kehagen]

yes the routes are the same.

[the_rock]

Got it. Any luck so far or still same issue?

Andy

[kehagen]

Same issue.  Also I tried to renew the cert and got an error message.