Cluster Upgrade 77.30 to 80.20 with traffic handling problems

[Nikolaos_Tsitso]

Hi @all,

yesterday we have try to upgrade a cluster from 77.30 to 80.20.

The connectivity upgrade works fine without any problems. After the upgrade the web servers behind the cluster was not reachable from the Iinternet.

On the tcpdump we can see that the traffic can reach the firewall, but on fw monitor we cannot see any traffic that is handled by the firewall.

Also we don't see any drops in the fw ctl zdebug + drop.

We have also try to change the nat rules to automatic but the problem still exists.

We have revert to prior version 77.30 and everything works again fine.

Has anyone a idea?

[PhoneBoy]

Providing exactly what you saw on tcpdump and fw monitor might be helpful in answering your question.

[Nikolaos_Tsitso]

hi, i saw only the syn packets on the tcpdump output.
on the fw monitor output was nothing

[Maarten_Sjouw]

This problem was identified a couple of weeks ago already, all your inbound NAT is not working.
Before you do the upgrade, type 'fw ctl arp' and you will see all your NAT's will have a MAC address. When you are done with the upgrade do it again, you will see nothing.
Push policy 3 times in total to get them to show up again.
Problem exists in R80.20 and R80.30

[Nikolaos_Tsitso]

i see the mac address after the upgrade when i run 'fw ctl arp'
maybe it is a inbound NAT problem and also with the take 91 the problem still exists

[FedericoMeiners]

This happened to us two times in different customers of different kind of traffics, we solved both time by installing JHF Take 87 or 91.

Also solution by Maarten is valid for Proxy ARP, just be sure to add or modify a Proxy ARP entry before pushing policies.

Regards,

[Nemz]

I just experienced this nightmare last night while trying to move to R80.10 from R77.30.

I created a new R80.10 deployment (non cluster) in my test lab and converted my R77.30 production policy over to it.  I had this R80.10 deployment in my test lab for weeks.  I moved the drives over to production and pushed many policies..everything seemed to work until everything just stopped working.

I asked my network team and my perimeter router's show arp showed that my NATed ARP entries from my external firewall as Incomplete:  Internet  x.x.x.x          0   Incomplete      ARPA.

I was running R80.10 with JHF 225.  I also had TAC on the line during the NAT fiasco and they had no idea what was causing it..another problem of course was that I was down down without any internet, so their remote session was dropped..

We have to go to R80.10, so our versions are in sync with a vendor.

Any ideas?

TIA

.

[FedericoMeiners]

@Nemz 

How did you perform the migration from one version to another? What kind of NAT are you using in your environment? Are you using Proxy ARP?

If you are using Proxy ARP don't use the import configuration option, just copy paste the involved lines from the show configuration option and install policy.

Remember to use fw ctl arp in # to check.

[Nemz]

How did you perform the migration from one version to another?

1) Imported R77.30 Policy to a to R77.30 VM then performed in place upgrade

2) exported policy and imported it into a clean R80.10

What kind of NAT are you using in your environment?

Automatic, Proxy and Manual..

[Maarten_Sjouw]

Be aware that when you use proxy arp and clusters, the best way forward is with enabling vMAC on clusters. This way you apply the command with the vMAC in the proxy arp command and the actual IP of the member. Do not use interface!!
And when you move to another box you need to update those MAC addresses.