SMB 1500 Upgrade not reliable

[Alex-]

I have a few 1500 series in different countries managed by Smart-1 Cloud running R80.20.X.

I've downloaded R80.20.40 from the SK for the 1500 series, verified the hash and uploaded it on the machine where it's recognized.

So far, so good. I've started upgrading the first machine and after reboot it didn't come back online.

After a couple of hours, the firewall was back online and had rolled itself back to R80.20.X. No particular messages in logs.

TAC told me that it can happen and I should try again until it succeeds as these systems have an embedded software and there's only one way of doing the upgrade. I had this behavior already on systems where I was onsite but in this case they're scattered in different countries and I don't feel like brute-forcing the upgrade on a customer infrastructure when everything is done by the book with potential hours-long downtimes.

Long story short,  is it an experience of Spark 1500 here users can relate to? I don't know if the fact they're managed by Smart-1 Cloud has any influence in this.

[PhoneBoy]

Smart-1 Cloud should not impact this, at least to my knowledge.
If you can get console access to the affected platforms during the upgrade, it might shed some light on what's happening during the upgrade process.

[Greg_Harbers]

We have had a similar issue, although only on 1590s, not on 1550s. 

In our case, the gateways have a connection to the customers internal WAN and the internet. Following the topology that we had in place when we were using 1400 series gateways, we used the connected the Internet to the gateway WAN port, and the customer WAN to the DMZ interface, with the site internal network connected to vlan interfaces on the LAN1 port. We disable the internal switch on these devices. 

On 1590s with such a configuration, attempts to upgrade the gateway would do exactly what you described. The firmware would appear to upload correctly but after some time, the gateway would finally come back to the original version. We had a connection to the devices console port and we could see the boot messages. Effectively what was happening was that the gateway would go into maintenance mode and reboot five times to boot of the new firmware. After five attempts, the device would revert back to the original release.

The fix was to use another interface for the customers wan connection, in this case we used LAN8. Having done this we have not experienced the issue since. Why we never struck the issue on 1550s is because these devices do not have a DMZ interface hence we always use LAN5 for the customers WAN connection.

Hope this helps.

[Alex-]

@Greg_Harbers this is great information and thank you for sharing.

It's 1590'sall over the place so it might be an issue with that series. I don't use the DMZ port and LAN switch is also disabled, but I use LANBOND interfaces. It's typical LAN - FW - Internet setups.

I wonder if this issue is known to Check Point as it's currently blocking the rollout of R80.20.40 as the first attempt caused quite an outage and the selling point of these boxes was the minimal administration and moving parts needed since they're in place in different countries.

I can't really change the network design either.

[PhoneBoy]

A TAC case is probably in order.
Getting console access to one of the systems will likely be required for troubleshooting.

[Alex-]

Tried again and same issue so it wasn't just a mishap the first time it failed.

It was solved by putting a bogus IP on the unused DMZ interface and proceeding with the upgrade, then the system went back online with R80.20.40 after a few minutes.