Failed to upgrade checkpoint 6200P from r81.10 to r81.20

[Rabin]

Hi checkmates !

Greetings !

Recently, we had planned to upgrade 6200 security gateway from r81.10 to r81.20 but we could not do that. We followed below steps:

1. Downloaded the checkpoint major version upgrade package from status and Action>packages

2. Verified the packages 

3. Displayed install and upgrade allowed was shown but below that i noticed install allowed section.

4. Proceed with upgrade option

5. Device reboots, all network goes down

5. Failed to open gaia portal or management network or any network that was passing through that gateway.

6. Used console cable to see the version shows r81.20 with warnings

I revert back using autosnapshot and the connectivity was restored, don't know what happened or what did i miss here. i observed same thing in my perimeter device as well the device was successfully upgraded but other internal network were not able to reach internet, while checkpoint itself was able to reach internet.

I revert it back and the all device were able to reach internet, If someone had faced same issue with 6200 device. please let me know. what did i miss or what was wrong in this scenario.

The management server which was in vm was only successfully upgraded following the same steps.

Thank You.

Rabindra

[AkosBakos]

Hi @Rabin,

Without logs, the investigation almost is impossible. 🙂

But no worries, we try to give tips and trick for the next time:

DAagent:

the DAagent was updated to the latest version?

Collect the Deployment agent logs use this command da_cli collect_logs ->check it, maybe there is any nasty in it.

https://support.checkpoint.com/results/sk/sk92449

The install logs are here:

/var/log/install_Major_R81.20_ivory_main_T631_detailed.log

And also a lot of log here: /opt/CPInstLog/

You wrote this "Used console cable to see the version shows r81.20 with warnings"

What were the warnings?

Akos

[Rabin]

Hi  @AkosBakos 

Thank You for the response, the Deployment Agent build was: 2443  and the device which did not come had warnings, i used show version all command, and the output was r81.20 with warning the device was not upgraded fully something like that, and  as you mentioned the logs, i have those install_Major_R81.20_ivory_main_T631_detailed.log  logs. Can you give what had happened by analysing it or should i open case for TAC and to add i also noticed there are no core dumps generated.

Thank You.

[AkosBakos]

Hi @Rabin 

Hard to reprodicate the issue in my  head, but from this:

"r81.20 with warning the device was not upgraded fully.... "

I conclude that, the post installations scripts, which run after there reboot didn't finished.  

How much time had passed betwen the reboot, and the revert? 

If you open a TAC case, the TAC wants a session where you can do the upgrade again.

Akos

[AkosBakos]

Hi @Rabin 

And one more thing. If you have time, do this upgrade in LAB. I mean that, install an R81.10, then upgrade it.  I always did this if I had time.

[Rabin]

Hi @AkosBakos 

Thank You for the guidance, I have upgraded the checkpoint firewall in lab from r81.10 to r81.20 in vm. I had upgraded in different version and model in production environment too but with only difference was from r80.30 or r80.40 to r81.10 upgrade path in different model device. 

I could not find out what, why this upgrade was unsuccessful, as it was not even in cluster but in standalone deployment.

Rabindra

[AkosBakos]

Hi @Rabin 

In this case there are two ways:

1. Open a TAC Case, arrange a maintanace window, and wait for the date.  (of course on-site)
2. Such strange things happen. I can list a lot of interesting things, which solution was to reinstall the GW from scratch, and (re)SIC it with the MGMT.

The second solution based on my experience. I can be fast. The disadvantage is that, we didn't find the exact solution or the root cause of the issue.

And one more: Does the GW have LOM?

Akos

[the_rock]

Can you list EXACT steps you followed to upgrade? Is it just single fw or cluster? Either way, you would need to check fw stat after reboot, as it may load initial policy, which blocks everything, except local ssh and web ui on port 443. Also, version in smart console object would need to be updated.

Andy

[Rabin]

Hi @the_rock 

It was in single fw and to check the firewall status after reboot, there was not reachability or any services up, no ssh, no web ui, after system went down for reboot, we waited long enough to finalize it had some issue and  we rushed to datacenter and through console, we revert it back using autosnapshot and the service was restored. 

The steps taken were:-

1. Take backup and snapshots 

2. Fetch updates from checkpoint cloud through cpuse

3.  Verified the deployment agent was latest

3. Verified the recommended package, clean install and upgrade was allowed.

4. Clicked upgrade

5. And monitor the system, system went down for reboot.

Rabindra

[the_rock]

Sounds like for some reason, though I cant say for sure, may had loaded whats called default filter policy, which would have blocked EVERYTHING.

Andy