Gateway issue post upgrade to R80.30

[TimLofgren]

Hello all,

I recently upgraded my SMS server from R80.20 to R80.30 and ran all pending hotfixes after the update.  I took new backups of the config and snapshots of the SMS server after all was done.  I then started the snapshots and backups of the one and only gateway that we utilize.  I ran the upgrade from CPUSE.  After it rebooted the management interface never came back up.  The device was located off site at a data center so I had to go capture it and it is now on my desk.  I can connect to the device CLISH from console but I have no idea how to troubleshoot what went wrong or how to proceed with the upgrade to the R80.30 version after this failure.  Luckily this is not in production yet.  We are still on our old firewall systems until I get this online and working in R80.30.  Can anyone point me in the right direction?

Tim

[G_W_Albrecht]

I would suggest to contact TAC for troubleshooting then - this is possible as you have CLI access and get log files from the GW.

Another possible workaround would be a fresh install on the box !

[Timothy_Hall]

What is the model of the SMS?  It sounds like you might have used the wrong upgrade image which does not have a driver for the NIC which is why it didn't come back on the network.  There are special install/upgrade images for certain new models, example:

sk157153: Smart-1 625

[TimLofgren]

@G_W_Albrecht 

I had that thought as well to do a fresh install but I wanted to try to troubleshoot this first to learn more about the process so that if this presents itself in the future I will know how to proceed.  Thanks for the suggestion.

@Timothy_Hall 

The SMS is an open server running Gaia on HyperV.  I am pretty sure I used the correct upgrades as they were both done with CPUSE from inside the Gaia portal.  I can post the exact updates I ran on the SMS server if needed.  Below is the output from the show interface Mgmt command on the gateway from CLISH in a console session.

DR-Checkpoint-01> show interface Mgmt
state on
mac-addr 
type ethernet
link-state link up
mtu 1500
auto-negotiation on
speed 1000M
ipv6-autoconfig Not configured
duplex full
monitor-mode off
link-speed 1000M/full
comments
ipv4-address 
ipv6-address Not Configured
ipv6-local-link-address Not Configured

Statistics:
TX bytes:3192 packets:76 errors:0 dropped:0 overruns:0 carrier:0
RX bytes:0 packets:0 errors:0 dropped:0 overruns:0 frame:0

I removed the MAC address and IP.  The interface appears to be up but I still can't reach it with ping or HTTPS.  I just plugged it in so that is why the packets are so low.  I see that it is only sending and not receiving, but I am unclear on how to proceed.  Is there any other Clish commands that would be useful in this situation or should I just give up and do a fresh install as @G_W_Albrecht has suggested?  Thank you for the help gentlemen.

Tim

[Timothy_Hall]

From expert mode what does ethtool -S Mgmt and ethtool -i Mgmt show?

Maybe from expert mode try a ifdown Mgmt;ifup Mgmt

[TimLofgren]

@Timothy_Hall 

Hmmm.....I thought I posted a reply to this but I do not see it here now.  Strange.  Below it the output requested from expert mode.  I did bounce the port with the ifdown and ifup commands, but it had no effect.  Also have the show version all command as well.

[Expert@DR-Checkpoint-01:0]# ethtool -S Mgmt
NIC statistics:
rx_packets: 0
tx_packets: 26202
rx_bytes: 0
tx_bytes: 1676928
rx_broadcast: 0
tx_broadcast: 26202
rx_multicast: 0
tx_multicast: 0
multicast: 0
collisions: 0
rx_crc_errors: 0
rx_no_buffer_count: 0
rx_missed_errors: 0
tx_aborted_errors: 0
tx_carrier_errors: 0
tx_window_errors: 0
tx_abort_late_coll: 0
tx_deferred_ok: 0
tx_single_coll_ok: 0
tx_multi_coll_ok: 0
tx_timeout_count: 0
rx_long_length_errors: 0
rx_short_length_errors: 0
rx_align_errors: 0
tx_tcp_seg_good: 0
tx_tcp_seg_failed: 0
rx_flow_control_xon: 0
rx_flow_control_xoff: 0
tx_flow_control_xon: 0
tx_flow_control_xoff: 0
rx_long_byte_count: 0
tx_dma_out_of_sync: 0
lro_aggregated: 0
lro_flushed: 0
lro_recycled: 0
tx_smbus: 0
rx_smbus: 0
dropped_smbus: 0
os2bmc_rx_by_bmc: 0
os2bmc_tx_by_bmc: 0
os2bmc_tx_by_host: 0
os2bmc_rx_by_host: 0
rx_errors: 0
tx_errors: 0
tx_dropped: 0
rx_length_errors: 0
rx_over_errors: 0
rx_frame_errors: 0
rx_fifo_errors: 0
tx_fifo_errors: 0
tx_heartbeat_errors: 0
tx_queue_0_packets: 26202
tx_queue_0_bytes: 1100484
tx_queue_0_restart: 0
rx_queue_0_packets: 0
rx_queue_0_bytes: 0
rx_queue_0_drops: 0
rx_queue_0_csum_err: 0
rx_queue_0_alloc_failed: 0
[Expert@DR-Checkpoint-01:0]# ethtool -i Mgmt
driver: igb
version: 4.1.2
firmware-version: 0. 6-2
bus-info: 0000:08:00.0

DR-Checkpoint-01> show version all
Product version Check Point Gaia R80.30
OS build 200
OS kernel version 2.6.18-92cpx86_64
OS edition 64-bit

So it appears the update did take but for some reason the interface is not receiving packets, only sending.

Tim

[Timothy_Hall]

This is obviously some kind of NIC or NIC driver problem at the Gaia level.  I've seen this before and could never figure out what was wrong. 

Last resort is forcibly unloading and reloading the igb driver like this which will sometimes shake things loose.

modprobe -r igb

modprobe igb

If that doesn't work, I think your best option is a reload unless you want to get TAC involved.

[TimLofgren]

@Timothy_Hall 

As you suspected, it did not work.  I guess I will have to involve TAC at this point.  Thanks for the assistance.  It is much appreciated.

Tim

[TimLofgren]

UPDATE:

@Timothy_Hall 

This issue was resolved by Checkpoint support engineers.  In case anyone wants to know the issue here it is.  The in place upgrade was supposed to keep the firewall policies installed.  Something went wrong in the upgrade and it did not stay in place.  It instead was using the initial out of the box policy which was blocking everything.  So a simple fw unload solved that and allowed the Mgmt interface to ping and connect to Smartconsole.  I also had to install the policy from SmartConsole as there were multiple changes not on the gateway since it had installed the initial policy.  So the changes waiting were all the implemented changes ever made from SmartConsole.  Once I installed that it seemed to be working.  However, there was also an issue with the Gaia portal Cnot presenting.  We had to change the port from standard 443 to 4434 and push that from an install in SmartConsole.  The service was not running on the gateway so we had to get that service started as well.  Once those items were done, it updated in SmartConsole and the Gaia portal was working.  Then we had to finish the last three upgrades from CPUSE which was the jumbo hotfix for R80.30, and an update for CPinfo and some other one that I forgot what it was.  But now all is working.  I just wanted to share this for anyone that might have a use for this info in the future.  Thanks to all that assisted me.