Gateway license disappears suddenly - do not sync your licenses with UC! :)

[Kaspars_Zibarts]

Just wondering if anyone has a good idea how to check license "history" on the gateway as we suddenly lost all contracts and licenses from our VSX, here you can see that all VSX and blade licenses are gone..

Reapplied from local file again but little spooky as no one was logged in. Want to find out what happened

[G_W_Albrecht]

License corruption ? 😎

[Kaspars_Zibarts]

Penny just dropped! This appliance is under RMA process and replacement arrived yesterday.. As soon as CP updated asset list in UC all the licenses were gone even though we have had not even opened the box..

 just because we are synching CMA / mgmt with UC:

OUCH! Not good at all CP! Could have resulted in some major outage...

[G_W_Albrecht]

This had resulted in a big outage last week ! RMA for Recovery BIOS boot, GW node in production as Active in VSX HA Cluster. Replacement has arrived at noon, and during the night, VS Quota went from 10 to 0. No failover happened as the is no license pNote in ClusterXL, TAC declared. All customers VSs went down until a manual failover could be done.

What we have learned: If you do RMA with VSX Clusters, install AllInOne Evals for every GW before the replacement arrives 😎.

[Kaspars_Zibarts]

@PhoneBoy this does not look right from support point of view - User Centre assets updates by RMA process killing production VSX! Could you find someone in support organisation to look at this?

[G_W_Albrecht]

This is currently escalated - the answer from TAC is clear. More a question of either timing or of procedure - EVAL is easily installed if you know why.

[SharonElmashaly]

Thanks for raising this!

I am reviewing the process with the relevant teams to understand what needs to be changed/improved.

Sharon Elmashaly
VP, Customer Support

[Kaspars_Zibarts]

Great, good to hear! Let us know if you need any help / samples @SharonElmashaly 

[SharonElmashaly]

Hello,

I would like to update that after reviewing thousands of RMA events, we found this kind of incident to be extremely rare. Changing the current processes will have an impact on our ability to execute fast, as required in an event of RMA.

However, we are changing the communication and allowing the customer/partner to delay moving of Support and Services until approved explicitly.

This is clearly stated in the new RMA Confirmation Letter:

Dear Customer,
ATTENTION: It is a standard RMA procedure that Software Blades and Support transfer automatically from the Original unit to the Replacement unit upon receipt of delivery enabling the unit to be License ready. If you wish to delay this process, please contact Check Point Hardware Services directly through your ticket and request a delay. 

Thank you again for sharing the feedback!

[Kaspars_Zibarts]

Great, it's probably not the "smoothest" method as it will be prone to human mistakes, but hopefully it helps most of those rare cases from going wrong and affecting production networks

[G_W_Albrecht]

Thank you, this is really needed - in the past, i have had customers aware of such issues that made me instruct CP before the RMA concerning licenses. I would suggest another process instead of delay: Move a full eval license into customers UC and instruct him to install it in the unit to be replaced - 30 days should be enough for a maintenance window at nearly every customer...

[Darren_Fine]

Hi Kaspars_Zibarts,

We had the same situation yesterday !

The device was RMA'd months ago. Yesterday all of a sudden all licenses gone. Total VPN outage. Luckily it was a cluster we failed over to the other node(which had its licenses) and everything started working.

Had to log a case and get TAC to re-apply the correct licenses by pointing them to the old RMA'd ck ...and asking that they apply those licenses to the new ck.

Not fun .

[Eduardo_Eiros]

We faced the same issue with the two last RMAs, in both cases were not a critical issue, so the appliance was working. We lost the licenses with the RMA process and VPNs stopped working.... big outage. 

This process should be reviewed. 

[G_W_Albrecht]

Currently a question: What would be the best procedure with a basically working appliance to get the new replacement into production? All preparation and vsx_util reconfigure needed ?

[Kaspars_Zibarts]

Correct - set up underlaying GAIA interfaces, DNS, routes, license (you can bastartise GAIA backup to do that), then vsx_util_reconfigure. Plus any SSH keys and local scripts/cronjobs if you had any.

[Bob_Zimmerman]

In general for VSX:

1. Go through initial configuration. I build config_system files. If you used config_system to build the original, you can use the same file to build the replacement.
2. If you change the volume sizes from their defaults, do it here.
3. Do any OS-level configuration for VS 0. For me, this is bonds, SNMP, and central authentication.
4. 'vsx_util reconfigure' on the management. This establishes SIC and pushes down the VSX config (building the contexts and interfaces) and policies.
5. Do any remaining local configuration. Most notably, any dynamic routing has to be set up after the 'vsx_util reconfigure', because that's when the contexts are built on the firewall. Manual proxy ARP outside VS 0 involves a local.arp file per VS doing manual proxy ARP.

[Dorit_Dor]

Indeed the process is being evaluated. We werent aware of this till your report

TNX for raising it

Dorit

[Danny]

Hi @Dorit_Dor,

one of our customers experienced a very similar issue today.
The customer recently ordered to replace his old Open Servers with new Check Point 7000 appliances.
So that's a trade-in of the old licenses for the 7000 appliances that come bundled with new licenses.
The new 7000 appliances didn't arrive yet and are not yet listed in the customers CP UserCenter Account.
However, due to the trade-in, Check Point already deleted the old licenses causing an immediate outage in our customers production network.

Regards, Danny

[Dorit_Dor]

Thank you for letting me know 

please send me direct message (on mail or checkmates) with the details of the customer. I take these very seriously and want to make sure we debrief it and take very serious actions to protect against such future cases. 

Dorit