R81.10 Jumbo Hotfix Accumulator - New Ongoing take #30

[eranzo]

Hi All,

A new Ongoing Jumbo Hotfix Accumulator take for R81.10  (take 30) was released today,  and is available for download. Please refer to sk175186

Please note the following:

•        Availability:

o   Available to download the via sk175186

o   Available for download via CPUSE by using package identifier.

o   Can be provided by customer support

Content included in this take:

• List of resolved issue in this take can be found in sk175186

New: Starting from R80.40, Central Deployment allows you to perform a batch deployment of Hotfixes on your Security Gateways and clusters from SmartConsole!! For more information, see sk168597.

Thanks ,

Release Operation Group

[the_rock]

The only thing I see is below, but cp products are protected anyway.

UPDATE: The Apache Log4j Java library is updated in order to harden the system. Check Point products are not vulnerable to Log4j. This change is motivated by cyber hygiene best practices. For more information, refer to sk176865. 

[Chris_Atkinson]

@the_rock There is also the CPview fix for the issue highlighted with the previous take. Getting closer to recommended status now. 🙂

[Norbert_Bohusch]

We got the following information for one of our customers:

We would like to bring to your attention that we recently identified an issue with installing new R81.10 Jumbo takes on a MGMT/SA machine with a specific setup.

According to our records, this information might be applicable to your setup and we advise that you note the following information.

Problem description –

R81.10 Jumbo Take 30 or higher can’t be installed on a MGMT/SA machines that were previously installed with take 9, take 14 and take 22 in a row,

Preventative steps –

For installing R81.10 Jumbo Take 30 or higher, please open a ticket to Check Point Support and they will assist you through the upgrade process. 

Note:  Once Take 30 or higher is installed, the issue is resolved and no future upgrade issues are expected.

[the_rock]

@Norbert_Bohusch ...thats good info actually. I will test this in my R81.10 lab shortly and update the thread.

[the_rock]

I dont know if there is way to get around below message, but this is what I see. I must have removed it at some point by mistake...

 The package failed to install at Thu Jan 13 10:43:10 2022
Reason of failure: There was a problem while creating backups for the files being updated. Please try again. If the issue persists, please contact Check Point Technical Services for further assistance.More Info: The backup file /opt/CPda/backup/CheckPoint#CPUpdates#All#6.0#5#3#BUNDLE_R81_10_JUMBO_HF_MAIN#9/LastTake/ReportingServer_backup_HOTFIX_R81_10_JUMBO_HF_MAIN.tgz doesn't exist.

[Scottc98]

Question:   

For those that are already on Take 22, is there going to be a future JHF that will remove the custom process for getting to Take 30 or above?    

I have this running in my lab at home right now and will just wait if there is an intended future JHF coming that will allow for normal BAU patching.

Thanks in advance 

[the_rock]

Maybe @eranzo , if you could confirm if there is a way to get around below error when I try install jumbo take 30. I would like to test it in lab myself, but since file is missing, not sure what options are, except install jumbo take 9 on another system and then get the file that way? I was going to try get .tgz file for jhf 9, but cant find it on support site.

The package failed to install at Thu Jan 13 10:43:10 2022
Reason of failure: There was a problem while creating backups for the files being updated. Please try again. If the issue persists, please contact Check Point Technical Services for further assistance.More Info: The backup file /opt/CPda/backup/CheckPoint#CPUpdates#All#6.0#5#3#BUNDLE_R81_10_JUMBO_HF_MAIN#9/LastTake/ReportingServer_backup_HOTFIX_R81_10_JUMBO_HF_MAIN.tgz doesn't exist.

[PhoneBoy]

If you installed Take 9, then Take 14, then Take 22, then this issue applies @Scottc98.
If you just installed Take 22, I don't think issue applies.
Also, I'm pretty sure that we will resolve this issue in a future JHF (particularly in one that goes GA).

[MatanYanay]

Hi all 

As we updated the problem is relevant only on MGMT/SA machines that were previously installed with take 9, take 14 and take 22 in a row

If you installed Take 9 and Take 14, we recommend skipping Take 22. Install Take 30 on top of Take 14

If you installed Take 9, Take 14 and Take 22,  Contact Check Point Support to get assistance with the upgrade process.

Once Take 30 or higher is installed, the issue is resolved and no future upgrade issues are expected.

Thanks 

Matan.

[Scottc98]

@PhoneBoy   I do hit that use case where I started with a fresh 81.10 management with take 9, then 14 with a few bugs I was hitting,  and then 22 after it was put as GA.

I can run like this for a while and check back if there is a graceful way to move up in the future.   If not, I'll open a case and get the detailed path.

Thanks for the updates 🙂

[the_rock]

Just a quick update...though I did have JHFs installed in order 9, then 14, then 22, take 30 installed no issues, worked like a charm on 2 gateways.. Now, on management server, even though I uninstalled take 22 completely, it still would not let me install take 30, its giving same error as one I mentioned previously.

Sooo...what I will do is try install take 9 and then 30, see if that works.

[the_rock]

Reinstalled take 9 on mgmt server, worked fine, so I upgraded to jhf 30. No issues with pushing policy on gateways either on jhf 30...so far, all looks fine.

[Fung_To_Puk]

Hi,

I installed uninstall T22, then uninstall T9 before installing T30, install is smooth whith this.

Confirm the metioned CPview fix, but the Network page Top protocol and connection are still no stat showing.

CPview -> Advanced -> UP still missing the connection stat too

[Daniel_]

> Confirm the metioned CPview fix, but the Network page Top protocol and connection are still no stat showing.

Take a look to sk167903. You have to activate it...

[Fung_To_Puk]

Great, thanks.

But the page CPview.Advanced.UP still missing some stats there.

[JozkoMrkvicka]

Why not somehow remove Take 22 if that one is causing the issue? Just remove all the links related to Take 22 to avoid download/installation via cpuse and add big warning in some SK (R81.10 homepage or R81.10 jumbo page)for all users which already installed Take 22...

[Arik_Ovtracht]

Hi @Fung_To_Puk ,

>But the page CPview.Advanced.UP still missing some stats there.

Can you please show an example of the missing stats?

[MatanYanay]

Hi @JozkoMrkvicka 

As we updated the problem is relevant only on MGMT/SA machines that were previously installed with take 9, take 14 and take 22 in a row

the issue is with take 14 and take 22 and we removed the ability to download take 14 in order to prevent such issues to happen. 

In addition we indeed added important note in the Jumbo Hotfix Accumulator for R81.10 sk  and update the relevant customers who has the above installation sequence 

Important Notes

• Installation of R81.10 Jumbo Take 30 or higher on Management/Standalone machines differs from the regular upgrade process:
• • If you installed Take 9 and Take 14, we recommend skipping Take 22. Install Take 30 on top of Take 14.
  • If you installed Take 9, Take 14 and Take 22,  Contact Check Point Support to get assistance with the upgrade process.

Thanks 

Matan.

[Pauli]

 @MatanYanay 

We have the following update path:

R81.10 => Jumbo 9 => Jumbo 22

This updateapath is not described in the SK. Is the update to Jumbo 22 possible without problems or does the support also have to be contacted?

[MatanYanay]

Hi @Pauli

as we previously mention its relevant only on MGMT/SA machines that were previously installed with take 9, take 14 and take 22 in a row

if you have have this path as you described R81.10 => Jumbo 9 => Jumbo 22  you should not have any issues to install take 30

Thanks 

Matan.

[RamGuy239]

In our environment we have a dedicated SmartCenter/management server, a dedicated log server and a dedicated SmartEvent Server on separate installations running on VMware ESXi, all being installed using the R81.10 OVF for management installations, all running R81.10 and they've all been going through this route of patches: GA -> Take 9 -> fw1_wrapper_HOTFIX_R81_10_JHF_T9_APACHE2_4_51_MAIN_GA_FULL -> Take 14 -> Take 22.

Trying to install Take 30 fails as expected. I tried to remove Take 22 on all three. Still not working. I tried to remove both Take 22 and Take 14, still not working. I tried to remove it all getting it back to GA. But once we are back at GA we aren't able to install Take 9, Take 14, Take 22 or Take 30 as it claims it's missing something in the repository related to Take 9 making us stuck on GA without the capability to install any Jumbo Hotfix at all.

Reverted all three back to the snapshot before testing so they are back to running Take 22.


On gateways, there seems to be no issue. I even tried to simulate the same thing, removing Take 22, removing Take 14, removing Take 9, revering the gateway back to R81.10 GA and the gateway is not faced with the same issue as the management installations where I no longer are able to install any Jumbo Hotfix after reverting it all the way back to R81.10 GA.


Doesn't seem like you are able to bypass this issue by uninstalling Take 22 or anything? Does this mean that anyone that has been going the route from GA -> Take 9 -> Take 14 -> Take 22 needs to get in contact with TAC to be able to update to any new Jumbo Hotfixes released in the future? Or will the next on-going jumbo hotfix for R81.10 include a fix that makes this a non-issue?

[eranzo]

Hi,

At the moment the only way to resolve the issue is by contacting the support center.

Once take 30 (or higher) is installed the issue will not reoccur.

Eran.

[the_rock]

I agree @eranzo ...I had exact same issue on lab mgmt and what I did was fresh installed R81.10, installed take 22 and then take 30 worked fine. On gateways, all worked fine without any problems.

[Arik_Ovtracht]

Hi @Fung_To_Puk ,

Regarding:

>But the page CPview.Advanced.UP still missing some stats there.

Can you please tell me if the stats were missing before you installed the JHF?

[Fung_To_Puk]

the connection stat

[Fung_To_Puk]

Hi,

After some trial and error, the reason of our lab keep getting dump file for fw_full and scanengine_* seems to be related to the DOS rate limit (fwaccel dos rate).

As soon as we remove all DOS rate limit (fwaccel dos rate del all) there are no more dump file generated after a few hours.

This symptoms begin after JHF14, when we were testing JHF9, no problem seems to happen from it.

[Fung_To_Puk]

Hi,

After a few more days of testing, we finally found what is the problem causes coredump.

First, the DOS rate limit would generate coredump  when I issue "fwaccel dos rate get", and a segmentation fault error message appear, this is a seperate issue with below.

A second problem is with the QoS blade, which after disabled, no more coredumps from fw_full anymore.

[Naama_Specktor]

Hi @Fung_To_Puk !

Please share with me the SR number so that we can review the core files.

If there is no SR open yet – please open one - and upload the 2 core files with it.

Many Thanks,

Naama

[Fung_To_Puk]

Hi,

The is just in our lab environment, no ticket opened, but if you want to test it, just create QoS rule with multiple IPs (putting GW IP here as in my lab) and services (e.g. https and ipsec ports) in 1 single QoS rule, that seems create the problem for QoS, if you want problem to happen fast, you may try to shorten the periodic update time of Antivirus/Antibot.

[Arik_Ovtracht]

Hi @Fung_To_Puk ,

thanks for sharing the CPview screenshot.

We have found the issue with this view and a fix will be in the next JHF.

[Peter_Lyndley]

@Fung_To_Puk 

Yes there is a known issue in both R81 and R81.10 with QoS and fw_full core dumping that we also came across in production. R&D have produced a fix , however I have been informed that the fix is not yet rolled in to any jumbo hotfix in either version.

[Arik_Ovtracht]

Hi @Fung_To_Puk ,

The fix for the CPview issue was released as part of the R81.10 JHF take 38.

Please let me know if it installing the take fixed your issues.

[Fung_To_Puk]

Its all good for JHF T38, I can enable everything and no more coredump files too, great work

[Scottc98]

I have a question regarding the issue with installing on the management server and this problem:

******

Problem description –

R81.10 Jumbo Take 30 or higher can’t be installed on a MGMT/SA machines that were previously installed with take 9, take 14 and take 22 in a row,

******

1) If you have this use case and run the verifier on the Take 30 JHF package, would the verifier fail (Therefore knowing you ran into the 'bug') or would it 'pass' but just fail during the upgrade?

[MeravAlon]

HI @Scottc98 

Yes, The Verifier will fail with the use case you described.

Regards, Merav Alon