Policy install fails with malloc failure

[Oliver_Fink]

At a customer, I face the problem that a policy install on 59 SMB (1430/R77.20.x and 1530/R81.10.x) appliances failed for 40 of them with a memory error:

Gateway: CP-XXX01
Policy: New_CP_Ext_Location_POL
Status: Failed
- malloc failed: Cannot allocate memory
- var::operator new: failed to allocate memory using fw_malloc.
- terminate called after throwing an instance of 'std::bad_alloc'
- what(): std::bad_alloc
- Operation failed, install/uninstall has been improperly terminated.

When I try to fetch the policy from a single appliance everything installs smoothly.  When I try to install a single appliance from R81.10 SmartConsole everything works well.

Did we hit a maximum number of parallel installation targets? Anyone with same experiences? Any hints?

Since all appliances share the same policy we would prefer if we could install the policy to all targets in parallel as we did for a long time.

[Tal_Paz-Fridman]

I suggest opening an SR with TAC.

I have also forward it to relevant owners in our R&D to receive their inputs.

[Oliver_Fink]

Thank for your reply. I opened a SR.

[the_rock]

Let us know how it gets solved.

Best,

Andy

[Oliver_Fink]

Update auf JHFA Take 130 did not do the job.

[Tal_Paz-Fridman]

Hi 

Ask the TAC engineers to see if it might be related to a similar issue I found in 6-0003538539 (and internal CFG task TM-45561)

It might help shorten the process.

Thanks

[the_rock]

Just a suggestion...what I would do is maybe monitor memory on mgmt server when doing this. Seems to me like that could be part of the problem.

Just thinking out loud as they say...

Best,

Andy

[Oliver_Fink]

Seems so. I see 2 fw_loader processes:

1 going up to 24,9 % of MEM and then vanishing. That is when all R81.10.x gateways show the malloc() error message.

1 is going up to 6,9 % of MEM and stays until shortly before the R77.20.x gateways have their policy installed.

[the_rock]

How much ram is on mgmt server?

Andy

[Oliver_Fink]

16 GB, you can see it in the top screenshot above (15.430 total).

[the_rock]

Yes, sorry I missed that. What did TAC say?

Andy

[Oliver_Fink]

They are requesting debug logs during policy install, CPinfo and management export. I just delivered that. 

[the_rock]

Mgmt export could help them...not sure if they plan to replicate the issue. Appears fw_loader process is whats consuming most memory when you are installing the policy, but thats always invoked by fwm process itself. Can you run cpwd_admin list when installing the policy please and send? I want to see whats fwm consuming...

Andy

[Oliver_Fink]

Thank you for your support. I do not understand, how do you want to see from "cpwd_admin list" what fwm is consuming? The policy installation is running for several minutes until it fails. When, do you think, to issue the command?

That is the output from the command when not installing policy:

[Expert@some_host:0]# cpwd_admin list
APP             PID   STAT #START START_TIME           MON COMMAND
CPVIEWD          9096 E    1      [16:51:13] 18/1/2024 N   cpviewd
CPVIEWS         26854 E    1      [14:25:47] 19/1/2024 N   cpview_services
CVIEWAPIS        9127 E    1      [16:51:13] 18/1/2024 N   cpview_api_service
CPD              9148 E    1      [16:51:13] 18/1/2024 Y   cpd
FWD              9259 E    1      [16:51:14] 18/1/2024 N   fwd -n
FWM              9269 E    1      [16:51:14] 18/1/2024 N   fwm
FWMHA            9375 E    1      [16:51:15] 18/1/2024 N   fwmha -H
STPR             9412 E    1      [16:51:15] 18/1/2024 N   status_proxy
CLOUDGUARD       9441 E    1      [16:51:15] 18/1/2024 N   vsec_controller_start
CPM              9979 E    1      [16:51:20] 18/1/2024 N   /opt/CPsuite-R81.10/fw1/scripts/cpm.sh -s
SOLR             9990 E    1      [16:51:20] 18/1/2024 N   java_solr
RFL             10130 E    1      [16:51:21] 18/1/2024 N   LogCore
SMARTVIEW       10196 E    1      [16:51:21] 18/1/2024 N   SmartView
INDEXER         10461 E    1      [16:51:24] 18/1/2024 N   /opt/CPrt-R81.10/log_indexer/log_indexer
SMARTLOG_SERVER 10584 E    1      [16:51:25] 18/1/2024 N   /opt/CPSmartLog-R81.10/smartlog_server
REPMAN          10896 E    1      [16:51:29] 18/1/2024 N   java_repository_manager
DASERVICE       10980 E    1      [16:51:30] 18/1/2024 N   DAService_script
AUTOUPDATER     11012 E    1      [16:51:31] 18/1/2024 N   AutoUpdaterService.sh
CPSM             8319 E    1      [16:53:41] 18/1/2024 N   cpstat_monitor
LPD             27238 E    1      [16:57:37] 18/1/2024 N   lpd

That did not change during policy install.

[the_rock]

Sorry, I said it wrong...I just wanted to confirm fwm process shows E 1, which ot does. Any news from TAC?

[Oliver_Fink]

They are still investigating.

[the_rock]

K, fair enough. My gut feeling tells me this is most likely management issue, because if you think about it logcially, its odd it would fail for all 40 firewalls.

Andy

[Oliver_Fink]

Yes, it seems so. Today, I tried again and had a policy install with 35 full and 5 accelerated installs. That worked for all gateways. If I enforce full policy install for all 40 gateways, it fails for all gateways. I am sure it is a management issue.

[the_rock]

Based on pure logic, I agree 100%

[the_rock]

Have you seen this?

https://support.checkpoint.com/results/sk/sk104783

[Oliver_Fink]

Yes. But very wrong version. And it requires a hotfix. I expect the same, here.

[the_rock]

I agree, it is totally wrong version, BUT, it might be applicable, you should ask them regardless.

Best,

Andy

[Oliver_Fink]

We will get an hotfix. TAC is working on it.

[the_rock]

Good deal, thanks for letting us know.

Best,

Andy

[Oliver_Fink · Accepted solution]

---

@Oliver_Fink wrote:

We will get an hotfix. TAC is working on it.

---

I marked this as solution. That was wrong. This is a known limitation since R80.20 GA as shown here with ID 02337281. It is not mentioned as a known limitation of R81.10.X. Maybe this sentence should do the trick:

All previous limitations are relevant to the following version unless stated as resolved.

It took TAC from 18th of January to the the 7th of March to discover that. No pun intended to them. Poor documentation meets even more poor functionality. There is no functionality in SmartConsole to configure batches for installation in an efficient way. And that is a thing SmartConsole should keep away from the user's responsibility.

[Oliver_Fink · Accepted solution]

---

@Oliver_Fink wrote:

We will get an hotfix. TAC is working on it.

---

I marked this as solution. That was wrong. This is a known limitation since R80.20 GA as shown here with ID 02337281. It is not mentioned as a known limitation of R81.10.X. Maybe this sentence should do the trick:

All previous limitations are relevant to the following version unless stated as resolved.

It took TAC from 18th of January to the the 7th of March to discover that. No pun intended to them. Poor documentation meets even more poor functionality. There is no functionality in SmartConsole to configure batches for installation in an efficient way. And that is a thing SmartConsole should keep away from the user's responsibility.

[Steffen_Appel]

The 75.20 bug was fixed for us (sk104783), but they refused to port the fix for the R77.20 loader see https://support.checkpoint.com/results/sk/sk113253 . 

The SK says 25 is the max, for us it worked until about 40 1100/1400s.

Good thing is, that it works now with r81.10.xx