topic Re: Policy install fails with malloc failure in Firewall and Security Management

Policy install fails with malloc failure

Oliver_Fink — Wed, 17 Jan 2024 16:33:59 GMT

At a customer, I face the problem that a policy install on 59 SMB (1430/R77.20.x and 1530/R81.10.x) appliances failed for 40 of them with a memory error:

Gateway: CP-XXX01
Policy: New_CP_Ext_Location_POL
Status: Failed
- malloc failed: Cannot allocate memory
- var::operator new: failed to allocate memory using fw_malloc.
- terminate called after throwing an instance of 'std::bad_alloc'
- what(): std::bad_alloc
- Operation failed, install/uninstall has been improperly terminated.

When I try to fetch the policy from a single appliance everything installs smoothly. When I try to install a single appliance from R81.10 SmartConsole everything works well.

Did we hit a maximum number of parallel installation targets? Anyone with same experiences? Any hints?

Since all appliances share the same policy we would prefer if we could install the policy to all targets in parallel as we did for a long time.

Re: Policy install fails with malloc failure

Tal_Paz-Fridman — Wed, 17 Jan 2024 17:03:52 GMT

I suggest opening an SR with TAC.

I have also forward it to relevant owners in our R&D to receive their inputs.

Re: Policy install fails with malloc failure

the_rock — Wed, 17 Jan 2024 17:10:48 GMT

Just a suggestion...what I would do is maybe monitor memory on mgmt server when doing this. Seems to me like that could be part of the problem.

Just thinking out loud as they say...

Best,

Andy

Re: Policy install fails with malloc failure

Oliver_Fink — Thu, 18 Jan 2024 13:19:49 GMT

Thank for your reply. I opened a SR.

Re: Policy install fails with malloc failure

the_rock — Thu, 18 Jan 2024 13:22:59 GMT

Let us know how it gets solved.

Best,

Andy

Re: Policy install fails with malloc failure

Oliver_Fink — Thu, 18 Jan 2024 13:38:23 GMT

Seems so. I see 2 fw_loader processes:

1 going up to 24,9 % of MEM and then vanishing. That is when all R81.10.x gateways show the malloc() error message.

1 is going up to 6,9 % of MEM and stays until shortly before the R77.20.x gateways have their policy installed.

Re: Policy install fails with malloc failure

the_rock — Thu, 18 Jan 2024 13:44:09 GMT

How much ram is on mgmt server?

Andy

Re: Policy install fails with malloc failure

Oliver_Fink — Fri, 19 Jan 2024 09:00:49 GMT

16 GB, you can see it in the top screenshot above (15.430 total).

Re: Policy install fails with malloc failure

Oliver_Fink — Fri, 19 Jan 2024 09:02:07 GMT

Update auf JHFA Take 130 did not do the job.

Re: Policy install fails with malloc failure

the_rock — Fri, 19 Jan 2024 12:19:50 GMT

Yes, sorry I missed that. What did TAC say?

Andy

Re: Policy install fails with malloc failure

Oliver_Fink — Fri, 19 Jan 2024 16:58:11 GMT

They are requesting debug logs during policy install, CPinfo and management export. I just delivered that.

Re: Policy install fails with malloc failure

the_rock — Fri, 19 Jan 2024 17:46:35 GMT

Mgmt export could help them...not sure if they plan to replicate the issue. Appears fw_loader process is whats consuming most memory when you are installing the policy, but thats always invoked by fwm process itself. Can you run cpwd_admin list when installing the policy please and send? I want to see whats fwm consuming...

Andy

Re: Policy install fails with malloc failure

Oliver_Fink — Wed, 24 Jan 2024 11:49:00 GMT

Thank you for your support. I do not understand, how do you want to see from "cpwd_admin list" what fwm is consuming? The policy installation is running for several minutes until it fails. When, do you think, to issue the command?

That is the output from the command when not installing policy:

[Expert@some_host:0]# cpwd_admin list
APP PID STAT #START START_TIME MON COMMAND
CPVIEWD 9096 E 1 [16:51:13] 18/1/2024 N cpviewd
CPVIEWS 26854 E 1 [14:25:47] 19/1/2024 N cpview_services
CVIEWAPIS 9127 E 1 [16:51:13] 18/1/2024 N cpview_api_service
CPD 9148 E 1 [16:51:13] 18/1/2024 Y cpd
FWD 9259 E 1 [16:51:14] 18/1/2024 N fwd -n
FWM 9269 E 1 [16:51:14] 18/1/2024 N fwm
FWMHA 9375 E 1 [16:51:15] 18/1/2024 N fwmha -H
STPR 9412 E 1 [16:51:15] 18/1/2024 N status_proxy
CLOUDGUARD 9441 E 1 [16:51:15] 18/1/2024 N vsec_controller_start
CPM 9979 E 1 [16:51:20] 18/1/2024 N /opt/CPsuite-R81.10/fw1/scripts/cpm.sh -s
SOLR 9990 E 1 [16:51:20] 18/1/2024 N java_solr
RFL 10130 E 1 [16:51:21] 18/1/2024 N LogCore
SMARTVIEW 10196 E 1 [16:51:21] 18/1/2024 N SmartView
INDEXER 10461 E 1 [16:51:24] 18/1/2024 N /opt/CPrt-R81.10/log_indexer/log_indexer
SMARTLOG_SERVER 10584 E 1 [16:51:25] 18/1/2024 N /opt/CPSmartLog-R81.10/smartlog_server
REPMAN 10896 E 1 [16:51:29] 18/1/2024 N java_repository_manager
DASERVICE 10980 E 1 [16:51:30] 18/1/2024 N DAService_script
AUTOUPDATER 11012 E 1 [16:51:31] 18/1/2024 N AutoUpdaterService.sh
CPSM 8319 E 1 [16:53:41] 18/1/2024 N cpstat_monitor
LPD 27238 E 1 [16:57:37] 18/1/2024 N lpd

That did not change during policy install.

Re: Policy install fails with malloc failure

the_rock — Wed, 24 Jan 2024 12:27:55 GMT

Sorry, I said it wrong...I just wanted to confirm fwm process shows E 1, which ot does. Any news from TAC?

Re: Policy install fails with malloc failure

Oliver_Fink — Wed, 24 Jan 2024 13:23:48 GMT

They are still investigating.

Re: Policy install fails with malloc failure

the_rock — Wed, 24 Jan 2024 13:25:15 GMT

K, fair enough. My gut feeling tells me this is most likely management issue, because if you think about it logcially, its odd it would fail for all 40 firewalls.

Andy

Re: Policy install fails with malloc failure

the_rock — Wed, 24 Jan 2024 13:27:35 GMT

Have you seen this?

https://support.checkpoint.com/results/sk/sk104783

Re: Policy install fails with malloc failure

Oliver_Fink — Wed, 24 Jan 2024 13:29:48 GMT

Yes, it seems so. Today, I tried again and had a policy install with 35 full and 5 accelerated installs. That worked for all gateways. If I enforce full policy install for all 40 gateways, it fails for all gateways. I am sure it is a management issue.

Re: Policy install fails with malloc failure

the_rock — Wed, 24 Jan 2024 13:31:09 GMT

Based on pure logic, I agree 100%

Re: Policy install fails with malloc failure

Oliver_Fink — Wed, 24 Jan 2024 13:33:47 GMT

Yes. But very wrong version. And it requires a hotfix. I expect the same, here.