topic Re: R80.20 issue with fw monitor - all the buffers are full in Hyperscale Firewall (Maestro)

R80.20 issue with fw monitor - all the buffers are full

HeikoAnkenbrand — Mon, 22 Jul 2019 09:57:47 GMT

Under R80.20 with the latest jumbo hotfix the following error occurs if I start fw monitor:

fwmonitor_kiss_add_to_global_buf: all the buffers are full

System 12000 appliance:

- 8 coreXl instances

- approximatly 25000 connections

- enough free memory

Re: R80.20 issue with fw monitor - all the buffers are full

kobil — Mon, 22 Jul 2019 17:40:23 GMT

is this R80.20SP?

Re: R80.20 issue with fw monitor - all the buffers are full

HeikoAnkenbrand — Fri, 26 Jul 2019 20:54:21 GMT

R80.20 on 12K appliance.

Re: R80.20 issue with fw monitor - all the buffers are full

Chris_Wilson — Tue, 13 Aug 2019 13:14:38 GMT

I am getting the same issue when I do a fw ctl zdebug drop.

@;1709167040;[cpu_0];[SIM-206652832];fwmonitor_kiss_add_to_global_buf: all the buffers are full ;

@;1709167041;[cpu_0];[SIM-206652832];fwmonitor_kiss_add_to_global_buf: all the buffers are full ;

Re: R80.20 issue with fw monitor - all the buffers are full

HeikoAnkenbrand — Tue, 13 Aug 2019 14:19:24 GMT

Hi @kobil,

any news in this case?

Re: R80.20 issue with fw monitor - all the buffers are full

Andreas_Aust — Thu, 12 Sep 2019 08:54:25 GMT

same here R80.20 latest Jumbo

Re: R80.20 issue with fw monitor - all the buffers are full

Thomas_Eichelbu — Thu, 19 Sep 2019 09:23:51 GMT

Hello,

yes i also got in on R80.20SP with Take 50 ...

[1_03]@;1417195069;[cpu_0];[SIM-206251208];fwmonitor_kiss_add_to_global_buf: all the buffers are full ;
[1_03]@;1417195269;[cpu_0];[SIM-206251208];fwmonitor_kiss_add_to_global_buf: all the buffers are full ;
[1_03]@;1417195569;[cpu_0];[SIM-206251208];fwmonitor_kiss_add_to_global_buf: all the buffers are full ;
[1_01]@;1472195669;[cpu_4];[SIM-206976200];fwmonitor_kiss_add_to_global_buf: all the buffers are full ;
[1_01]@;1472195969;[cpu_4];[SIM-206976200];fwmonitor_kiss_add_to_global_buf: all the buffers are full ;

is this issue for TAC case ... normal operation with this debugs is not really possible ...

best regards.
Thomas.

Re: R80.20 issue with fw monitor - all the buffers are full

Thomas_Eichelbu — Thu, 19 Sep 2019 09:35:04 GMT

hello,

i also see this sort of messages all over /var/log/messages:

Sep 19 11:31:19 2019 NWATSBGFWCL01-ch01-04 last message repeated 439284 times
Sep 19 11:31:33 2019 NWATSBGFWCL01-ch01-04 last message repeated 199578 times
Sep 19 11:31:33 2019 NWATSBGFWCL01-ch01-04 cpviewd: Destroying the lists of sensors
Sep 19 11:31:33 2019 NWATSBGFWCL01-ch01-04 kernel: [SIM4];fwmonitor_kiss_add_to_global_buf: all the buffers are full
Sep 19 11:31:48 2019 NWATSBGFWCL01-ch01-04 last message repeated 221176 times
Sep 19 11:31:48 2019 NWATSBGFWCL01-ch01-04 cpviewd: Destroying the lists of sensors
Sep 19 11:31:48 2019 NWATSBGFWCL01-ch01-04 kernel: [SIM4];fwmonitor_kiss_add_to_global_buf: all the buffers are full
Sep 19 11:32:19 2019 NWATSBGFWCL01-ch01-04 last message repeated 366866 times
Sep 19 11:32:33 2019 NWATSBGFWCL01-ch01-04 last message repeated 152620 times
Sep 19 11:32:33 2019 NWATSBGFWCL01-ch01-04 cpviewd: Destroying the lists of sensors
Sep 19 11:32:33 2019 NWATSBGFWCL01-ch01-04 kernel: [SIM4];fwmonitor_kiss_add_to_global_buf: all the buffers are full
Sep 19 11:32:48 2019 NWATSBGFWCL01-ch01-04 last message repeated 168848 times
Sep 19 11:32:48 2019 NWATSBGFWCL01-ch01-04 cpviewd: Destroying the lists of sensors
Sep 19 11:32:48 2019 NWATSBGFWCL01-ch01-04 kernel: [SIM4];fwmonitor_kiss_add_to_global_buf: all the buffers are full
Sep 19 11:33:19 2019 NWATSBGFWCL01-ch01-04 last message repeated 324457 times

maybe this SK helps out:
"/var/log/messages shows 'Log buffer is full'
sk52100
i will try this in a maintenance window!

best regards
Thomas

Re: R80.20 issue with fw monitor - all the buffers are full

Chris_Wilson — Thu, 19 Sep 2019 14:12:03 GMT

try running this command. Fixed it for me.

run fw monitor -U on the gateway

Re: R80.20 issue with fw monitor - all the buffers are full

Chen_Muchtar — Thu, 19 Sep 2019 17:36:21 GMT

This is a known issue, affecting fw monitor only, errors can be ignored

We will make sure relevant fix is to be included in future JHF

Re: R80.20 issue with fw monitor - all the buffers are full

3315cd11-6366-4 — Fri, 20 Sep 2019 14:20:55 GMT

I also have this issue, can you tell me what the -U switch does before I try it?

thanks

Chris

Re: R80.20 issue with fw monitor - all the buffers are full

Chen_Muchtar — Sun, 27 Oct 2019 23:15:52 GMT

A fix was released as part of R80.20 JHF Take 117