R80.40 100% CPU

[ggiordano]

Hi mates

Does someone meet performance issue after upgrade to R80.40?

At my 2 customers I faced out the same performance issue when I upgraded the cluster from R80.xx to R80.40 last take.

For both I had to downgrade to the previous version because critical environments where I cannot wait for TAC investigation.

this is the reason why I'm sharing my findings

In both cases I see the active cluster member suddenly has more CPUs 100% usage.

When it happens the gateway is unresponsive and the TOP output shows high usage for "watchdog" daemons.

Reverting to previous version, the performance on the gateway is as expected.

[JackPrendergast]

Hi,

Firstly a few questions.

What hardware are you upgrading?

What version are you upgrading from?

also, when you say ‘watchdog’ daemons - are you referring to any of the daemons monitored by watchdog? Or are you referring to ‘cpwd’ running at 100%?

any other log files collected you could share?

Seems suspicious either way. I’ve upgraded countless clusters to R80.40 without a hitch.

[ggiordano]

Hi

the upgrade was performed from R80.10.

in a case the cluster is based on 15600 appliances and the other case the cluster is based on 5600 appliances.

TheTOP output, when I meet the issue, I saw 2 "watchdog" processes are 100%

Unfortunately I didn't get any log files.

The messages log file showed errors about GNAT isn't able to de-allocate resources. This issue was mitigated disabling the GNAT feature, but it didn't fix the issue

[HeikoAnkenbrand · Accepted solution]

Hi @ggiordano,

You may be able to share the following information:
top (press 1)
fwaccel stats -s
fw ctl affinity -l
cpwd_admin list
more /var/log/messages | grep -B 2 -A 5 error
cpinfo -y all

Open Server or appliance?

PS:
I have also running many CusterXL with R80.40 without problems.

[ggiordano]

unfortunately I cannot provide the output because I downgraded the cluster to R80.30 because the business impact was very high