topic Re: Wa -IO-wait in General Topics

Wa -IO-wait

Marco_Valenti — Tue, 05 Jun 2018 10:41:03 GMT

Hey check mates

Sometimes on some old type of appliance from the top command you can see that one of the highest value is %wa causing high cpu load

Most of the time is a temporary issue that will auto resolve.

My question is if someone was bothered enough from that issue to find a way to discover wich process is causing high %wa or in his experience the time spent on resolving that is just wasted time

Thanks

Re: Wa -IO-wait

Vladimir — Tue, 05 Jun 2018 12:34:50 GMT

Please check your memory utilization. If it is consistently high and you are swapping to disk, it may be better to simply upgrade the RAM on the appliance and, if they are in 32 bit mode, change them to 64 bit, to take advantage of it.

Re: Wa -IO-wait

G_W_Albrecht — Tue, 05 Jun 2018 12:44:37 GMT

Which appliance model ? Which CP version ?

Re: Wa -IO-wait

Marco_Valenti — Tue, 05 Jun 2018 13:35:37 GMT

sadly was not a memory issue

Re: Wa -IO-wait

Marco_Valenti — Tue, 05 Jun 2018 13:38:12 GMT

gaia 4000 series r77.30

Re: Wa -IO-wait

Kaspars_Zibarts — Tue, 05 Jun 2018 14:11:17 GMT

can be be wait for disk access, I would check that too

High CPU utilization after upgrade to R77.30

Re: Wa -IO-wait

Marco_Valenti — Tue, 05 Jun 2018 14:18:43 GMT

thanks , despite it is not an upgrade could be relevant anyway

Re: Wa -IO-wait

Kaspars_Zibarts — Tue, 05 Jun 2018 14:41:08 GMT

There are more articles if you search on io wait and r77.30 that was just to throw ideas in the air

Re: Wa -IO-wait

Timothy_Hall — Tue, 05 Jun 2018 19:22:48 GMT

> sadly was not a memory issue

Are you sure Marco Valenti? Running free -m reports zero for swap utilization on the last line of the output? Lack of RAM is by far the most common cause of high wa values on a gateway. This can also potentially be caused by a runaway process, failing hard drive, or possibly enabling gateway features that incur a large number of process space trips (such as HTTPS Inspection), process space trips and what causes them are covered extensively in Chapter 10 of the second edition of my book.

The tool of choice for identifying other events that are causing high wa is iotop, but that tool is not available for the 2.6.18 kernel version that Gaia uses.

Be careful trying to observe different types of CPU utilization with Check Point tools like the SmartView Monitor and cpview/cpstat, as these tools tend to roll up us/ni into a single "User CPU" number and sy/si/hi/wa/st into a single "Kernel CPU" number. I assume you are using top or sar to observe the high wa?

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

Re: Wa -IO-wait

Marco_Valenti — Wed, 06 Jun 2018 07:16:11 GMT

yep top was used for that case and it does not seem to be a memory issue at least for the last case that was brought to my attention

Re: Wa -IO-wait

Markus_Genser — Wed, 06 Jun 2018 13:50:17 GMT

As %wa is caused by disc usage and you checked the available memory if the gateway started swapping.

Did the GW lost the connection to the management server and had to log locally?

You can check $FWDIR/log/ for local logs.

Alternatively, what was the "top -S" output during when you noticed the high %wa, was gzip involved?

A classic would be a repeatedly crashing process with coredumps.

Re: Wa -IO-wait

RickHoppe — Wed, 06 Jun 2018 15:43:38 GMT

Security Management R80.20 EA is running on kernel 3.10.0, so iotop is included in there (for future troubleshooting).

[Expert@FWMGMT:0]# fw ver
This is Check Point's software version R80.20 - Build 068

[Expert@FWMGMT:0]# uname -a
Linux FWMGMT 3.10.0-693cpx86_64 #1 SMP Tue Feb 6 12:13:02 IST 2018 x86_64 x86_64 x86_64 GNU/Linux

Security Gateway R80.20 EA is still running on kernel 2.6.18, so no iotop for now.

[Expert@GW2:0]# fw ver
This is Check Point's software version R80.20 - Build 068
[Expert@GW2:0]# uname -a
Linux GW2 2.6.18-92cpx86_64 #1 SMP Tue May 15 18:55:11 IDT 2018 x86_64 x86_64 x86_64 GNU/Linux