After migrating a HP ProLiant DL380 G7 HA-Cluster from R77.30 to R80.20 today I'm experiencing an extremely high CPU usage by caused the pdpd daemon causing all identity agents not being able to connect and authenticate end users. When users are at home in the evening hours everything becomes normal. Anyone experienced this as well? Besides replacing the gateway with a better sized one is there anything we could tune? The onboard NIC's are in use while HCL recommends to avoid it (Ouch). pdpd is already set to use CPU 8.
System Firewall Cluster Node (HA)
Type ProLiant DL380 G7
OS Gaia R80.20 JHF (Take 74) @ 64-bit
CPUSE Build 1676
CPU 12 Cores 8 licensed | SMT: - | Load 7.23%
RAM 14 GB (Free: 0 GB) | Swapping 176 KB
SecureXL On | Multi-Queue Interfaces -
CoreXL On (11 Cores) | Dynamic Dispatcher: On
@Timothy_Hall , this is the result from your Super7:
[Executing:]# fwaccel stat
+---------------------------------------------------------------------------------+
|Id|Name |Status |Interfaces |Features |
+---------------------------------------------------------------------------------+
|0 |SND |enabled |eth8,eth9,eth10,eth11, |
| | | |eth4,eth5,eth6,eth7,eth0,|
| | | |eth1,eth2,eth3 |Acceleration,Cryptography |
| | | | |Crypto: Tunnel,UDPEncap,MD5, |
| | | | |SHA1,NULL,3DES,DES,AES-128, |
| | | | |AES-256,ESP,LinkSelection, |
| | | | |DynamicVPN,NatTraversal, |
| | | | |AES-XCBC,SHA256 |
+---------------------------------------------------------------------------------+
Accept Templates : disabled by Firewall
Layer FWEXT Security disables template offloads from rule #230
Throughput acceleration still enabled.
Drop Templates : enabled
NAT Templates : disabled by Firewall
Layer FWEXT Security disables template offloads from rule #230
Throughput acceleration still enabled.
[Executing:]# fwaccel stats -s
Accelerated conns/Total conns : 816/44272 (1%)
Accelerated pkts/Total pkts : 5463775040/5959914034 (91%)
F2Fed pkts/Total pkts : 496138994/5959914034 (8%)
F2V pkts/Total pkts : 20585639/5959914034 (0%)
CPASXL pkts/Total pkts : 498278614/5959914034 (8%)
PSLXL pkts/Total pkts : 2212031456/5959914034 (37%)
CPAS inline pkts/Total pkts : 0/5959914034 (0%)
PSL inline pkts/Total pkts : 0/5959914034 (0%)
QOS inbound pkts/Total pkts : 0/5959914034 (0%)
QOS outbound pkts/Total pkts : 0/5959914034 (0%)
Corrected pkts/Total pkts : 0/5959914034 (0%)
[Executing:]# grep -c ^processor /proc/cpuinfo && /sbin/cpuinfo
12
HyperThreading=disabled
[Executing:]# fw ctl affinity -l -r | more
CPU 0: eth8 eth9 eth10 eth11 eth4 eth5 eth6 eth7 eth0 eth1 eth2 eth3
CPU 1: fw_5
in.geod usrchkd pepd scanengine_s vpnd mpdaemon pdpd in.acapd in.emaild.smtp lpd in.asessiond rtmd in.msd fwd rad cpd cprid
CPU 2: fw_8
in.geod usrchkd pepd scanengine_s vpnd mpdaemon pdpd in.acapd in.emaild.smtp lpd in.asessiond rtmd in.msd fwd rad cpd cprid
CPU 3: fw_2
in.geod usrchkd pepd scanengine_s vpnd mpdaemon pdpd in.acapd in.emaild.smtp lpd in.asessiond rtmd in.msd fwd rad cpd cprid
CPU 4: fw_9
in.geod usrchkd pepd scanengine_s vpnd mpdaemon pdpd in.acapd in.emaild.smtp lpd in.asessiond rtmd in.msd fwd rad cpd cprid
CPU 5: fw_3
in.geod usrchkd pepd scanengine_s vpnd mpdaemon pdpd in.acapd in.emaild.smtp lpd in.asessiond rtmd in.msd fwd rad cpd cprid
CPU 6: fw_6
in.geod usrchkd pepd scanengine_s vpnd mpdaemon pdpd in.acapd in.emaild.smtp lpd in.asessiond rtmd in.msd fwd rad cpd cprid
CPU 7: fw_0
in.geod usrchkd pepd scanengine_s vpnd mpdaemon in.acapd in.emaild.smtp lpd in.asessiond rtmd in.msd rad cpd cprid
CPU 8:
CPU 9: fw_4
in.geod usrchkd pepd scanengine_s vpnd mpdaemon in.acapd in.emaild.smtp lpd in.asessiond rtmd in.msd rad cpd cprid
CPU 10: fw_7
in.geod usrchkd pepd scanengine_s vpnd mpdaemon in.acapd in.emaild.smtp lpd in.asessiond rtmd in.msd rad cpd cprid
CPU 11: fw_1
in.geod usrchkd pepd scanengine_s vpnd mpdaemon in.acapd in.emaild.smtp lpd in.asessiond rtmd in.msd rad cpd cprid
All:
The current license permits the use of CPUs 0, 1, 2, 3, 4, 5, 6, 7 only.
[Executing:]# netstat -ni | more
Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0 1500 0 462740 0 0 0 20035876 0 0 0 BMRU
eth1 1500 0 0 0 0 0 0 0 0 0 BMU
eth2 1500 0 14380 0 0 0 66 0 0 0 BMRU
eth3 1500 0 0 0 0 0 0 0 0 0 BMU
eth4 1500 0 703032870 0 0 0 717938649 0 0 0 BMRU
eth4.604 1500 0 5648687 0 0 0 15032263 0 0 0 BMRU
eth4.614 1500 0 2192997 0 0 0 4829218 0 0 0 BMRU
eth4.624 1500 0 456325848 0 0 0 518681961 0 0 0 BMRU
eth4.634 1500 0 230299374 0 0 0 181932000 0 0 0 BMRU
eth4.670 1500 0 33711 0 0 0 14341 0 0 0 BMRU
eth4.742 1500 0 8437521 0 0 0 3943037 0 0 0 BMRU
eth4.770 1500 0 90401 0 0 0 386716 0 0 0 BMRU
eth5 1500 0 238714661 0 0 0 257576241 0 0 0 BMRU
eth5.602 1500 0 58496455 0 0 0 54996071 0 0 0 BMRU
eth5.605 1500 0 180064740 0 0 0 202893390 0 0 0 BMRU
eth5.615 1500 0 149135 0 0 0 443051 0 0 0 BMRU
eth6 1500 0 1084032057 0 321 0 1031148166 0 0 0 BMRU
eth6.603 1500 0 28780589 0 0 0 29674771 0 0 0 BMRU
eth6.606 1500 0 200973355 0 0 0 203472426 0 0 0 BMRU
eth6.616 1500 0 60 0 0 0 1375 0 0 0 BMRU
eth6.623 1500 0 685674334 0 0 0 679943082 0 0 0 BMRU
eth6.626 1500 0 48853 0 0 0 55223 0 0 0 BMRU
eth6.633 1500 0 89167501 0 0 0 66527473 0 0 0 BMRU
eth6.724 1500 0 79383049 0 0 0 55542371 0 0 0 BMRU
eth7 1500 0 1510933184 0 4460 0 1715055862 0 0 0 BMRU
eth8 1500 0 410325078 0 2132 0 14642643 0 0 0 BMRU
eth8.608 1500 0 395668331 0 0 0 466538 0 0 0 BMRU
eth8.800 1500 0 14652423 0 0 0 14176945 0 0 0 BMRU
eth9 1500 0 4418240 0 0 0 43687204 0 0 0 BMRU
eth10 1500 0 1050639628 0 0 0 934246991 0 0 0 BMRU
eth10.601 1500 0 530894165 0 0 0 547398536 0 0 0 BMRU
eth10.611 1500 0 209048 0 0 0 154341 0 0 0 BMRU
eth10.621 1500 0 456124650 0 0 0 360206871 0 0 0 BMRU
eth10.631 1500 0 63407433 0 0 0 29237069 0 0 0 BMRU
eth11 1500 0 987797444 0 182 0 1456539685 0 0 0 BMRU
eth11.600 1500 0 987793112 0 0 0 1468969765 0 0 0 BMRU
lo 16436 0 54653517 0 0 0 54653517 0 0 0 LRU
[Executing:]# fw ctl multik stat
ID | Active | CPU | Connections | Peak
----------------------------------------------
0 | Yes | 7 | 5557 | 15247
1 | Yes | 11 | 5542 | 8577
2 | Yes | 3 | 5728 | 8341
3 | Yes | 5 | 5620 | 8465
4 | Yes | 9 | 5850 | 8675
5 | Yes | 1 | 5550 | 8470
6 | Yes | 6 | 5612 | 8364
7 | Yes | 10 | 5796 | 8525
8 | Yes | 2 | 5621 | 8392
9 | Yes | 4 | 5739 | 8788
[Executing:]# cpstat os -f multi_cpu
Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1| 0| 76| 24| 76| ?| 4922|
| 2| 8| 32| 60| 40| ?| 4922|
| 3| 11| 29| 60| 40| ?| 4923|
| 4| 9| 31| 60| 40| ?| 4923|
| 5| 12| 31| 57| 43| ?| 4923|
| 6| 9| 32| 59| 41| ?| 4924|
| 7| 13| 26| 62| 38| ?| 4924|
| 8| 7| 31| 62| 38| ?| 4924|
| 9| 0| 2| 98| 2| ?| 4925|
| 10| 9| 26| 65| 35| ?| 4925|
| 11| 12| 26| 62| 38| ?| 4926|
| 12| 7| 29| 63| 37| ?| 4926|
---------------------------------------------------------------------------------
[Executing:]# fw ctl affinity -l -a
eth8: CPU 0
eth9: CPU 0
eth10: CPU 0
eth11: CPU 0
eth4: CPU 0
eth5: CPU 0
eth6: CPU 0
eth7: CPU 0
eth0: CPU 0
eth1: CPU 0
eth2: CPU 0
eth3: CPU 0
fw_0: CPU 7
fw_1: CPU 11
fw_2: CPU 3
fw_3: CPU 5
fw_4: CPU 9
fw_5: CPU 1
fw_6: CPU 6
fw_7: CPU 10
fw_8: CPU 2
fw_9: CPU 4
in.geod: CPU 1 2 3 4 5 6 7 9 10 11
usrchkd: CPU 1 2 3 4 5 6 7 9 10 11
pepd: CPU 1 2 3 4 5 6 7 9 10 11
scanengine_s: CPU 1 2 3 4 5 6 7 9 10 11
vpnd: CPU 1 2 3 4 5 6 7 9 10 11
mpdaemon: CPU 1 2 3 4 5 6 7 9 10 11
pdpd: CPU 8
in.acapd: CPU 1 2 3 4 5 6 7 9 10 11
in.emaild.smtp: CPU 1 2 3 4 5 6 7 9 10 11
lpd: CPU 1 2 3 4 5 6 7 9 10 11
in.asessiond: CPU 1 2 3 4 5 6 7 9 10 11
rtmd: CPU 1 2 3 4 5 6 7 9 10 11
in.msd: CPU 1 2 3 4 5 6 7 9 10 11
fwd: CPU 1 2 3 4 5 6
rad: CPU 1 2 3 4 5 6 7 9 10 11
cpd: CPU 1 2 3 4 5 6 7 9 10 11
cprid: CPU 1 2 3 4 5 6 7 9 10 11
The current license permits the use of CPUs 0, 1, 2, 3, 4, 5, 6, 7 only.
Thanks in advance for any comments and suggestions.
